feat #10: add validated administrative area lookup with boundary polygons - #11
feat #10: add validated administrative area lookup with boundary polygons#11manjudr wants to merge 12 commits into
Conversation
Resolves an OpenAgriNet AdministrativeAreaReference (codeScheme, areaCode,
areaLevel) to a name, parent chain and coordinate via a cached CSV snapshot,
so coverageAreas resolution needs no live API or database.
Two stages:
build_gazetteer.py pulls LGD codes and names (GODL-India) into a dated
snapshot: 25,097 rows across Country, State, District,
Block and PostalCode.
join_geometry.py joins published boundary layers onto those LGD codes
and fills latitude, longitude, bbox and provenance.
Boundary geometry has two selectable sources. Survey of India is the default
because it is openly licensed; --source lgd uses BharatMaps geometry, which
resolves Block level as well but is not openly licensed. Every row records
which layer supplied its point, and the manifest records the licence.
The join keys on LGD codes only, never names: 117 areas join on a stable code
while the boundary layer spells the name differently. Points are area-weighted
centroids with an interior-point fallback for concave shapes; unmatched rows
inherit their parent's point, marked as such in point_method.
Standalone build-time utility. Adds no Go package and is not imported by the
adapter runtime. Stdlib Python only.
See tools/gazetteer/README.md for execution steps, per-source detail, the
stored schema, measured coverage and known data-quality outliers.
Reduce the README to what an operator needs: the purpose, the single refresh command, and an appendix stating which source fills which columns. 422 lines down to 138. Also fix a defect the rewrite surfaced while verifying the refresh command: build_gazetteer.py caught only HTTPError, so a dropped TLS connection during a download produced a raw traceback instead of a message. Both stages now catch URLError, retry transient failures with backoff, and remove any partial file. HTTP status errors are still fatal without retry, since a 404 means the asset is genuinely not published.
Adds the built lookup table so consumers can use it without running the pipeline: 25,097 rows resolving Country, State, District, Block and PostalCode to a name, parent and coordinate. Geometry is from Survey of India (openly licensed); every row records its source layer in geometry_source. The download cache (~500 MB) and the optional boundaries.geojsonl export (335 MB) remain gitignored, as both are reproducible from the scripts.
"Gazetteer" is the standard geospatial term but needs looking up, so the tool is renamed to say what it does. The word is removed everywhere rather than only in the directory name, so nothing is left half-renamed: tools/gazetteer/ -> tools/area-lookups/ build_gazetteer.py -> build_areas.py data/gazetteer/ -> data/areas/ gazetteer.csv -> areas.csv --gazetteer -> --areas No behaviour change. Verified the pipeline still runs: 25,097 rows, State 36/36, District 728/784, containment 764/764 clean.
Full refresh re-run with the renamed scripts, which also verifies stage 1's download path end to end. manifest.json now records build_areas.py as its generator.
The rename commit staged the path moves but not the in-file edits, leaving join_geometry.py defaulting to the old data/gazetteer path and still exposing --gazetteer. This carries the actual content: renamed flags, defaults, docstrings and User-Agent.
manjudr
changed the title
A coded coverageArea is an area, not a point, so resolving one to a centroid cannot answer whether a location falls inside it. Add areas.geojsonl: one simplified outline per area, keyed identically to areas.csv, with has_polygon in the CSV so a consumer can tell an outline exists without opening it. Simplification is tolerance-scaled per part, holding the district layer to 11 MB against 460 MB at full fidelity for a 0.23% worst-case area error. Add validate.py as a third stage. It reimplements point-in-polygon rather than importing it, so a bug in the join cannot pass by agreeing with itself, and it exits non-zero to gate a refresh. It found four defects in the CSV, all fixed here: - parent_code was ambiguous. An LGD code is unique within a level, not across levels: 765 codes name a State, a District and a Block at once, so walking a parent chain resolved the wrong area. Add parent_level. - Hairline slivers inflated bounding boxes. A 0.1 m2 splinter stretched Hailakandi 41 km into a neighbour and a 1 m2 spike pulled Kancheepuram's western edge out 16 km, both of which would make a bbox prefilter match points well outside the area. Parts are now floored on area, not extent. - Genuine small parts were dropped by simplification, silently losing extent. The tolerance now backs off rather than discarding a part. - Sundaragada's point sat outside its own simplified outline, and 62 child Blocks had already inherited the pre-correction coordinate. Points are re-derived from the shipped outline before inheritance runs. Both sources and every flag combination now pass validation.
Update: polygons added, and the CSV is now validatedA coded New artifact
Full fidelity would be ~460 MB. The tolerance is scaled per part rather than fixed, because a flat 110 m costs 2% of Lakshadweep's 30 km² while being negligible on a 7,000 km² district; that alone cut worst-case area error from 2.14% to 0.23%.
New stage:
|
manjudr
changed the title
Every example in the OpenAgriNet schema packs that names a State uses
ISO-3166-2, never LGD, and the lookup carried no ISO-3166-2 rows at all.
An `{"codeScheme": "ISO-3166-2", "areaCode": "IN-KA"}` reference matched
nothing, so the planned ONIX plugin would have published no geometry for
the specs' own examples.
Add the ISO 3166-2:IN crosswalk to stage 1: 36 current codes covering
every State and Union Territory, plus the 7 ISO has withdrawn but
publishers still send. This repository was itself sending IN-TG, retired
in favour of IN-TS in November 2023.
Alias rows hold their own copy of the point, box and outline rather than
a reference, so resolving an ISO code is one exact-match lookup on
(code_scheme, area_code, area_level) and the plugin needs no chain
walking. The new same_as column records provenance and lets the
validator prove the copy has not drifted; nothing has to read it.
Stage 2 fills aliases between joining and inheriting, so an ISO row takes
its state's real geometry instead of an inherited country-level point.
Verified: all 43 fill even under --no-inherit.
validate.py gains 6 checks, each confirmed to fail when broken
deliberately: self-reference, dangling same_as, cyclic chains, geometry
that differs from its target, aliases left without a coordinate, and
has_polygon=true with no feature under the alias's own key. The stats
table gains an alias column so rows still reconcile against
joined + inherited + empty.
Costs 43 rows and 2.9 MB of repeated state outlines. Code scanning every
feature for containment should skip rows with a non-empty same_as.
Refreshed end to end, both sources, all flag combinations pass.
|
Pushed Why this was needed. Walking a real event through the lookup by hand rather { "codeScheme": "ISO-3166-2", "areaCode": "IN-KA", "areaLevel": "State" }matched nothing. The plugin would have published no geometry for the specs' own What changed. The ISO 3166-2:IN crosswalk in Alias rows carry their own copy of the point, box and outline rather than a
Verified against the refreshed files:
Rebuilt from a deleted snapshot with the documented chain — exit 0, Two things to know. The 43 state outlines now appear twice, 2.9 MB of a Unrelated gap found while testing and logged on #10: |
The README covered how to refresh the snapshot but not how to consume it, so the resolution rules that matter lived only in the code: key on three fields and never on areaName, infer areaLevel only where the scheme allows it, gate on has_polygon, and reject inherited points. Replaces the ISO-3166-2 prose with five steps and a three-line example.
.DS_Store and .claude were already ignored, but gitignore patterns match exact names, so .claude did not cover .claude-flow. Two directories of per-session scratch were surfacing as untracked; removed, neither held tracked content.
1 participant
Problem
coverageAreasin the OpenAgriNet schema packs is aoneOfbetween a codedAdministrativeAreaReferenceand a BecknGeoJSONGeometry. Publishers preferthe coded form because it is short and stable:
{ "codeScheme": "LGD", "areaCode": "466", "areaLevel": "District" }Consumers that need to draw a map, run a distance filter, or test whether a
location falls inside a provider's coverage need real geometry. There is no
lookup that turns one into the other, and doing it at request time would add a
runtime dependency on a government API.
An ONIX plugin will do this denormalization at publish time, reading a cached
snapshot. That makes the snapshot's correctness the gating concern: a wrong
coordinate here becomes wrong geometry in a published catalog.
Approach
A refreshable snapshot, built by three stdlib-Python scripts under
tools/area-lookups/. Build-time utility only: not a plugin, not part of theadapter runtime, adds no Go package.
build_areas.pypulls LGD codes and names (GODL-India) into a datedsnapshot: 25,097 rows across Country, State, District, Block and PostalCode.
join_geometry.pyjoins published boundary layers onto those LGD codes,filling coordinates and bounding boxes, and writes one simplified outline per
area.
validate.pytries to break the result. 21 checks, exits non-zero, so arefresh is gated on it. Point-in-polygon is reimplemented rather than
imported, so a bug in the join cannot pass by agreeing with itself.
Two output files in
data/areas/latest/, joined on the same(code_scheme, area_code, area_level)key, plus amanifest.jsonrecordingprovenance, licence, coverage and every anomaly for the run:
areas.csvareas.geojsonlPolygon/MultiPolygonper area, for containment testsA coded area is an area, not a point, so the outline is the part that answers
containment. The point remains useful for map pins and distance sorting, and
has_polygonlets a consumer tell from the CSV alone whether an outline exists.The join keys on LGD codes only, never names — 117 areas join on a stable code
while the boundary layer spells the name differently (LGD 466 is
Ahilyanagar;every boundary layer still says
Ahmednagar).Status
Implemented in PR #11 on
feat/area-gazetteer-tool, with the built snapshotcommitted. Full refresh verified end to end; both sources and every flag
combination pass validation.
soipointssoipolygonslgdpointslgdpolygonsNo row is left without a coordinate; anything unjoined inherits its parent's
point, marked in
point_method. Outlines are only ever present for an area'sown geometry, never inherited.
Simplification is tolerance-scaled per part, which holds the default snapshot to
11 MB against ~460 MB at full fidelity for a 0.23% worst-case area error. A flat
tolerance cost 2.14%, because 110 m is negligible on a 7,000 km² district but
consumes 2% of Lakshadweep's 30 km².
Defects
validate.pyfound, all fixedparent_codewas ambiguous. An LGD code is unique within a level, notacross levels: 765 codes name a State, a District and a Block at once —
code 35 is Andaman & Nicobar Islands, Kapurthala, and Baramulla. Walking a
parent chain resolved the wrong area. Added a
parent_levelcolumn.Hailakandi's bbox 41 km into a neighbouring district; a 1 m² spike pulled
Kancheepuram's western edge out 16 km. Either makes a bbox prefilter match
points well outside the area. Parts are now floored on area, not extent —
these slivers have a large extent and no area, which is why an extent-based
check missed them.
extent. The tolerance now backs off rather than discarding a part.
child Blocks had already inherited the pre-correction coordinate. Points are
now re-derived from the shipped outline before inheritance runs.
Open decision — geometry source licence
Two selectable sources, and this needs a call before derived coordinates are
published:
--source soi(default) — Survey of India, openly licensed. Cannotresolve Block at all:
SOI_Subdistrictscarries no LGD field.--source lgd— BharatMaps / NIC. Resolves Block (6,117 points andoutlines), but the upstream is not openly licensed.
The manifest records the licence per run and
geometry_sourcerecords theoriginating layer per row, so the choice is auditable either way.
Follow-ups
published as a release asset —
areas.csvplusareas.geojsonlis~18 MB per refresh and dated snapshots accumulate in git history)
--source lgd(2 genuine upstream errors, 3 knock-ons, 2 correct Puducherry exclaves —
reported, never auto-corrected)
table) keep inheriting district points and will never get an outline
coverageAreasat publish timeSee
tools/area-lookups/README.mdfor the refresh command and the per-sourcecolumn appendix.