Skip to content

[ENHANCEMENT](schema) Remove the non-null defaults from the schema models - #697

Open
Seth Fitzsimmons (sethfitz) wants to merge 2 commits into
vnextfrom
remove-remaining-defaults
Open

[ENHANCEMENT](schema) Remove the non-null defaults from the schema models#697
Seth Fitzsimmons (sethfitz) wants to merge 2 commits into
vnextfrom
remove-remaining-defaults

Conversation

@sethfitz

Copy link
Copy Markdown
Collaborator

Closes #696

Implements the policy adopted in #695 for the four remaining declaration sites. No schema field carries a non-null default after this change; a census over discover_models() reports zero, across 54 models.

Eleven fields, four sites in four files — the other seven are inherited and re-reported by subclasses.

Declaration Was Now
common/level.py Stacked.level = 0 Level | None = None
base/land.py Land.class_, Land.subtype LandClass.LAND, LandSubtype.LAND required
base/water.py Water.class_, Water.subtype WaterClass.WATER, WaterSubtype.WATER required
transportation/segment/_common.py SpeedLimitRule.is_max_speed_variable = False bool | None, no default

Land and Water now match Infrastructure and LandUse, which already declared class_ and subtype required with no default. Removing the level default also retires a # type: ignore[assignment] that existed only to admit it — Level is a NewType over Annotated[int32, ...], so 0 was never a value of the field's declared type.

The absence sentence went on the field, not the type

Stacked.level's description states that a feature without a level is at visual level. That sentence is on the field annotation rather than on the Level NewType, because Level is also the type of LevelRule.value in transportation/segment/_common.py, which is required — an absence sentence there renders into a context where absence is impossible.

is_max_speed_variable keeps its description

The default is removed; the description is untouched. What absence means for this field is a domain question, and the existing wording ("Indicates a variable speed corridor") does not define the term precisely enough to extend safely. Both are worth settling with the transportation folks, separately from this change. Flagging it explicitly because #696's second acceptance criterion — that every meaning a removed default carried is stated in the field description — is met for level and moot for class_/subtype, but left open here.

No consumer sees a difference

Measured against release 2026-08-19.0, one partition per type. No published row carries any of these defaults:

Field Rows measured Equal to the default Null
Land.level 1,174,629 0 1,174,529
Water.level 1,985,644 0 1,985,620
Land.class/subtype 1,174,629 n/a 0 null
Water.class/subtype 1,985,644 n/a 0 null
SpeedLimitRule.is_max_speed_variable 601,764 rules 0 (false) 601,497

The publisher already writes class and subtype on every row, so making them required tightens the model to what the release already contains.

Verification

  • The JSON Schema goldens were hand-written to the predicted shape first, then the models changed. Seven failed as predicted, and all fifteen went green on the model edit — so the emitted schema matches what was intended rather than whatever the generator happened to produce. That ordering is what caught the Level NewType problem above.
  • land, water and segment partitions from 2026-08-19.0 validate clean under overture-validate --skip-schema-check: 0 errors across 5,816,361 rows. That zero is a real result, not a dead check — injecting nulls into 333 class and 200 subtype values makes the same command report class:required / subtype:required and exit 1.
  • Regenerated PySpark and Markdown diffs read, not just regenerated. PySpark gains only the four required-field checks (_class_check splits into _class_required_check + _class_enum_check, same for subtype); there is no StructType diff, since schema_builder.py renders every field nullable regardless of requiredness. Markdown drops (optional) from the four fields and the example rows change 0null and falsenull — those examples were displaying the injected default rather than what the release contains.
  • Full suite green on both commits via the pre-commit gate.

Follow-ups

  • What is_max_speed_variable asserts, and what its absence means.
  • Enforcement — a check that fails on any field carrying a default, so this cannot regress. Better placed in the CLI than in a test, so extensions and third-party schemas built on overture-schema-system can run it too.

Implements the policy in #695 for three of the four remaining sites: no
schema field carries a non-null default, and where a value would come
from a default the publisher supplies it in the published data.

- `Stacked.level` drops `= 0` and becomes `None`. The absence sentence
  goes on the field rather than the `Level` NewType, because `Level` is
  also the type of the required `LevelRule.value`, where absence is
  impossible. Removing the default also retires the `# type:
  ignore[assignment]` that existed only to admit it.
- `Land.class_`, `Land.subtype`, `Water.class_`, `Water.subtype` become
  required, matching `Infrastructure` and `LandUse`.

No consumer sees a difference. Measured on release 2026-08-19.0, one
partition per type: `level = 0` appears in 0 of 1,174,629 land and 0 of
1,985,644 water rows, and `class`/`subtype` are never null in either.
Both partitions validate clean against the changed models.

`SpeedLimitRule.is_max_speed_variable` is deliberately left out. Its
default cannot be removed without stating what absence means, and the
data does not settle that: the publisher writes an explicit `false` for
closed-world booleans (`building.has_parts` is 0 null / 4,777,998 false)
but never for this one (0 false in 1,376,379 rules across three
partitions), so absence is an unrecorded value rather than a recorded
"no". That call needs a domain owner.

Refs #696

Signed-off-by: Seth Fitzsimmons <seth@mojodna.net>
Completes the removal begun in the previous commit: `SpeedLimitRule.
is_max_speed_variable` drops `= False`. A rule parsed without the flag
now yields `None` instead of `False`.

The description is left as it stands. What absence means here is a
domain question the schema should not answer by inference, and the
existing wording does not define the term well enough to extend safely.
Both are worth settling, separately, with the transportation folks.

Refs #696

Signed-off-by: Seth Fitzsimmons <seth@mojodna.net>
@github-actions

Copy link
Copy Markdown

🗺️ Schema reference docs preview is live!

🌍 Preview https://staging.overturemaps.org/schema/pr/697/schema/index.html
🕐 Updated Aug 26, 2026 17:55 UTC
📝 Commit daf99c7
🔧 env SCHEMA_PREVIEW true

Note

♻️ This preview updates automatically with each push to this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

change type - major 🚨 Major schema change. See https://lf-overturemaps.atlassian.net/wiki/x/GgDa

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant