[ENHANCEMENT](transportation) Make segment connectors required - #692
Conversation
`connectors` was optional with a `[]` default that its own `min_length=2` constraint rejected. Pydantic does not validate defaults unless `validate_default=True`, so the invalid value was accepted on the way in and refused on the way back out: a segment built without connectors dumped to `connectors: []`, which re-validation rejected. Every segment is physically connected to at least two connectors, and the published data agrees -- across all 350,469,378 segments in release 2026-08-19.0 there is not one null, empty, or single-element `connectors`. So the field becomes required rather than optional-with-a-default, and moves up beside `subtype` in the model's required section. The YAML schema gains `connectors` in its segment `required` list, dropping the matching `default: []` beside `minItems: 2`. Every valid segment example gains a two-connector array, and so does every segment counterexample that was not testing connectors, so each keeps failing only for the defect it names. Both corpora are updated: `reference/` (read by the test suite) and the root `examples/`/`counterexamples/` trees (the docs site symlinks `examples/`). `theme-type-mismatch.json` is left alone -- it declares `theme: buildings` with `type: segment` and never reaches a segment arm. Fixes #669 Signed-off-by: Seth Fitzsimmons <seth@mojodna.net>
🗺️ Schema reference docs preview is live!
Note ♻️ This preview updates automatically with each push to this PR. |
We should discuss this. I'm not 100% clear on how we should handle the pre- |
|
I think it makes sense to target this to John McCall (@lowlydba) Do we have to do anything to keep |
No, that's already automated: If a rebase ever conflicts, the workflow opens an issue assigned to the triggering PR's author for manual resolution. |
The base branch was changed.
Major change release plan
A. Expected release date for this MAJOR change
Bundled into 2.0.0. Since it matches the published data, it doesn't strictly need to be treated as a breaking change.
B. Related MINOR change steps
C. Public documentation and messaging plan
The schema reference regenerates from the model, and every Docusaurus segment example in this PR now carries a
connectorsarray, so the visible change is thatconnectorsmoves into the required-property list for all three segment subtypes. This is effectively a fix for the docs to match what's published in the data.Description
connectorswas declared optional with an empty-list default that its ownmin_length=2constraint rejects:Pydantic does not validate defaults unless
validate_default=True, so the invalid value was accepted on the way in and refused on the way back out. A segment constructed without connectors dumped toconnectors: [], and re-validating that dump failed — read → dump → read was not idempotent for any segment lacking connectors.Rather than correct the default, this makes the field required. A segment is by definition physically connected to at least two connectors, the published data has always reflected that, and an optional field whose only legal values are arrays of two or more was never expressing the real constraint.
Changes:
connectorsis nowlist[ConnectorReference]with no default, declared besidesubtypein the model's required section.connectorsin its segmentrequiredlist and drops the matchingdefault: []that sat besideminItems: 2.reference/(read by the test suite) and the rootexamples//counterexamples/trees (the docs build symlinksexamples/).theme-type-mismatch.jsonis deliberately untouched — it declarestheme: buildingswithtype: segmentand never reaches a segment arm.Reference
Testing
No published data is affected. Every segment in the most recent release satisfies the tightened constraint — verified against release
2026-08-19.0(350,469,378 segments across 128 partitions), which returned zero nulls, zero empty arrays, and zero single-element arrays for all three subtypes:This PR adds no new test, and the example updates are not proofs of the fix — they are the corpus catching up to a tightened schema. What pins the change is the regenerated JSON Schema baseline, which fails if
connectorsstops being required, and the round trip that motivated #669, which now succeeds because the field has no default at all rather than a corrected one.Separately verified while investigating: across every field default in the whole model tree, the three segment arms and their shared
TransportationSegmentbase were the only ones their own annotations reject. Nothing else in the schema had the defect.Required-ness propagated to the generated PySpark layer, not just the JSON Schema: the regenerated
transportation/segment.pynow emits_connectors_checkwithcheck_required(F.col("connectors")). Control: optionalaccess_restrictionsstill gets no such check.JSON Schema baseline delta is exactly
connectorsjoiningrequiredin all three arms. Theconnectorssubschema itself is byte-identical to before, because the generator already emitted a non-nullable array and expressed optionality solely throughrequired.Full suite: 6215 passed.
Checklist
Documentation website
Docs preview for this PR.