[FEAT]: Canonical versioned trace/result serializer (WS0-05 PR 1)

[Nina Chikanov (nina-msft)]

WS0-05 — PR 1: Canonical versioned trace/result serializer

Establishes the neutral schema API and migration policy. Existing reporting and xdist consumers are not rewired in this PR.

What this lands

• JSON-only conversion API in rampart/core/serialization.py: serialize_record(*, record: ResultRecord) -> str and deserialize_record(*, data: str) -> ResultRecord. These functions own envelope construction, body conversion, and version dispatch. Neither Result nor ResultRecord exposes to_dict() / from_dict(). Projections obtain structured canonical data with json.loads(serialize_record(record=...)); structured transports can supply JSON text to the reader without importing private conversion helpers or maintaining another serializer.
• One canonical body codec — one cached Pydantic TypeAdapter[Result] over the existing standard dataclasses. The adapter and body-schema generation machinery live in the serialization module, not on Result; there is no parallel body model or set of per-type encode/decode functions. Encoding validates the live Result and uses normal Pydantic JSON-mode serialization, including native enum encoding, adapter-local Unicode validation, and Python ISO datetime formatting. The writer does not reconstruct its output through the reader. Nesting limits vary by platform/runtime and can differ between reader and writer: a deeply nested external record may decode but fail to re-encode; limit failures surface as SchemaError.
• Adapter-local policies — a private GetPydanticSchema hook configures copies of generated core-schema nodes. Public fields retain plain dictionary/datetime annotations; no class-wide Pydantic configuration or trace validators are attached to the live dataclasses. Other Pydantic adapters retain their normal behavior. The canonical adapter supplies its datetime/Path resolution namespace. Both result.py and types.py are unchanged from the PR base.
• Thin ResultRecord envelope — a frozen dataclass referencing the live result and optional pytest_nodeid / nonnegative result_index attribution, omitted from output when not recorded. Every serialized record carries the single root version rampart.trace.v1; unsupported versions fail closed through the decoder registry. The nested result body is not a standalone durable record. Stable identity fields and durable artifact descriptors remain deferred until their implementations exist.
• Strict boundary validation — all nine nested types, finite numbers, JSON-only free-form maps, closed enums, and passthrough harm-category strings. Missing fields use declared defaults without retaining original field presence; explicit null is accepted only for nullable fields. Payload IDs are required on read rather than generated. Population references require positive size, a nonnegative index below size, and threshold in [0, 1], without changing live dataclass construction. Python ISO timestamp formatting is preserved. Strings and mapping keys, including attribution, must contain Unicode scalar values: Python surrogate code points are rejected without lossy normalization, while valid JSON surrogate-pair escapes for characters such as emoji remain supported. Malformed JSON, unsupported values, recursion-limit failures, and adapter serialization ValueErrors surface as SchemaError.
• Trace-only payload restrictions — binary formats and non-null artifacts fail before filesystem access; ordinary live binary payload construction and independent Pydantic validation are unchanged.
• Nonmutating metadata preservation — encoding and decoding preserve supported metadata, including transport bookkeeping and loss/truncation markers. Unsupported values are rejected regardless of key; there is no reserved-key filter in the canonical codec. Shared consumer preparation owns bookkeeping separation and must preserve or explicitly relocate loss/truncation evidence. That integration and its real-consumer coverage are tracked in reporting #12820 and xdist #10596.
• Generated open JSON Schema at schemas/trace.v1.schema.json. ResultRecord.json_schema() remains the schema-description API, composing the adapter-derived body and versioned envelope with trace payload restrictions, the Request invariant, and ordinal/population numeric bounds. Runtime class docstrings are omitted in favor of concise trace-specific descriptions. scripts/generate_trace_schema.py --check is the CI drift gate; generation selects the schema filename from TRACE_SCHEMA_VERSION.
• Explicit compatibility decisions — schemas/trace-compatibility.json binds the decision and rationale to fingerprints of the models, codec policies, and published schemas. PR CI compares against the target base and requires a same-major compatibility declaration or an adjacent major bump with retained historical schemas and a migration-document reference; the initial contract is declared explicitly. This enforces a recorded decision, not proof of semantic compatibility or migration correctness, which remain review obligations.
• Explicit structural/semantic distinction — passing JSON Schema validation is necessary but not sufficient for decoding. Integer notation, Unicode scalar strings, Python ISO datetime parseability, finite representable numbers, and the population cross-field constraint index < size remain decoder requirements. These requirements and runtime nesting limits are documented rather than claimed as JSON Schema guarantees.
• Migration and integration policy at docs/concepts/trace-schema.md: additive optional fields require no bump; structural changes require a new major, an adjacent upcaster, and an explicit migration API/CLI; absence means not recorded, not a semantic negative. General extension guidance and the release-based support window are documented; future field names and shapes are not reserved. Transport normalization must prepare a separate result before encoding; a binary text placeholder is a lossy transport view, not a durable replay artifact.

Coverage

Hypothesis-generated record round-trips through the public JSON APIs cover nested JSON values, all nine nested types, enums, optional fields, timestamps, metadata preservation, output stability, and nonmutation. Example-based coverage includes the removed dictionary-method surface, field exhaustiveness, version dispatch, attribution, ordinal/population invariants on both conversion boundaries, additive tolerance, invalid JSON/types/cycles, independent-adapter isolation, schema semantics, and payload rejection before filesystem access. Additional regressions cover surrogate-containing text and mapping keys, valid Unicode/JSON surrogate-pair escapes, supported nested round-trips, and portable recursion/error-boundary handling with exception chaining. A prepared-copy boundary case demonstrates canonical reuse without modifying or relaxing the original result. Compatibility-gate coverage includes stale declarations, semantic-only input changes, version decisions, migration references, historical-schema preservation, and version-aware generation. hypothesis and jsonschema are development-only dependencies.

Scope

Reporting projections, xdist/failure-attachment adoption, and stable identity remain separate follow-up implementations. Existing transport coercion and payload-display behavior require explicit preparation and compatibility work, not a mechanical serializer replacement. No public transport-preparation API or artifact representation is introduced here.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[Behnam (behnam-o)]

Overall, this looks pretty good to me. Please take a look at the inline comments. The schema-version verification comment is materially more important than the others: the current drift check can accept a regenerated, breaking v1 schema without requiring TRACE_SCHEMA_VERSION to change. The remaining comments are primarily about keeping this foundation scoped and deferring consumer-specific or speculative behavior.

[Behnam (behnam-o)]

The latest revision addresses my review comments correctly. The compatibility gate now requires an explicit schema-version decision, and the scope, serialization path, metadata behavior, and invariants look good. Focused tests and static checks pass.