CONFORMANCE.md

Conformance

The conformance scorer (tools/lib/conformance.mjs) evaluates a canonical pack against a hand-curated subset of the maturity rubric from spec §5 + §7. The studio exposes the scoring under Advanced -> Conformance and via GET /api/packs/:id/conformance?env=<name>.

Rubric

30 clauses across L1–L5 plus L2X. Each clause is { id, dimension, severity, minTier, description, specRef, evaluate }:

• severity: MUST (weight 1) or SHOULD (weight 0.5).
• minTier: the least-stringent tier at which the clause applies. Rank tier-3 < tier-2 < tier-1. A clause with minTier: tier-3 applies to every pack; one with minTier: tier-1 only applies to tier-1 packs.
• evaluate(canonical) → boolean: run against the env-overlaid spec.
• specRef: section in the upstream spec.md where the rule is defined.

The rubric grades what is declared, placeholders included; npm run pack-conformance -- <pack> (tools/pack-conformance.mjs, DOWNSTREAM.md §11) lists what is still a placeholder — the two answer different questions.

The rubric is intentionally content-focused — schema-enforced rules (e.g. "every SLO has objective + window") aren't duplicated here because the validator already covers them. Conformance only checks what sits above the schema floor.

ID	Dimension	Severity	applies @	Spec
L1.MUST.availability_slo	L1	MUST	tier-3	§5.1
L1.MUST.sli_covered_by_slo	L1	MUST	tier-3	§5.1
L1.MUST.latency_slo	L1	MUST	tier-2	§5.1
L1.SHOULD.domain_slo	L1	SHOULD	tier-1	§5.1
L2.MUST.otlp_receiver	L2	MUST	tier-3	§5.2
L2.MUST.service_name_required	L2	MUST	tier-3	§5.3
L2.MUST.semconv_floor	L2	MUST	tier-3	§5.3
L2.MUST.metrics_exporter	L2	MUST	tier-2	§5.2
L2.MUST.metrics_logs_traces_backends	L2	MUST	tier-2	§5.12.1
L2.MUST.semconv_current	L2	MUST	tier-1	§5.3
L2.MUST.resource_attrs_5plus	L2	MUST	tier-1	§5.3
L2.MUST.log_correlation	L2	MUST	tier-1	§5.3
L2.MUST.logs_and_traces_exporters	L2	MUST	tier-1	§5.2
L2.MUST.tail_sampling	L2	MUST	tier-1	§5.2
L2.SHOULD.backend_gating_enforce	L2	SHOULD	tier-1	§5.12.3
L2X.MUST.extended_backend_refs_resolve	L2X	MUST	tier-3	§5.12.4
L3.MUST.recording_rule_per_slo	L3	MUST	tier-3	§5.5
L3.MUST.service_overview_dashboard	L3	MUST	tier-3	§5.6
L3.MUST.slo_burn_dashboard	L3	MUST	tier-2	§5.6
L3.SHOULD.derived_view	L3	SHOULD	tier-2	§5.5
L3.MUST.tier1_dashboards	L3	MUST	tier-1	§5.6
L4.MUST.multi_window_burn_rate	L4	MUST	tier-2	§5.7
L4.MUST.tier1_voice_route	L4	MUST	tier-1	§5.8
L4.MUST.tier1_at_least_one_automation	L4	MUST	tier-1	§5.9
L4.SHOULD.forecast_on_availability	L4	SHOULD	tier-1	§5.7
L5.MUST.synthetic_probe	L5	MUST	tier-3	§5.11
L5.MUST.tier2_chaos_staging	L5	MUST	tier-2	§5.11
L5.MUST.tier1_chaos_for_each_slo	L5	MUST	tier-1	§5.11
L5.MUST.tier1_weekly_prod_chaos	L5	MUST	tier-1	§5.11
L5.SHOULD.tier1_release_gate	L5	SHOULD	tier-1	§5.10

L1.MUST.latency_slo counts by type, not by direction. The clause asks for one SLI of type threshold or distribution under an SLO. Since spec 1.3 such an SLI carries a direction (good_when: below | above, absent meaning below), and a floor — good_when: above, connected consumers, in-sync replicas — satisfies the clause exactly as a latency ceiling does: the check is by type, as the upstream maturity model states in the clause's conformance cell. Tightening it to good_when: below would be a rubric revision, not part of the 1.3 adoption; no clause changed.

Scoring

must.passed   = applicable MUSTs that pass
must.total    = applicable MUSTs
should.passed = applicable SHOULDs that pass
should.total  = applicable SHOULDs

mustPercent    = must.passed / must.total * 100
scorePercent   = (must.passed + 0.5*should.passed) / (must.total + 0.5*should.total) * 100
conformant     = must.passed === must.total   (per spec §8 definition)

Bundled-pack scores

Pack	Tier	MUST	SHOULD	Conformant
demo-skeleton.pack.yaml	tier-3	9/9 (100%)	0/0	yes
production-curated.pack.yaml	tier-2	13/15 (87%)	0/1	no — 2 honest gaps
target-advanced.pack.yaml	tier-1	25/25 (100%)	5/5 (100%)	yes
payment-service.pack.yaml (canonical example)	tier-1	21/25 (84%)	5/5 (100%)	no — 4 orphan SLOs

The canonical example itself fails 4 MUSTs (L3.MUST.recording_rule_per_slo, L4.MUST.multi_window_burn_rate, L5.MUST.tier1_chaos_for_each_slo, L5.MUST.tier1_weekly_prod_chaos) because two of its five SLOs (api_latency_99_p99_500ms, consumer_success_99_95) aren't covered by a recording rule, burn-rate alert, or chaos experiment. The studio surfaces these as red-X items in Advanced -> Conformance: visible drift the spec's own reference example carries.

Waivers

A waiver (GAP batch 2, B3.2; tools/lib/waivers.mjs) is a time-boxed, reasoned suppression of one finding: a rubric clause (ruleId) and, for the four per-item clauses — L1.MUST.sli_covered_by_slo, L3.MUST.recording_rule_per_slo, L4.MUST.multi_window_burn_rate, L5.MUST.tier1_chaos_for_each_slo (SUBJECT_CLAUSES) — optionally one canonical symbol of it (artefactId: slos.<id> / slis.<id>, the adapter's defines vocabulary, never a JSONPath). clauseSubjects(clauseId, canonical) names the FAILING subjects of such a clause (the spec's own example fails L3 on slos.consumer_success_99_95, L4 on slos.api_latency_99_p99_500ms and L5 chaos on three SLOs — the "orphan SLOs" above, per clause); it is [] exactly when the clause passes, so subjects and verdict cannot disagree.

A waiver never rewrites the rubric. evaluateConformance is untouched; applyWaiversToConformance returns the SAME report object when no open waiver exists and otherwise adds a waivers block beside the engine's numbers: per clause waived (every failing subject covered — a pack-level waiver covers them all, a scoped one its own and beats the pack-level), partial (some covered: the clause still fails) or expired (none covered; a lapsed waiver would have — the finding fails again and the waiver is shown), the waivers quoted, effective — the same arithmetic as Scoring with the waived clauses read as met (conformant, must, should, scorePercent, mustPercent, byDimension) — and the unused waivers that match no failing clause. States are computed, never stored: active, expired (at or after expiresAt), revoked (history; matches nothing). Schema errors are not waivable: an invalid pack is never graded.

Two homes, one object. On the server a waiver lives on the service record a pack is primarily linked to (POST /api/services/:id/waivers, operator; at most 366 days ahead; one active per (ruleId, artefactId), a new one after expiry; POST /api/waivers/:id/revoke; README "Waive A Conformance Finding") and GET /api/packs/:id/conformance carries the overlay — /api/validate and the library routes keep the bare report. Without a server, packc conformance --waivers <file> reads { "version": 1, "waivers": [...] } — the object the API serves per row — and applies the rubric waivers to the rubric line and --json's rubric.waivers, and the placeholder-rule waivers (placeholder.<family>.<field>) to the placeholder rows (waived partition; --strict fails on unwaived rows alone). The author and revokedBy are the audit actor — a login or the token label, never an email — visible to every member and to anyone a sidecar file or report is shared with.

The service audit report

GET /api/packs/:id/audit-report and packc audit-report (GAP batch 2; README "Export A Service Audit Report", docs/ADAPTER.md "The service audit report") put this rubric's grade first in one document — the engine's numbers headline, a waivers overlay's effective beside them — followed by the placeholders, the verdicts, the waivers, the coverage by family (a family a clause that applies at the graded tier names is required), the goes-blind risks and the declared response path. The report reads the same conformanceReportFor() body the /conformance route sends, so the two never grade one pack differently.

Extending the rubric

Each clause is a self-contained {...} block in tools/lib/conformance.mjs. Add new ones inline with their specRef. The server's GET /api/maturity-rubric will pick them up automatically; tools/test-packs.mjs will re-score every bundled pack against the new clause set.