Skip to content

DIFF.md

Latest commit

 

History

History
354 lines (289 loc) · 18.2 KB

DIFF.md

File metadata and controls

354 lines (289 loc) · 18.2 KB
Raw

Diff — Behavioural Artefact Matching

tools/lib/diff.mjs compares two adapted, layered packs and powers the drift/gap view exposed by GET /api/diff?a=<packId>&b=<packId>.

The comparison is directional: A is the declared pack under review, B is the reference or live pack. Observogram reports in-scope drift for A against B and keeps unrelated live inventory in a separate outOfScope bucket.

The Claim

Observogram separates two questions:

  1. Identity: is this the same control?
  2. Agreement: do both sides declare the same contract for that control?

This is the core matcher claim that is safe to repeat: a shared identity does not receive credit unless the comparable contract also aligns.

Identity

keyOf(artefact) delegates to tools/lib/artefact-model.mjs, which classifies each artefact family and builds a behavioural identity object. The adapter's positional ids (SLI-01, BAK-03, ALR-02) are never identity.

Representative identities:

Family Identity
SLI / SLO declared contract id, because other controls bind to it
Backend product + signal
Metric series name
Recording rule output series name
Dashboard declared dashboard id
Panel dashboard parent + binds_to target
Pipeline receiver/processor stage name
Pipeline exporter signal + exporter kind
Alert route severity, with duplicates preserved via occurrence ordinals. Channel kinds deliberately stay OUT of identity — live connectors fabricate them when routing cannot be introspected — and surface as decision-bearing drift instead
L2X profiling/network/policy product
L2X mesh/collection product + role

The precise phrasing matters: Observogram matches on behaviour. For reliability contract artefacts, the declared id is the behavioural handle by design.

Identity modes

Behaviour is the default and the only key the server's GET /api/diff uses. diffPacks(a, b, { identity }) pairs by another key — rebadge batch 3, C3, tools/lib/identity-modes.mjs — while behaviour still decides aligned vs drifted, so a pair formed by name or id can drift:

identity Pairs by Answer
omitted, 'behaviour', or identityKeyOf itself the behavioural identity above byte-identical to today's; no identity key (the stored goldens tools/fixtures/golden/diff/ run all three)
'name' nameKeyOf: the name or title, trimmed, white space collapsed, lower case identity: { mode: 'name' } after scope
'id' idKeyOf: a stable id or uid (a dashboard's uid, a spec id); a family whose name is its id (metric, recording rule, alert rule, scrape job) uses the name identity: { mode: 'id' }
a function (artefact, { side }) → '<kind>::…' the caller's key identity: { mode: 'custom' }

Every key keeps the <kind>:: prefix — scope, notObserved and collisions read it — and a function whose key drops it is a TypeError (diffPacks: an identity function must return "<kind>::…" for each artefact (got "<x>" for a <kind>)); an unknown mode id names the three. The engine calls the key with the side ('a' or 'b'), which is how an artefact with no name (or no stable id) is keyed — dashboard::{"unnamed":"a:DASH-01"} — so two unnamed artefacts of a family never pair; each lands in onlyInA / onlyInB. The parked placeholders (scaffold) are keyed by the same function. Where each mode's material comes from, per family, is the table in docs/ADAPTER.md ("Identity modes"). Verdict keys, the traceability graph, the remediation flow and chain history stay on behaviour.

Collision Handling

Multiple artefacts in one pack can share the same identity key: multiple SEV2 routes, duplicate dashboard ids, same-named pipeline stages, or more than one Prometheus metrics backend. diffPacks groups by identity instead of using a last-write-wins Map.

Within each identity group it:

  1. Pairs exact behavioural matches first.
  2. Pairs remaining same-identity artefacts as drifted, choosing the closest remaining contract.
  3. Leaves surplus controls visible as onlyInA or onlyInB.

Duplicate entries receive stable occurrence suffixes such as dashboard::{"id":"overview"}#02, so counts preserve artefacts instead of collapsing them to identity classes.

The result also carries a top-level collisions array — one entry per identity key held by more than one artefact on either side:

{ layer, kind, key, aCount, bCount }

The buckets already preserve every instance; collisions is the explicit fail-loud surface, so callers (including downstream vendors who bucket by identityKeyOf themselves) never have to infer collisions from #NN key suffixes. buildDependencyGraph exposes the same surface as graph.collisions ({ key, kind, count }). otel and baselines keep deliberately empty identities: the spec makes them singular objects, so at most one exists per pack and their {} identity cannot collide.

Alert-route identity precision was evaluated and deliberately declined: fetch-live-pack.mjs and the crawler fabricate a channel kind (an msteams placeholder) when live routing cannot be introspected, so channel-kind identity would split declared routes from their live placeholders and report them falsely missing in production. Severity-keyed identity keeps that pair matched, with the channel difference reported as decision-bearing drift.

Agreement

projectOf(artefact) returns the canonical comparable contract, and deltasOf(a, b) reports top-level fields that differ. A matched pair is:

{ key, a, b, match: 'aligned' | 'drifted', deltas }

aligned means the declared contract matches after normalisation. It does not mean byte-identical source.

Normalisation rules:

  • Object keys are sorted.
  • Arrays are normalised element-wise, sorted, and empty arrays are treated like absent fields.
  • Empty null, undefined, '', {}, and [] values are dropped.
  • Expressions in expr, query, promql, and expression are order-canonicalized when the expression parses cleanly (parser-proven): selector matcher order ({b="2",a="1"} ≡ {a="1",b="2"}), aggregation grouping label order (sum by (a,b) ≡ sum by (b,a)), structural whitespace (rate( x [5m] ) ≡ rate(x[5m])), and whitespace around the symbolic binary operators + * / % ^ == != <= >= < > =~ !~ (a / b ≡ a/b, rate(x[5m]) > 0.5 ≡ rate(x[5m])>0.5). - is never tightened (unary/binary ambiguity) and keyword operators (and, or, unless, bool, offset, by, on, …) keep the space that bounds them. Anything that fails to parse falls back to whitespace collapse only — recorded as textual-fallback by tools/lib/promql-canon.mjs. Explicit non-goals (per PHASE_1_VERDICT_TRUST_RESEARCH.md Workstream B): no algebraic rewrites, no binary-expression or vector-matching reordering, no regex equivalence, no histogram folding.
  • A leading ref: on the reference-bearing fields slo, sli, trigger, error_budget_policy — and on expr when the whole value is a reference (ref:slis.x) — is authoring syntax, not behaviour: slo: ref:x and slo: x compare equal, matching what identity already does.
  • version blocks compare by declared when present.
  • Deployment/presentation fields are stripped:
endpoints, endpoint, url, address, host, auth,
description, desc, title, summary, annotations,
source, evidence, mcp, default, folder, provider

Buckets

Each layer (L1, L2, L2X, L3, L4, L5, GOV) contains:

Bucket Meaning
onlyInA A declares it; B does not have an in-scope counterpart
onlyInB B has it in a family A participates in; A does not
inBoth same identity on both sides, with aligned or drifted verdict
outOfScope B has it, but A declares nothing in that artefact family
scaffold a placeholder (source: Scaffold) on either side — parked before pairing, never counted; entries carry side (a/b) and artefact
notObserved one pack holds it, in a family the OTHER pack says it had no way to observe — unchecked, never counted; entries carry side, artefact and the other side's reason

outOfScope prevents a single-service drift view from being flooded by the rest of a platform's live inventory. It is reported, but excluded from the in-scope ratios.

scaffold holds the schema-forced placeholders the crawler (crawler.scaffold.<symbol>) or the live fetcher (mcp.scaffold.<symbol>) had to invent. A placeholder is neither a declaration nor live evidence, so it is removed from both sides before identity pairing: a declared burn-rate alert never reads aligned (or drifted) against the live pack's fallback entry — even when both carry the compiler's default windows on the same SLO — and a repo placeholder never reads declared, not live. A real live artefact whose repo counterpart is only a placeholder reads live, not declared as it should. summary.scaffold counts the parked entries.

notObserved separates "the other pack looked and it is not there" from "the other pack could not look". A pack's producer names the artefact families it had no way to observe in observogram.unobserved.<kind> annotations — <kind> is the model's family (dashboard, pipeline_receiver, alert_route, …), the value the reason (the dashboards probe got no answer: HTTP 401, no MCP tool exposes the collector configuration). An artefact with no counterpart whose family the other pack names there goes to notObserved instead of onlyInA / onlyInB: a Grafana that refuses the fetcher's login does not make seven declared dashboards missing. An artefact that DOES have a counterpart is compared as usual, whatever the annotation says. The live fetcher writes these keys (docs/MCP_INTEGRATION.md, What the fetch could not look at); a pack without them behaves exactly as before. summary.notObserved counts the entries; they are outside the union.

A live snapshot read within a scope (rebadge batch 3, C1: metric prefixes, dashboard or rule folder uids) records it per family as observogram.scope.<kind> (tools/lib/live-fetch.mjs scopeAnnotations), and the diff reads those keys beside observogram.unobserved.<kind>. An unmatched artefact on the other side, in a scoped family, goes to notObserved with one of two reasons, each stating only what is known: outside the scope — outside the snapshot's metric scope (prefixes payments_, checkout_), outside the snapshot's dashboard folders (Payments) — or cannot tell, when the artefact does not say what the scope needs (a crawled dashboard names no folder uid): the snapshot read only the folders Payments; Pack A does not say which folder this dashboard is in, so it was not checked. One artefact of a key inside the scope keeps the group compared as usual; an unobserved family keeps its own reason. Without the annotation the answer is byte-identical.

Metric families

Metrics are compared as families, not series. One declared metric is several series on the wire — a histogram x is x_bucket, x_count and x_sum; the Python client adds x_created; a counter declared x is exposed as x_total. Source code declares the metric, a metrics store lists the series, a query references one of them. Paired name for name, a declared histogram read "not live" while its own three series read "live, not declared".

foldMetricFamilies(aItems, bItems) (artefact-model.mjs) folds each side's metric artefacts into one artefact per family before pairing, using what both sides know (metricFamilyResolver):

Series Family When
x_bucket x always — the suffix is reserved for histograms
x_count, x_sum x x_bucket is known, or x is declared a histogram / summary / timer, or both _count and _sum exist
x_total x x is known and declared a counter
x_created what x_total resolves to, else x that counter is known; or x is a distribution or is itself known
anything else itself a lone queue_count gauge is not folded

A family's entry carries series (the names it stands for on that side) and memberIds (the ids of the artefacts it replaces). A metric alone in its family is passed through untouched.

Addresses a source does not state

An alert route's channels are compared by kind and address. Two readers can know a channel exists without being able to state its address: a running Alertmanager prints every secret URL as <secret>, and a repository wires one in at deploy time (url: ${WEBHOOK_URL}). They write redacted:secret and unresolved:<VAR> in its place (tools/lib/alert-routes.mjs). deltasOf treats such a value as partial evidence, like a reference-only expression: when the channel KINDS agree one for one, and every kind where neither side withholds an address agrees on its addresses, the route is aligned. A changed e-mail address beside a redacted webhook is still drift.

Summary Ratios

union = onlyInA + onlyInB + inBoth is the in-scope union. outOfScope is reported separately.

  • jaccard = inBoth / union: identity overlap.
  • alignment = aligned / union: true agreement ratio.

The strongest diagnostic signal is the gap between them: high Jaccard and low alignment means Observogram found the same controls, but their definitions drift.

Weighted Drift Fidelity

The Diagnose view does not score every delta equally. It uses weighted badness:

Bucket Weight Rationale
Declared, not live 1.0 false reassurance: the pack claims production protection that live did not confirm
Drifted 0.5 default miscalibration: same control exists, but fields differ
Drifted decision field 1.0 SLO objective, query, alert window, severity, route/channel, MTTD/MTTR, etc.
Drifted cosmetic field 0.1 label, display, folder, tag, description, etc.
Live, not declared 0.15 shadow signal: useful inventory gap, but less dangerous than false reassurance
Out-of-scope live 0.0 excluded platform inventory
Scaffold 0.0 schema-required fallback with no source evidence

Scaffold is the adapter's projection of a placeholder stamp on either side: crawler.scaffold.<symbol> from the repo crawler, mcp.scaffold.<symbol> from the live fetcher (the spec.otel block, collector receivers/processors, logs/traces exporters, fallback backends, the platform-overview stub, the SEV1 route, baselines, guessed SLI/SLOs and the burn-rate placeholder — see MCP_INTEGRATION.md). Both sides can therefore contribute parked artefacts: diffPacks moves them to the scaffold bucket before pairing (see Buckets), the requirement-chain comparison (comparePackBranches) refuses a Scaffold live node as evidence — a declared node against it reads declared_only, and a placeholder is never a live_only node or an undeclared branch root — and the requirement traceability chain skips a scaffold burn-rate entry when deciding missing_alert_evidence. A live-pack placeholder therefore never weighs in as Live, not declared and a repo scaffold never as Declared, not live on any path: the diff-bucket grade, the requirement-chain grade the studio and the journey CLI score on, and the journey's alignmentPct / declaredNotLive gate facts (isScaffoldDiffEntry still filters older diffs that carry no scaffold bucket).

Weighted fidelity is:

aligned / (aligned + weighted_badness)

The current trusted-fidelity threshold is derived from decision cost: weighted badness must stay below about 0.176 per confirmed artefact, roughly one high-cost false reassurance per six confirmed controls. That maps to about 85% fidelity. The drift-free criterion contributes fractional credit equal to the measured weighted fidelity; crossing the threshold decides whether the row is safe to trust.

Public API

import { keyOf, projectOf, deltasOf, diffPacks } from './tools/lib/diff.mjs';

keyOf(artefact);              // behavioural identity key
projectOf(artefact);          // comparable contract projection
deltasOf(a, b);               // top-level contract deltas
diffPacks(aLayered, bLayered); // directional drift report
diffPacks(aLayered, bLayered, { scopeMode, service, identity }); // scopeMode service | family | all; identity as above

Regression Coverage

tools/test-diff.mjs verifies:

  • pack-vs-self preserves every artefact, including duplicate identity keys;
  • duplicate SEV2 routes survive as distinct controls, and every duplicated identity group is reported once on collisions with its layer and counts;
  • a declared route pairs with the live connector's fabricated placeholder route as channel drift, never as missing-in-live;
  • surplus duplicate controls are reported as drift rather than dropped;
  • empty arrays normalise like absent fields;
  • metric families (each folding rule, and the lone _count that is not folded), notObserved in both directions and outside the union, and an address one side withholds.

tools/test-golden-diff.mjs (npm run test:golden:diff) pins the whole default answer of 27 cases, written before rebadge batch 3 touched the engine, and runs each with identity: 'behaviour' and identity: identityKeyOf too. tools/test-identity-modes.mjs holds the identity modes: every family's material, the three count sets of tools/fixtures/compare-modes/ (behaviour 15 in both / 5 / 5, name 16 / 4 / 4, id 17 / 3 / 3), a custom function, the unnamed and placeholder keys, and the parity of diffPacks over the JSON GET /api/packs/:id answers with GET /api/diff.

tools/test-scan-live.mjs is the end-to-end form: one system described as a repository and as the answers a live MCP gives for it, scanned, drafted and compared — nothing only on one side, nothing drifted — then with one rule, one route and one metric taken away from the live side, each of which shows and only it.

tools/test-packs.mjs additionally asserts the self-diff invariant — diffPacks(pack, pack) preserves every flat-comparable artefact with alignment and Jaccard 1.0 — for every bundled example and reference pack, including the crawled krystaline-repo-carlos.pack.yaml with its real identity collisions. tools/test-traceability-graph.mjs covers duplicate identity keys in the dependency graph and branch comparison.