One YAML per feed, replacing layer-keyed config and three Python tuples

[rubinder]

Closes #3.

The axis was wrong, not the amount of config

The repo already had declarative config — ten monitors in monitors/{bronze,silver,gold}.yaml, validated against a KINDS registry that fails loudly. What it lacked was config organised by feed. A feed was spread across four places, three of them Python:

Was	Held
monitors/{bronze,silver,gold}.yaml	monitor definitions, keyed by layer
graph.py:WATCHED	which table the agent senses
arrival.py:ARRIVAL_SLAS	arrival SLAs, hardcoded tuple
runner.py:TYPED_TABLES	which contracts get column-type checks

feeds/<name>.yaml now holds all of it. Three feeds — yodlee_transactions, stock_prices, and forecast_training_set (a derived feed: a Gold mart has the same operational needs as an ingested one and belongs to neither upstream feed cleanly).

Iceberg schemas stay in Python. They declare field IDs and partition transforms — that's code, and moving it to YAML would be config theatre.

The migration was gated on measurement, not on tests passing

The risk here isn't failing loudly. It's succeeding while quietly changing a threshold or dropping a monitor — with every test green, because the tests were written against the same config that moved.

So the gate came first. Before any code changed, run_monitors, check_arrival and the agent were captured on two warehouses. Two, because a clean one yields 17 ok results and 0 findings, and an all-ok baseline is satisfied by a migration that breaks every check into silence.

It caught a real bug on the first comparison. Dropping a now-unused schemas import left a NameError in runner, and run_monitors's deliberately broad except — which exists so one malformed monitor can't kill a run — turned it into a breach on all seventeen monitors.

Seventeen results appeared. The run exited non-zero. Nothing about that output says "broken" — it reads as a platform correctly reporting that everything is on fire. Only the before/after diff showed every metric was null and every detail read monitor failed: NameError: name 'schemas' is not defined.

After the fix, both captures match exactly: 17 monitor results, 8 arrival checks, and on the drifted warehouse the same 5 findings with the same severities.

The one difference turned out not to be the migration

silver_txn_mean_abs_amount differed at the 13th decimal. Measured rather than assumed — three consecutive runs of identical code:

171.4137026799999
171.41370268000034
171.4137026800003

DuckDB sums in parallel. Relative magnitude ~2.5e-15, irrelevant to the z-score that consumes it — but the daily report was rendering it as a +1.56e-12 change, churning the committed reports on every regeneration. _delta now returns 0 below a 1e-9 relative threshold.

A typo'd kind is now a config error, not a breach

Rejected at load, naming the file, the monitor, and the valid kinds. A breach means your data is bad; a typo means your config is bad, and conflating them sends someone to look at a table.

The runtime breach path is kept, not replaced — a MonitorDef can be built without the loader, and one bad monitor must never take down the other sixteen. It has its own test now.

A second ordering defect this surfaced

Feeds load alphabetically, so the arrival findings changed order and three committed reports churned with identical content. Cause: #6's ordering fix covered the diff sections and missed the verdict, schema, anomaly and incident renderers, which iterated in check-execution order.

Every render path now sorts on a key ending in finding_key or monitor. That defect predated this PR and was invisible because nothing had reordered the checks yet.

What the loader refuses

Each with a message naming the offending key: a monitor on an undeclared layer; a table absent from ALL_TABLES; a missing contract: file; typed: true with no contract (would generate zero type checks and read as coverage); unknown calendar / ingest.mode / feed_type / symbolic row floor; and a duplicate monitor name across feeds — monitor names are the join key to persisted baseline history, so two feeds sharing one would interleave different measurements into a single baseline series.

Verification

• 300 passed (279 before), ruff clean.
• Both golden captures match, clean and drifted.
• Four-day report sequence re-run end to end: identical finding counts (0, 1, 2, 8).
• Reports verified byte-stable across two regenerations.
• Pinned expectations in tests/test_feeds.py are literals, not derived from feeds/*.yaml.

🤖 Generated with Claude Code

https://claude.ai/code/session_0199ePw5w34FyfGAB41mkv3G

[rubinder]

Review round 1 — four findings, all fixed

Probed the shipped loader rather than re-reading the diff.

1. An empty or missing feeds/ directory loaded zero feeds, silently.

This is the failure this repo keeps closing, one level up from where it had been closed before. Every monitor, arrival SLA and agent watch is declared in feeds/*.yaml now — so a deleted or mistyped directory made make monitor print:

monitors: 0 checks at as_of=2026-06-30 — 0 breach, 0 warn

…and exit 0. A platform reporting perfect health because it is looking at nothing.

Worth naming plainly: centralising the config concentrated this risk. Losing one monitors/<layer>.yaml used to lose a third of the checks. Losing feeds/ now loses all of them, plus the arrival SLAs, plus the agent's watch list. load_feeds raises.

2. Two feeds typing the same table minted colliding monitor names.

Column-type monitors are generated as f"{table}_type_{field}", so two feeds with typed: true on one table produce identical names — exactly the baseline corruption the duplicate-monitor-name check exists to prevent, arriving by a route that check could not see, because generated names never appear in feed.monitors. Same for two feeds watching one table, which would raise every finding on it twice.

3. A tables: alias naming an unknown table was unvalidated. It would fail at query time and be swallowed into a breach by run_monitors's broad except — visible, but reported as a data problem when it's a config typo.

4. Two feeds merely naming the same table — left legal. A shared dimension is a real thing; only the collisions that actually corrupt something are rejected.

Verification

• 305 passed (300 before this round, 279 on main), ruff clean.
• Both golden captures re-run after the hardening and still match the pre-migration baseline exactly: 17 monitor results, 8 arrival checks, 0 findings clean / 5 drifted.