Run all three engines against the site daily, and pin the Docker base

[jehrr]

The audit's last two findings. One is taken as offered, one is taken in half — with the declined half argued rather than ignored.

The canary tested one engine of three

Fair, and this takes the audit's second suggested remedy rather than its first.

The offline suite already asserts that the three engines take the same flags (in both directions — closing a documented difference fails too), bind the same shared-module signatures, and share one finish_run, so their exit codes and sidecars cannot drift. What none of that tests is whether they still agree about the site, and only Playwright ran against it on a schedule.

The first remedy on offer was to declare Playwright the only supported engine and demote the others. That contradicts the README and throws away three working engines to close a gap that one job closes.

Compared on ids and schema, not on values. The three runs happen minutes apart and a prediction market reprices continuously, so requiring equal prices would produce a red badge about nothing — and a check that is always red teaches everyone to ignore checks. What must not differ:

• which markets exist (the sku set),
• what the columns are called (the key set, per row),
• every field a market does not change by trading.

That last one is an explicit allowlist, inverted from the obvious blocklist on purpose: with a blocklist, a column added later is compared by default and starts failing the badge the first time the market moves. The allowlist is itself checked against the real columns on each run, or it could quietly shrink to nothing and the step would pass by comparing almost nothing.

data_source is in it deliberately. If one engine fell back to the DOM while its twins read the payload, every other field would still match and only that column would say so.

Verified by running it here before shipping it, which is the part that would otherwise be a workflow nobody has executed:

compared ['playwright', 'puppeteer', 'selenium']: {playwright: 5, selenium: 5, puppeteer: 5}
stable columns actually compared: 22
RESULT: the engines agree
playwright vs selenium,  fields that differ on one row: ['scraped_at']
playwright vs puppeteer, fields that differ on one row: ['scraped_at']

The three engine runs are spelled out rather than looped over a $engine variable. A loop reads better and makes the engine names invisible to anything grepping the file — including the new offline check that asserts the canary still covers every engine the README promises. Three lines are worth a check that can fail.

The job needs no secrets, like the rest of this canary. An engine that gets refused warns and says in the step summary that it did not test anything, rather than failing the badge for the runner's address; fewer than two engines returning rows is reported as "not a pass" rather than as success.

Pinning: the Docker base, yes; lock files, no

Taken. The base image is pinned by digest, with the tag kept beside it so a reader can see which release that digest is. python:3.12-slim is a moving target, so the same Dockerfile built a month apart is a different image — a release and its own rebuild are then not the same artefact, and a regression arriving through the base reads as a regression in this code. Resolved from the tag it replaces, so it is 3.12-slim as of today rather than a version chosen here. Built and run after pinning: 53 markets live from inside the container.

Declined: lock files for the Python dependencies. The audit puts these in the same breath as the Docker pin, and they are not the same trade.

A lock file would stop the daily canary from testing new releases of requests, beautifulsoup4 and the engine libraries. That daily test is the thing that catches a dependency change the morning it happens, rather than whenever someone next bumps a pin. The image is a build artefact and should be reproducible; the canary is a detector and should be exposed to exactly what it is meant to detect. Lower bounds plus a daily live run is the combination that reports breakage — pinning both halves would buy reproducibility by turning the detector off.

The audit's related point about pyproject.toml extras drifting from requirements*.txt is real and is already checked offline; nothing new was needed.

Verification

Four faults planted, suite required to go RED on a named check:

planted fault	caught by
the Docker base back to a moving tag	...pinned by digest, not by a moving tag
the digest kept but the tag dropped	...with the tag kept beside it
selenium dropped from the canary	the canary runs selenium_scraper
the canary comparing values instead of ids	...and compares them on ids rather than on prices

The canary-coverage check is pinned against ENGINES, the same tuple the rest of the suite uses, so dropping an engine from the README and the canary together still fails. It skips only when the whole .github/ directory is absent — the Dockerfile copies none, and this suite runs inside the image at build time — rather than when the file is missing, so a deleted workflow fails rather than passing silently.

🤖 Generated with Claude Code