Score the first forecast against real ONS data#93
Open
vahid-ahmadi wants to merge 4 commits into
Open
Conversation
The site could show a current forecast but not a record of past ones: current_forecast.json is overwritten on every data-edge refresh, so a superseded forecast survived only in git history. That is right for the hero chart and wrong for evidence — a forecast only counts if it provably existed before the outturn did. forecasts/ archives each round immutably and scores it once outturns land: - archive.py snapshots the moving artifact into rounds/<date>/<model>.json, refusing to overwrite an existing round. - score.py scores rounds against versioned outturns on four measures — MAE, signed bias, 68/90% band coverage, and naive baselines — and generates both scorecard.json and the marked blocks of the page, with --check for staleness. - forecast-archive.yml fails any PR that modifies, renames or deletes an archived round. The page states the empty record plainly (1 round, 0 scored, first scorable period 2026Q2) rather than deriving a headline accuracy figure from nothing. It also distinguishes this from the existing pseudo-out-of-sample evaluation, which tests the method but was run knowing the answer. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Everything on the site was pinned to published forecast vintages with no live official data behind it, so there was no way to score a forecast, and no way to tell whether an analysis had used data that existed at the time. data/fetch.py pulls three ONS series (real GDP ABMI, CPI annual rate D7G7, unemployment rate MGSX) and writes a dated snapshot per fetch. Nothing is ever edited in place: an upstream revision arrives as a new file beside the old one, so revisions are visible and any past number stays reproducible. - A fetch finding no upstream change writes nothing, so real revisions are not buried in a stream of identical files. Network failures are reported as failures, never as "no revision". - latest/ is a flattened copy for the page builders. The site is static under a CSP with connect-src 'self', so the browser cannot call the ONS; anything shown has to be baked in at build time. - JSON rather than Parquet: these series are small, and a git-diffable format makes a revision a reviewable diff instead of an opaque blob. - fetch-vintages.yml runs weekday mornings after the release window and opens a PR rather than pushing. Verified against the live API: 285 quarters of GDP to 2026Q1, 150 of CPI to 2026Q2 (2.8%), 221 of unemployment to 2026Q1. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…first-forecast-score
Stacked on the forecast archive and the data vintage store: this connects them. forecasts/ingest_outturns.py maps the forecast variables onto the ONS series (CPI direct; GDP converted from the chained-volume level to year-on-year), and appends outturns tagged with the data vintage they were read from. Only periods some archived round actually forecast are ingested — this is evidence for the track record, not a general data mirror. First result: 2026Q2 CPI, forecast 2.68% against an outturn of 2.80%. Error -0.12pp, inside the 68% credible band. 2026Q2 GDP is still pending; the "next result due" logic now checks per variable rather than per period, since CPI lands about six weeks ahead of the quarterly GDP estimate. The page records, rather than glosses, the weakness of this first data point: the 2026-07-21 round was archived retroactively out of current_forecast.json, so its provenance rests on that file's git history (committed 2026-07-21 14:54 BST) rather than on the archive, and the ONS published 2026Q2 CPI the following morning. A sixteen-hour lead is a real lead but thin evidence, and saying so is cheaper than having a reader find it. CI now also fails when an outturn is available upstream but not recorded, so the record cannot silently stop being scored. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #90 (forecast archive) and #92 (data vintage store) — this is the piece that connects them. Merge those two first.
What
forecasts/ingest_outturns.pymaps forecast variables onto the ONS series (CPI direct; GDP converted from the chained-volume level to year-on-year) and appends outturns tagged with the vintage they were read from. Only periods some archived round actually forecast are ingested — this file is evidence for the track record, not a general data mirror.First result
2026Q2 GDP is still pending. The "next result due" logic now checks per variable rather than per period, since CPI lands roughly six weeks ahead of the quarterly GDP estimate — otherwise a half-scored period looks finished.
The caveat, stated on the page rather than buried
The
2026-07-21round was archived retroactively out ofcurrent_forecast.json, so its provenance rests on that file's git history rather than on the archive machinery. That commit landed 2026-07-21 14:54 BST; the ONS published 2026Q2 CPI on the following morning's release. So the forecast did precede the outturn, by about sixteen hours, with the model conditioned only through 2026Q1 — but a sixteen-hour lead is not the same evidence as a round archived weeks ahead. The page says exactly this. Every round from here is archived before the fact.CI
forecast-archive.ymlnow also fails when an outturn is available upstream but unrecorded, so the record cannot silently stop being scored.🤖 Generated with Claude Code