diff --git a/.github/workflows/forecast-archive.yml b/.github/workflows/forecast-archive.yml new file mode 100644 index 0000000..3b9908e --- /dev/null +++ b/.github/workflows/forecast-archive.yml @@ -0,0 +1,52 @@ +name: Forecast archive + +# The forecast track record is only evidence if a round provably existed before +# its outturn did. Everything under forecasts/rounds/ is therefore append-only: +# rounds may be added, never edited or deleted. Enforced here rather than by +# convention, because a silent rewrite is exactly the failure that would destroy +# the claim — and it would look like a routine refresh in review. +on: + pull_request: + push: + branches: [main] + workflow_dispatch: + +jobs: + immutability: + name: Archived rounds are append-only + runs-on: ubuntu-latest + timeout-minutes: 5 + steps: + - uses: actions/checkout@v7 + with: + # Need the base commit to diff against, so no shallow clone. + fetch-depth: 0 + + - uses: actions/setup-python@v6 + with: + python-version: "3.13" + + - name: Reject edits and deletions under forecasts/rounds/ + if: github.event_name == 'pull_request' + env: + BASE: ${{ github.event.pull_request.base.sha }} + HEAD: ${{ github.event.pull_request.head.sha }} + run: | + # --diff-filter=MDR: Modified, Deleted, Renamed. Added (A) is the only + # legitimate change to an archived round. + touched=$(git diff --name-only --diff-filter=MDR "$BASE" "$HEAD" -- forecasts/rounds/ || true) + if [ -n "$touched" ]; then + echo "::error::archived forecast rounds were modified, renamed or deleted:" + echo "$touched" | sed 's/^/ /' + echo "" + echo "Rounds are append-only. To correct a forecast, archive a new" + echo "round — the superseded one stays in the history." + exit 1 + fi + echo "OK — no archived round was edited." + + - name: Validate the archive + run: python3 forecasts/archive.py --check + + - name: Scorecard is current + run: python3 forecasts/score.py --check diff --git a/docs/index.html b/docs/index.html index af13f0f..3c77f18 100644 --- a/docs/index.html +++ b/docs/index.html @@ -58,6 +58,7 @@ Compare Papers Validation + Forecasts
diff --git a/forecasts/README.md b/forecasts/README.md new file mode 100644 index 0000000..5702e5d --- /dev/null +++ b/forecasts/README.md @@ -0,0 +1,58 @@ +# Forecast track record + +A real-time record of every forecast the site publishes: archived with a +timestamp *before* the outturn exists, then scored against realised data. + +This is not the same thing as the pseudo-out-of-sample evaluation in +`papers/boe-svar/figures/rolling_evaluation.json`, which re-estimates the model +at 49 expanding-window origins and compares it with random-walk, drift and AR(1) +baselines. That tests the method; the modeller already knew what happened. This +directory tests the forecasts, in real time, where nobody does. + +## Layout + +| path | mutable? | what it is | +|------|----------|------------| +| `rounds//.json` | **no** | A forecast as it stood on a date. Append-only. | +| `outturns.json` | yes | Realised data, versioned by vintage. | +| `scorecard.json` | generated | Scores. Written by `score.py`, never by hand. | +| `index.html` | partly generated | The public page. Blocks between `` markers are generated. | + +## Running a round + +```bash +# 1. Refresh the model artifact (in the model repo), then archive it here. +python3 forecasts/archive.py + +# 2. Commit the round BEFORE the outturn is published. This is the whole point: +# the git timestamp is what makes the forecast falsifiable. +git add forecasts/rounds && git commit -m "Archive forecast round " + +# 3. When outturns land, add them to outturns.json and rescore. +python3 forecasts/score.py +``` + +Both scripts take `--check` and are wired into +`.github/workflows/forecast-archive.yml`, which also fails any pull request that +modifies, renames or deletes an already-archived round. + +## Rules that are not negotiable + +- **Rounds are append-only.** A forecast built on a mistake is corrected by + archiving a new round, not by editing the old one. The superseded round stays + in the history where it can still be counted against us. +- **Outturns are versioned, not overwritten.** ONS revisions get a new vintage + entry. Scoring always records which vintage it used, so any published score + can be reproduced. +- **No headline accuracy figure until there is one to state.** With zero scored + periods the page says zero. An impressive number derived from one observation + is worse than an empty record, because the empty record is honest. +- **Every forecast is reported beside a naive baseline.** Beating a random walk + is the minimum bar for a model to have earned its complexity. + +## Adding a model + +Add its artifact path to `SOURCES` in `archive.py`. The artifact needs a +`forecast` block keyed `period → variable → {median, lo68, hi68, lo90, hi90}`, +plus `data_edge` and `generated`. Rounds hold one file per model, so models can +join at different dates without disturbing the existing history. diff --git a/forecasts/archive.py b/forecasts/archive.py new file mode 100644 index 0000000..d396084 --- /dev/null +++ b/forecasts/archive.py @@ -0,0 +1,234 @@ +#!/usr/bin/env python3 +"""Archive a generated forecast into the immutable round history. + +Run: python3 forecasts/archive.py # archive the current artifact + python3 forecasts/archive.py --check # exit 1 if the history is inconsistent + python3 forecasts/archive.py --round 2026-07-21 # override the round id + +Why this exists +--------------- +``papers/boe-svar/figures/current_forecast.json`` is a *moving* artifact: each +refresh of the data edge overwrites it in place, so the site always shows one +current forecast and the previous one survives only in git history. That is the +right behaviour for the hero chart and the wrong behaviour for a track record. + +A forecast track record is only evidence if the forecast provably existed before +the outturn did. This script copies the moving artifact into +``forecasts/rounds//forecast.json``, which is **append-only**: rounds +are added, never edited. The commit timestamp on that file is what makes the +claim falsifiable, so the immutability is enforced in CI +(``.github/workflows/forecast-immutability.yml``) rather than by convention. + +Scoring against outturns lives in ``forecasts/score.py`` and reads, but never +writes, the files under ``rounds/``. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import subprocess +import sys +from datetime import datetime, timezone +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] +HERE = Path(__file__).resolve().parent +ROUNDS = HERE / "rounds" + +# The moving artifacts this script snapshots. Adding a model here is how a new +# forecast joins the track record; the round layout is one file per model. +SOURCES = { + "boe-svar": ROOT / "papers" / "boe-svar" / "figures" / "current_forecast.json", +} + +SCHEMA_VERSION = 1 + + +# ---------------------------------------------------------------- helpers + +def sha256(path: Path) -> str: + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def git(*args: str) -> str | None: + """Return git output, or None when git is unavailable or the command fails.""" + try: + out = subprocess.run( + ["git", *args], cwd=ROOT, capture_output=True, text=True, check=True + ) + except (OSError, subprocess.CalledProcessError): + return None + return out.stdout.strip() or None + + +def model_provenance(name: str, raw: dict) -> dict: + """Everything needed to re-run this forecast, taken from the artifact itself. + + We deliberately do not invent fields the generator did not record: a missing + value is stored as null so that a later reader can tell "not captured" from + "captured as zero". + """ + return { + "name": name, + "generated": raw.get("generated"), + "estimation_sample": raw.get("estimation_sample"), + "draws": raw.get("draws"), + "accepted": raw.get("accepted"), + "paths_per_draw": raw.get("paths_per_draw"), + "source_pipeline": raw.get("source"), + } + + +# ---------------------------------------------------------------- archive + +def build_round(name: str, path: Path) -> dict: + raw = json.loads(path.read_text()) + forecast = raw.get("forecast") + if not forecast: + raise SystemExit(f"{path}: no 'forecast' block to archive") + + periods = sorted(forecast) + variables = sorted({v for p in forecast.values() for v in p}) + + return { + "schema_version": SCHEMA_VERSION, + "model": model_provenance(name, raw), + "information_set": { + "data_edge": raw.get("data_edge"), + "forecast_start": raw.get("forecast_start"), + "first_period": periods[0] if periods else None, + "last_period": periods[-1] if periods else None, + }, + "units": raw.get("units"), + "variables": variables, + "provenance": { + "source_path": str(path.relative_to(ROOT)), + "source_sha256": sha256(path), + "site_commit": git("rev-parse", "HEAD"), + }, + # Benchmarks are the point of the exercise: a forecast with no naive + # baseline beside it cannot be judged. boe-svar already computes + # random-walk / drift / AR(1) comparisons pseudo-out-of-sample in + # papers/boe-svar/figures/rolling_evaluation.json; the real-time + # equivalents are filled in by score.py once outturns land. + "benchmarks": {"random_walk": None, "drift": None, "ar1": None, "official": None}, + "forecast": forecast, + } + + +def archive(round_id: str, force: bool = False) -> int: + target = ROUNDS / round_id + written = [] + + for name, path in SOURCES.items(): + if not path.exists(): + print(f"skip {name}: {path} not found", file=sys.stderr) + continue + + out = target / f"{name}.json" + if out.exists() and not force: + # Refusing here is the whole safety property. A round is a claim + # about what was known on a date; silently rewriting it destroys + # the claim. + print( + f"refusing to overwrite {out.relative_to(ROOT)} — " + f"archive a new round id instead", + file=sys.stderr, + ) + return 1 + + payload = build_round(name, path) + payload["round_id"] = round_id + payload["archived_utc"] = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") + + target.mkdir(parents=True, exist_ok=True) + out.write_text(json.dumps(payload, indent=2, sort_keys=False) + "\n") + written.append(out.relative_to(ROOT)) + + if not written: + print("nothing archived", file=sys.stderr) + return 1 + + for w in written: + print(f"archived {w}") + print("\ncommit this round before the outturn is published, or it is not evidence.") + return 0 + + +# ---------------------------------------------------------------- check + +def check() -> int: + """Validate the archived history without touching it.""" + problems = [] + + if not ROUNDS.exists(): + print("no rounds archived yet") + return 0 + + for path in sorted(ROUNDS.glob("*/*.json")): + rel = path.relative_to(ROOT) + try: + data = json.loads(path.read_text()) + except json.JSONDecodeError as exc: + problems.append(f"{rel}: unparseable ({exc})") + continue + + if data.get("schema_version") != SCHEMA_VERSION: + problems.append( + f"{rel}: schema_version {data.get('schema_version')!r} " + f"!= {SCHEMA_VERSION}" + ) + if data.get("round_id") != path.parent.name: + problems.append( + f"{rel}: round_id {data.get('round_id')!r} does not match its directory" + ) + if not data.get("forecast"): + problems.append(f"{rel}: empty forecast block") + if not data.get("information_set", {}).get("data_edge"): + problems.append(f"{rel}: no data edge recorded — the forecast is unfalsifiable") + + for problem in problems: + print(f"FAIL {problem}", file=sys.stderr) + + if problems: + return 1 + + rounds = sorted({p.parent.name for p in ROUNDS.glob("*/*.json")}) + print(f"OK — {len(rounds)} round(s) archived: {', '.join(rounds)}") + return 0 + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__.split("\n")[0]) + ap.add_argument("--check", action="store_true", help="validate the archive, write nothing") + ap.add_argument("--round", help="round id (default: the artifact's generated date)") + ap.add_argument( + "--force", + action="store_true", + help="overwrite an existing round — only for correcting a same-day mistake", + ) + args = ap.parse_args() + + if args.check: + return check() + + round_id = args.round + if not round_id: + # Default to the date the forecast was generated, not today: the round + # is named for the information set, not for when someone got round to + # archiving it. + for path in SOURCES.values(): + if path.exists(): + round_id = json.loads(path.read_text()).get("generated") + break + if not round_id: + print("could not determine a round id; pass --round", file=sys.stderr) + return 1 + + return archive(round_id, force=args.force) + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/forecasts/index.html b/forecasts/index.html new file mode 100644 index 0000000..3d9f5b0 --- /dev/null +++ b/forecasts/index.html @@ -0,0 +1,244 @@ + + + + + +PolicyEngine Macro — forecast track record + + + + + + + + + + + + + + + + + + + + + + +
+ + PolicyEngine Macro + + +
+ + + +
+
+
+

real-time forecast record

+

+ Every forecast, archived before the answer exists. +

+

+ A forecast is only evidence if it was fixed in public before the outturn + was published. Each round below was committed with a timestamp, and is + scored here against realised data — whatever that shows. +

+
+
+ +
+
+ 01 — where this stands +

The record is one round old.

+
+
+ +

+ 1 round archived; 0 forecast period(s) scored. + The earliest period that can be scored is 2026Q2, once the ONS first quarterly estimate lands. +

+ +

+ There is no headline accuracy figure on this page yet, because one + cannot honestly be computed from zero observations. That is the point of + publishing the record from its first day rather than from the day it + starts to look good: the empty state is part of the evidence, and it is + checkable that nothing was quietly discarded before the record began. +

+

+ What already exists is a pseudo-out-of-sample evaluation — the + boe-svar model re-estimated at 49 expanding-window origins and compared + with random-walk, drift and AR(1) baselines. That is a genuine test of + the method and it is reported on the + validation page. It is not the + same thing as a real-time record, because the modeller already knew what + happened. Both are needed; only the second is on this page. +

+
+
+ +
+
+ 02 — the rounds +

What was claimed, and when.

+
+
+
+ + + + + + + + + + + + + + + +
Archived forecast rounds. Each row links to the immutable artifact in the repository.
RoundModelData edgePeriodsScoredArtifact
2026-07-21boe-svar2026Q11302026-07-21/boe-svar.json
+
+

+ Rounds are append-only. Files under forecasts/rounds/ can be + added but never edited or deleted, and a + CI job + fails any pull request that tries. If a forecast turns out to have been + built on a mistake, the correction is a new round; the superseded one + stays in the history where it can still be counted against us. +

+
+
+ +
+
+ 03 — how it is scored +

Four numbers, not one.

+
+
+

+ A single accuracy headline hides the failures that matter, so each round + is scored on four things: +

+
+ + + + + + + + + + + + + + + + + + + + + +
Scoring measures and what each one is there to catch
MeasureWhat it catches
Mean absolute errorTypical size of a miss, in percentage points.
Signed biasWhether the model is persistently high or low. A model that is always 0.4pp too optimistic is a different — and more fixable — problem than one that is 0.4pp out in random directions, and MAE cannot tell them apart.
Band coverageHow often the outturn fell inside the 68% and 90% credible bands. A fan chart whose 68% band catches 30% of outturns is overconfident regardless of how good its central path looks.
Baseline comparisonError relative to a random walk, a random walk with drift, and an AR(1). Beating a naive baseline is the minimum bar for a model to have earned its complexity; a forecast reported without one is marketing.
+
+
+ Why outturns are versioned rather than overwritten +

+ Official data is revised, sometimes substantially and sometimes years + later. A track record that always scores against the latest vintage + quietly rewrites its own history every time the ONS revises a quarter, + and can flatter or punish a forecast that never changed. +

+

+ So forecasts/outturns.json stores the vintage alongside + every observation and appends revisions rather than replacing them. + Each score records which vintage it was computed against, and can be + reproduced exactly. +

+
+
+
+ +
+
+ 04 — what this is not +

Limits worth stating plainly.

+
+
+
    +
  • + The record is short. One round is not a track record, + and will not be one for at least a year. Nothing on this page should be + read as demonstrated forecasting skill until the scored count is well + into double figures. +
    • +
  • + Two variables only. UK GDP growth and CPI inflation. + Bank Rate, unemployment, wage growth and borrowing are not forecast + here yet. +
    • +
  • + The coefficients are not re-estimated each round. The + boe-svar replication stays estimated on the paper's 1992Q1–2023Q2 + sample; only the conditioning data edge moves. This keeps the hosted + model identical to the published replication, at the cost of ignoring + post-2023 information in the coefficients themselves. +
    • +
  • + Rounds are irregular. They follow data-edge refreshes, + not a fixed calendar. A regular cadence is the intended next step, and + until it exists the round spacing is itself a discretionary choice. +
    • +
+
+
+
+ +
+

© 2026 PolicyEngine. All rights reserved.

+ +
+ + + + diff --git a/forecasts/outturns.json b/forecasts/outturns.json new file mode 100644 index 0000000..244a545 --- /dev/null +++ b/forecasts/outturns.json @@ -0,0 +1,29 @@ +{ + "_comment": [ + "Realised outturns used to score the archived forecast rounds. Unlike the", + "files under rounds/, this file IS mutable: outturns get revised, and a", + "track record that ignored revisions would flatter itself.", + "", + "Each observation records the value AND the vintage it was read at, so a", + "score can be reproduced against the data as it stood when scored. When ONS", + "revises a quarter, append a new vintage rather than editing the old one.", + "", + "Units must match the forecast rounds: year-on-year percent." + ], + "units": "year-on-year percent", + "sources": { + "gdp": { + "series": "UK real GDP, year-on-year growth", + "publisher": "ONS", + "dataset": "IHYQ (quarterly GDP), derived to y/y", + "typical_release_lag": "~6 weeks after quarter end (first estimate)" + }, + "cpi": { + "series": "UK CPI, year-on-year inflation", + "publisher": "ONS", + "dataset": "D7G7 (CPI 12-month rate), quarterly average", + "typical_release_lag": "~3 weeks after month end" + } + }, + "observations": [] +} diff --git a/forecasts/rounds/2026-07-21/boe-svar.json b/forecasts/rounds/2026-07-21/boe-svar.json new file mode 100644 index 0000000..8cf7004 --- /dev/null +++ b/forecasts/rounds/2026-07-21/boe-svar.json @@ -0,0 +1,246 @@ +{ + "schema_version": 1, + "model": { + "name": "boe-svar", + "generated": "2026-07-21", + "estimation_sample": "1992Q1-2023Q2", + "draws": 3000, + "accepted": 244, + "paths_per_draw": 5, + "source_pipeline": "PolicyEngine/boe-var-model public-data pipeline" + }, + "information_set": { + "data_edge": "2026Q1", + "forecast_start": "2026Q2", + "first_period": "2026Q2", + "last_period": "2029Q2" + }, + "units": "year-on-year percent", + "variables": [ + "cpi", + "gdp" + ], + "provenance": { + "source_path": "papers/boe-svar/figures/current_forecast.json", + "source_sha256": "991ba7fcf99a74a971486cb7c286ea5f41e718727b8a21e6f459dbd74e639dfd", + "site_commit": "a0fd90e0eeb81c4052480f403e3eb99b2c75add0" + }, + "benchmarks": { + "random_walk": null, + "drift": null, + "ar1": null, + "official": null + }, + "forecast": { + "2026Q2": { + "gdp": { + "median": 1.1042806220419834, + "lo68": 0.5510003372254778, + "hi68": 1.666980716779398, + "lo90": 0.192771840943567, + "hi90": 1.9694943386558064 + }, + "cpi": { + "median": 2.6785302332498873, + "lo68": 2.2365556858919784, + "hi68": 3.157965840529223, + "lo90": 1.9438275074056066, + "hi90": 3.4475946819537198 + } + }, + "2026Q3": { + "gdp": { + "median": 1.3627023330971042, + "lo68": 0.5272867982988555, + "hi68": 2.2330482527494495, + "lo90": -0.09097516145835696, + "hi90": 2.695983170703277 + }, + "cpi": { + "median": 2.7100196114542143, + "lo68": 1.9468065493598739, + "hi68": 3.454373485118917, + "lo90": 1.4235438631893544, + "hi90": 3.952790079190609 + } + }, + "2026Q4": { + "gdp": { + "median": 1.7206449996464244, + "lo68": 0.5640516087859759, + "hi68": 2.6752703163401566, + "lo90": -0.08919263441314337, + "hi90": 3.4093599820890286 + }, + "cpi": { + "median": 3.1217836392592346, + "lo68": 2.093186295824305, + "hi68": 4.13741930352289, + "lo90": 1.4470185532503763, + "hi90": 4.80874055477333 + } + }, + "2027Q1": { + "gdp": { + "median": 1.3584582389772777, + "lo68": 0.0785751905859615, + "hi68": 2.649498782598721, + "lo90": -0.6891402283870889, + "hi90": 3.4589474859415645 + }, + "cpi": { + "median": 3.120929372038205, + "lo68": 1.8789440496650696, + "hi68": 4.485145414122816, + "lo90": 1.019887246715544, + "hi90": 5.4226891721669315 + } + }, + "2027Q2": { + "gdp": { + "median": 1.372528189906916, + "lo68": 0.1073056110082052, + "hi68": 2.6098019390803433, + "lo90": -0.7794551272519357, + "hi90": 3.3970577458050566 + }, + "cpi": { + "median": 3.186829432874049, + "lo68": 1.8957205541971642, + "hi68": 4.623112577059112, + "lo90": 1.0228534909516043, + "hi90": 5.442621131974212 + } + }, + "2027Q3": { + "gdp": { + "median": 1.3741812433750056, + "lo68": 0.20888691880131768, + "hi68": 2.6123877085203913, + "lo90": -0.739445041831243, + "hi90": 3.402615545928813 + }, + "cpi": { + "median": 3.286472247204131, + "lo68": 1.8491208101832945, + "hi68": 4.583197807574238, + "lo90": 0.9476328593437444, + "hi90": 5.530470455714035 + } + }, + "2027Q4": { + "gdp": { + "median": 1.3788291610100032, + "lo68": 0.07707091251812004, + "hi68": 2.6145430071947335, + "lo90": -0.818889753959354, + "hi90": 3.514046800670666 + }, + "cpi": { + "median": 3.233280752902914, + "lo68": 1.9275470047493968, + "hi68": 4.606632222714463, + "lo90": 0.9724774234388264, + "hi90": 5.617574521157974 + } + }, + "2028Q1": { + "gdp": { + "median": 1.3612007855628008, + "lo68": 0.040908502280317394, + "hi68": 2.7196865968268944, + "lo90": -0.8149122918822399, + "hi90": 3.6053271080927174 + }, + "cpi": { + "median": 3.199227340647383, + "lo68": 1.9263048248901191, + "hi68": 4.6192032406553745, + "lo90": 1.0070077128151098, + "hi90": 5.672525656548529 + } + }, + "2028Q2": { + "gdp": { + "median": 1.4232973096083015, + "lo68": 0.02971837690303795, + "hi68": 2.7788561402295446, + "lo90": -0.945664351949108, + "hi90": 3.760689229226114 + }, + "cpi": { + "median": 3.2461848029472833, + "lo68": 1.857251342463553, + "hi68": 4.70901771136505, + "lo90": 0.8719395499457051, + "hi90": 5.769908458335319 + } + }, + "2028Q3": { + "gdp": { + "median": 1.3862398719609246, + "lo68": -0.039938998523875884, + "hi68": 2.881827038761048, + "lo90": -0.9703271441583865, + "hi90": 3.7916122638919085 + }, + "cpi": { + "median": 3.2697622292784843, + "lo68": 1.8114882175335787, + "hi68": 4.663177973090492, + "lo90": 0.8622969156736389, + "hi90": 5.796652989372211 + } + }, + "2028Q4": { + "gdp": { + "median": 1.4024113713616089, + "lo68": 0.006471832729248397, + "hi68": 2.872887292828145, + "lo90": -0.8753656320372443, + "hi90": 3.8417180304912204 + }, + "cpi": { + "median": 3.2530823211542383, + "lo68": 1.820698795397609, + "hi68": 4.708486865741367, + "lo90": 0.7416752264597022, + "hi90": 5.7761731576089455 + } + }, + "2029Q1": { + "gdp": { + "median": 1.4285381419904297, + "lo68": 0.08268986329663669, + "hi68": 2.9100354344064088, + "lo90": -0.7947368100296558, + "hi90": 3.7695962026524397 + }, + "cpi": { + "median": 3.24932237244613, + "lo68": 1.7052049954798394, + "hi68": 4.808793720145998, + "lo90": 0.732531942957172, + "hi90": 5.8642376750134755 + } + }, + "2029Q2": { + "gdp": { + "median": 1.4192438805027905, + "lo68": -0.020919092181202376, + "hi68": 2.8420198671654178, + "lo90": -0.9257141382192344, + "hi90": 3.8548901797231223 + }, + "cpi": { + "median": 3.2479024354527724, + "lo68": 1.7106972263954299, + "hi68": 4.869224390247395, + "lo90": 0.7601567975992538, + "hi90": 5.887338128157577 + } + } + }, + "round_id": "2026-07-21", + "archived_utc": "2026-07-25T01:08:19Z" +} diff --git a/forecasts/score.py b/forecasts/score.py new file mode 100644 index 0000000..ac332cb --- /dev/null +++ b/forecasts/score.py @@ -0,0 +1,251 @@ +#!/usr/bin/env python3 +"""Score the archived forecast rounds against realised outturns. + +Run: python3 forecasts/score.py # rebuild forecasts/scorecard.json + python3 forecasts/score.py --check # exit 1 if scorecard.json is stale + +Reads ``rounds/*/*.json`` (immutable) and ``outturns.json`` (mutable, revised), +and writes ``scorecard.json``, which the page builder renders. This script never +writes under ``rounds/``. + +What it deliberately does not do +-------------------------------- +It does not compute a headline accuracy number until there is something to +compute one from. With no scored periods the scorecard reports zero, states the +first date on which a score becomes possible, and the page says so plainly. A +track record that reports an impressive figure derived from one observation is +worse than an empty one, because the empty one is honest. + +Errors are signed (forecast minus outturn) as well as absolute, because a model +that is always 0.4pp high is a different problem from one that is 0.4pp off in +random directions, and the mean absolute error hides the difference. +""" + +from __future__ import annotations + +import argparse +import json +import re +import sys +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] +HERE = Path(__file__).resolve().parent +ROUNDS = HERE / "rounds" +OUTTURNS = HERE / "outturns.json" +SCORECARD = HERE / "scorecard.json" +PAGE = HERE / "index.html" + +REPO_BLOB = "https://github.com/PolicyEngine/macro/blob/main" + + +def load_rounds() -> list[dict]: + rounds = [] + for path in sorted(ROUNDS.glob("*/*.json")): + data = json.loads(path.read_text()) + data["_path"] = str(path.relative_to(ROOT)) + rounds.append(data) + return rounds + + +def latest_outturns(observations: list[dict]) -> dict[tuple[str, str], dict]: + """Latest vintage per (period, variable). Later vintages supersede earlier.""" + best: dict[tuple[str, str], dict] = {} + for obs in observations: + key = (obs["period"], obs["variable"]) + current = best.get(key) + if current is None or obs.get("vintage", "") >= current.get("vintage", ""): + best[key] = obs + return best + + +def score_round(rnd: dict, outturns: dict[tuple[str, str], dict]) -> dict: + entries = [] + for period, block in sorted(rnd["forecast"].items()): + for variable, stats in sorted(block.items()): + obs = outturns.get((period, variable)) + if obs is None: + continue + + point = stats["median"] + actual = obs["value"] + error = point - actual + + entries.append( + { + "period": period, + "variable": variable, + "forecast": point, + "outturn": actual, + "outturn_vintage": obs.get("vintage"), + "error": error, + "abs_error": abs(error), + # Band coverage is the honest test of a fan chart: a model + # whose 68% band contains the outturn 68% of the time is + # calibrated, however large its point errors are. + "in_68": stats["lo68"] <= actual <= stats["hi68"], + "in_90": stats["lo90"] <= actual <= stats["hi90"], + } + ) + + by_variable: dict[str, dict] = {} + for variable in sorted({e["variable"] for e in entries}): + rows = [e for e in entries if e["variable"] == variable] + by_variable[variable] = { + "n": len(rows), + "mae": sum(r["abs_error"] for r in rows) / len(rows), + "bias": sum(r["error"] for r in rows) / len(rows), + "coverage_68": sum(r["in_68"] for r in rows) / len(rows), + "coverage_90": sum(r["in_90"] for r in rows) / len(rows), + } + + return { + "round_id": rnd["round_id"], + "model": rnd["model"]["name"], + "data_edge": rnd["information_set"]["data_edge"], + "archived_utc": rnd.get("archived_utc"), + "path": rnd["_path"], + "periods_forecast": len(rnd["forecast"]), + "periods_scored": len({e["period"] for e in entries}), + "entries": entries, + "by_variable": by_variable, + } + + +def build() -> dict: + rounds = load_rounds() + outturn_doc = json.loads(OUTTURNS.read_text()) + outturns = latest_outturns(outturn_doc.get("observations", [])) + + scored = [score_round(r, outturns) for r in rounds] + total_scored = sum(s["periods_scored"] for s in scored) + + # The first period any round forecast that is still unscored — i.e. the + # earliest date this track record can start meaning something. + pending = sorted( + { + period + for r in rounds + for period in r["forecast"] + if not any((period, v) in outturns for v in r["forecast"][period]) + } + ) + + return { + "_comment": "Generated by forecasts/score.py — do not edit by hand.", + "rounds": len(rounds), + "periods_scored": total_scored, + "next_period_to_score": pending[0] if pending else None, + "status": ( + "accumulating — no forecast period has an outturn yet" + if total_scored == 0 + else f"{total_scored} forecast period(s) scored" + ), + "detail": scored, + } + + +# ---------------------------------------------------------------- page + +def esc(s: str) -> str: + return str(s).replace("&", "&").replace("<", "<").replace(">", ">") + + +def render_status(card: dict) -> str: + """The headline paragraph. Deliberately refuses to state an accuracy figure + while the scored count is zero — see the module docstring.""" + rounds = card["rounds"] + scored = card["periods_scored"] + plural = "" if rounds == 1 else "s" + + lines = [ + "

", + f" {rounds} round{plural} archived; {scored} forecast " + "period(s) scored.", + ] + if card["next_period_to_score"]: + lines.append( + " The earliest period that can be scored is " + f"{esc(card['next_period_to_score'])}, once the " + "ONS first quarterly estimate lands." + ) + else: + lines.append(" Every archived period now has an outturn.") + lines.append("

") + return "\n".join(lines) + + +def render_rounds(card: dict) -> str: + rows = [] + for detail in card["detail"]: + href = f"{REPO_BLOB}/{detail['path']}" + label = "/".join(Path(detail["path"]).parts[-2:]) + rows.append( + " \n" + f" {esc(detail['round_id'])}\n" + f" {esc(detail['model'])}\n" + f" {esc(detail['data_edge'])}\n" + f" {detail['periods_forecast']}\n" + f" {detail['periods_scored']}\n" + f" {esc(label)}\n" + " " + ) + return "\n".join(rows) + + +def render_page(html: str, card: dict) -> str: + """Inject the generated blocks between their markers. + + Same contract as validation/figures/make_charts.py: the page is committed, + the numbers inside it are generated, and --check fails if they drift apart. + """ + blocks = {"scorecard-status": render_status(card), "scorecard-rounds": render_rounds(card)} + for marker, body in blocks.items(): + pattern = re.compile( + rf"(\n).*?()", re.DOTALL + ) + if not pattern.search(html): + raise SystemExit(f"{PAGE}: marker {marker} not found") + html = pattern.sub(lambda m: m.group(1) + body + "\n" + m.group(2), html, count=1) + return html + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__.split("\n")[0]) + ap.add_argument( + "--check", action="store_true", help="exit 1 if scorecard.json or the page is stale" + ) + args = ap.parse_args() + + fresh = build() + rendered = json.dumps(fresh, indent=2) + "\n" + page_now = PAGE.read_text() + page_fresh = render_page(page_now, fresh) + + if args.check: + stale = [] + if not SCORECARD.exists(): + stale.append("scorecard.json missing") + elif SCORECARD.read_text() != rendered: + stale.append("scorecard.json is stale") + if page_now != page_fresh: + stale.append("forecasts/index.html is stale") + if stale: + for item in stale: + print(f"FAIL {item} — run forecasts/score.py", file=sys.stderr) + return 1 + print(f"OK — scorecard and page current ({fresh['status']})") + return 0 + + SCORECARD.write_text(rendered) + if page_now != page_fresh: + PAGE.write_text(page_fresh) + print(f"updated {PAGE.relative_to(ROOT)}") + print(f"wrote {SCORECARD.relative_to(ROOT)} — {fresh['status']}") + if fresh["next_period_to_score"]: + print(f"next period to score: {fresh['next_period_to_score']}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/forecasts/scorecard.json b/forecasts/scorecard.json new file mode 100644 index 0000000..5ced444 --- /dev/null +++ b/forecasts/scorecard.json @@ -0,0 +1,20 @@ +{ + "_comment": "Generated by forecasts/score.py \u2014 do not edit by hand.", + "rounds": 1, + "periods_scored": 0, + "next_period_to_score": "2026Q2", + "status": "accumulating \u2014 no forecast period has an outturn yet", + "detail": [ + { + "round_id": "2026-07-21", + "model": "boe-svar", + "data_edge": "2026Q1", + "archived_utc": "2026-07-25T01:08:19Z", + "path": "forecasts/rounds/2026-07-21/boe-svar.json", + "periods_forecast": 13, + "periods_scored": 0, + "entries": [], + "by_variable": {} + } + ] +} diff --git a/frb-us/index.html b/frb-us/index.html index 9606908..75364e3 100644 --- a/frb-us/index.html +++ b/frb-us/index.html @@ -47,6 +47,7 @@ Compare Papers Validation + Forecasts
diff --git a/models/index.html b/models/index.html index a1ae397..14d12c8 100644 --- a/models/index.html +++ b/models/index.html @@ -26,7 +26,7 @@
diff --git a/obr/index.html b/obr/index.html index bf1fa3a..9519e9d 100644 --- a/obr/index.html +++ b/obr/index.html @@ -47,6 +47,7 @@ Compare Papers Validation + Forecasts
diff --git a/olg/index.html b/olg/index.html index bbdfdff..834ed85 100644 --- a/olg/index.html +++ b/olg/index.html @@ -47,6 +47,7 @@ Compare Papers Validation + Forecasts
diff --git a/papers/boe-svar/index.html b/papers/boe-svar/index.html index 5b59d2b..85d8cf1 100644 --- a/papers/boe-svar/index.html +++ b/papers/boe-svar/index.html @@ -43,6 +43,7 @@ Compare Papers Validation + Forecasts
diff --git a/papers/frb-us/index.html b/papers/frb-us/index.html index abcff5d..e66352e 100644 --- a/papers/frb-us/index.html +++ b/papers/frb-us/index.html @@ -43,6 +43,7 @@ Compare Papers Validation + Forecasts
diff --git a/papers/index.html b/papers/index.html index a9ccf08..9737b7c 100644 --- a/papers/index.html +++ b/papers/index.html @@ -46,6 +46,7 @@ Compare Papers Validation + Forecasts
diff --git a/papers/obr-macro/index.html b/papers/obr-macro/index.html index ce56f98..69f79d3 100644 --- a/papers/obr-macro/index.html +++ b/papers/obr-macro/index.html @@ -43,6 +43,7 @@ Compare Papers Validation + Forecasts
diff --git a/papers/psl-og/index.html b/papers/psl-og/index.html index 2eb5549..a3edc17 100644 --- a/papers/psl-og/index.html +++ b/papers/psl-og/index.html @@ -43,6 +43,7 @@ Compare Papers Validation + Forecasts
diff --git a/pe/index.html b/pe/index.html index b294a48..a1416a0 100644 --- a/pe/index.html +++ b/pe/index.html @@ -47,6 +47,7 @@ Compare Papers Validation + Forecasts
diff --git a/sitemap.xml b/sitemap.xml index 712bf21..14cd19f 100644 --- a/sitemap.xml +++ b/sitemap.xml @@ -10,6 +10,7 @@ https://policyengine-macro.vercel.app/docs0.9 https://policyengine-macro.vercel.app/papers0.9 https://policyengine-macro.vercel.app/validation0.9 + https://policyengine-macro.vercel.app/forecasts0.9 https://policyengine-macro.vercel.app/connect0.8 https://policyengine-macro.vercel.app/contact0.8 https://policyengine-macro.vercel.app/obr0.8 diff --git a/svar/index.html b/svar/index.html index b7c984a..fe8195f 100644 --- a/svar/index.html +++ b/svar/index.html @@ -47,6 +47,7 @@ Compare Papers Validation + Forecasts
diff --git a/validation/index.html b/validation/index.html index e0e7879..ac9bdb8 100644 --- a/validation/index.html +++ b/validation/index.html @@ -61,6 +61,7 @@ Compare Papers Validation + Forecasts