[Feature]: Ship merchant dictionary updates to running instances via GitHub Releases

[suiramdev]

Problem or use case

The merchant dictionary is a signed static file (~2 MB) that maps merchant names to spending categories. It reaches an instance only at image build time: packages/api/scripts/download-data.ts runs once during docker build (apps/server/Dockerfile), so picking up a new dictionary today means rebuilding the image and redeploying. Instances should be able to fetch new dictionary versions without a full app release.

Two smaller gaps block any client that would try: build-merchant-dictionary.ts signs the artifact with Ed25519 and writes merchants.jsonl.gz.sig, but the tar step in .github/workflows/generate-data.yml packs only merchants.jsonl.gz, so today's release cannot be verified even by a client that checks signatures; and there is no manifest, so a client cannot compare versions without downloading the artifact.

Proposed solution

Keep hosting dictionary artifacts on tagged GitHub Releases of this repo — free, coupled to the release process already in generate-data.yml, no extra infrastructure. Already in place: weekly and push-triggered CI publishing to a data-YYYY-MM-DD release, Ed25519 signing in the build script, and the recategorise route (packages/api/src/routers/budget.ts) that re-evaluates transactions after a dictionary swap.

What still needs to be built:

• Attach merchants.jsonl.gz.sig to the release alongside merchants.jsonl.gz.
• Attach a version manifest JSON giving the current dictionary version, so a client compares versions without downloading ~2 MB.
• A fetch-on-schedule job inside the running instance (daily by default) that checks the latest release via the GitHub API, compares the dictionary version, downloads it, verifies the Ed25519 signature, and swaps it into the running process — distinct from the build-time-only download above.

Alternatives or additional context

• Per-country artifacts (merchants-FR.jsonl.gz and friends) were evaluated and rejected: the worldwide tail every country needs would be duplicated into each file, and generating one artifact per country blows the CI time budget. Consumers filter the single file in memory instead — packages/api/src/categorisation/merchant-scope.ts and the loader's scope protocol in dictionary.ts › loadedScopeCovers. (The original rationale cited ADR-001; docs/adr/ was removed and the durable part of it now lives under Reference data and Dictionary in docs/engineering/categorisation.md.)
• Docs impact: the update mechanism and the new schedule are operator-facing, so apps/fumadocs/content/docs/next/self-hosting/ needs a page update with the change. Two pages state today's gap and must both change: configuration.mdx ("The published merchant data carries no signature") and bank-providers.mdx ("DICTIONARY_PUBLIC_KEY disables the dictionary").

---

Re-verified 2026-09-22 against docs/pin-moving-refs: still open and unchanged in substance. build-merchant-dictionary.ts:869 writes ${OUTPUT_PATH}.sig; the Create release tarball step in generate-data.yml packs place-tokens.json, merchants.jsonl.gz and wikidata-brands.json only — no signature, no manifest; download-data.ts is still reached only through build:data at image build time. Runtime verification exists (categorisation/verify.ts) and fails closed, which is why the unshipped .sig currently makes a set DICTIONARY_PUBLIC_KEY disable the dictionary outright. Only the two links above were stale and are now corrected.