Skip to content

feat(corpus): US/EP patent corpus loads (ADR 0007, stage 1 of ADR 0005) - #16

Merged
corlin merged 1 commit into
mainfrom
feat/us-ep-patent-corpus
Oct 4, 2026
Merged

corlin merged 1 commit into
mainfrom
feat/us-ep-patent-corpus

Conversation

@corlin

@corlin corlin commented Oct 4, 2026

Copy link
Copy Markdown
Owner

What

Stage 1 of ADR 0005: the candidate set of US/EP patents that the stage-2 legal-status check (#11, #14) runs over. The design is in ADR 0007, and every scope parameter there is backed by a measurement.

Decision Basis (measured 2026-10-04)
US B1/B2/E1, EP B1/B2/B3, granted, H01M/H02J by CPC OR IPC Reissues (E1, 201) replace the original patent. CPC-only filtering misses IPC-only documents (see the spike report)
US: filed ≥ as_of − 20y − 5y Patent term adjustment is common: 48/59 sampled patents had it, up to 1,220 days. Stage 2 computes the exact expiry
EP: filed ≥ as_of − 20y, no margin The EPC has no term adjustment. BigQuery's EP filing date matched EPO's for 57/57 samples
Cost of the margin +11% rows (US 140,344 → 156,306; EP 48,173 → 53,452)

ADR 0007 also defines a calibration rule for the US margin: after the first full legal-status run, check whether any US patent whose term started 20–25 years ago is still presumed in force.

Storage (ADR 0006 pattern)

  • Global, append-only tables patent_corpus_loads and patent_corpus_members. Worker: SELECT+INSERT; app: SELECT; no UPDATE/DELETE; no tenant columns.
  • Each load records its scope and cutoffs, the exact query and its SHA-256, bytes billed, the source table's last-modified time and the row count, so an FTO analysis can name the corpus it used.
  • A load is written atomically (tested: a failure part-way leaves nothing). The row count is known up front because the tables can't be updated.
  • Inventors are not loaded. Assignee names are, since they're needed for ownership; some may be individuals' names, which are public bibliographic data.

Loader

scripts/load-patent-corpus.py is run by an operator; it's not part of any image.

  • It dry-runs by default (no cost), and the load query dry-runs at 58.88 GB.
  • --execute requires --max-gb, and the script refuses if the scan exceeds it (both guards tested: exit 2 and exit 1).
  • The BigQuery client is an optional ingest extra. It's absent from the API image (verified on the rebuilt image).
  • Credentials come from the operator's Google Cloud ADC; nothing is stored in the repo.

Not done yet

  • No data has been loaded. The first full load runs after this is reviewed (≈59 GB scan; this month's usage so far is about 333 GB of the 1 TB free tier).
  • Legal-status support for US reissues (RE-prefixed numbers) isn't there yet; they're flagged.
  • Next steps: batch legal-status job, then the watch tier (published applications), then margin calibration.

Validation

  • ./scripts/test-postgres.sh: 103 passed (99 + 4 new), including the downgrade/upgrade cycle through 0019
  • .venv/bin/pytest apps/api/tests/unit/: 382 passed (6 new)
  • pnpm test:web: 62 passed; pnpm build:web: ok
  • verify-source-lock, verify-scaffold, verify-p0-02, verify-p0-e2e, docker compose config: ok
  • Loader dry run against BigQuery: 58.88 GB; guard refusals verified

🤖 Generated with Claude Code

Defines and stores the candidate set that stage-2 legal-status checking
runs over. Scope and margins come from measurements:

- US B1/B2/E1 (reissues replace the original) and EP B1/B2/B3, granted,
  H01M/H02J by CPC OR IPC.
- US: filed on or after as_of - 20y - 5y. The 5-year margin covers patent
  term adjustment (measured up to 1,220 days); the exact expiry is computed
  in stage 2. EP: as_of - 20y, no margin, since the EPC has no term
  adjustment and BigQuery's EP filing date matched EPO's for 57/57 samples.
  ADR 0007 defines a calibration rule for the US margin after the first
  full legal-status run.
- Measured size at 2026-10-04: about 156k US and 53k EP publications;
  the load query dry-runs at 58.88 GB.

Storage follows ADR 0006: global, append-only tables
(patent_corpus_loads, patent_corpus_members); worker SELECT+INSERT, app
SELECT, no tenant columns. Each load records its scope, exact query and
SHA-256, bytes billed, source-table modification time and row count, so an
analysis can name the corpus it used. A load is written atomically, and
the row count is known up front because the tables cannot be updated.

scripts/load-patent-corpus.py is operator-run. It dry-runs by default,
--execute requires --max-gb, and it refuses when the scan exceeds the
cap. The BigQuery client is an optional `ingest` extra and is absent from
the API image (verified). Inventors are not loaded.

Tests: 6 unit (cutoffs, deterministic query and hash, literal
validation, row normalization) and 4 Postgres (exact grants, round trip,
failed load leaves nothing, runtime roles cannot update or delete).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@corlin
corlin merged commit 52ea95d into main Oct 4, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant