Skip to content

feat(legal-status): global append-only store for public legal status … - #17

Merged
corlin merged 1 commit into
mainfrom
feat/legal-status-us-ep
Oct 4, 2026
Merged

corlin merged 1 commit into
mainfrom
feat/legal-status-us-ep

Conversation

@corlin

@corlin corlin commented Oct 4, 2026

Copy link
Copy Markdown
Owner

…(ADR 0006) (#13)

Stacked on #11. Merge #11 first; this diff is storage-only.

Design (ADR 0006)

Legal status of a published patent is public and identical for every organization. It's stored once, in two global tables with no organization_id and no tenant RLS. This follows the repo's existing global-table pattern (global_identities), protected by exact grants. The integration suite requires RLS only on tenant tables.

Which organization looks at which patent is itself sensitive (it reveals product direction and competitors). So these tables may contain no requester, case or user field; a test asserts this. Tenant references will live in tenant-scoped FORCE-RLS tables and bind assessment_id plus content_sha256 at sealing time.

Table Contents
legal_status_source_records One row per lookup, including not_found: request_ref (no credentials), retrieved_at, the source's own data time (ODP lastIngestionDateTime), the rule-relevant extract, and the SHA-256, size and storage key of the exact response bytes
legal_status_assessments status, term, countries, evidence, review reasons, rules_version, content_sha256; unique per (source_record_id, as_of, rules_version)
Role Grants
patent_evidence_worker SELECT, INSERT
patent_evidence_app SELECT
patent_evidence_platform none

There are no UPDATE/DELETE grants, so both tables are append-only: a new lookup or a new rules version adds rows and never rewrites history.

  • Raw responses are content-addressed in object storage (public/legal-status/<source>/<sha256>.json). The DB keeps only the fields the rules read.
  • Worker flow (LegalStatusRefresher): fetch → archive exact bytes → record → assess from the stored extract (so any stored assessment can be reproduced from the DB) → record. A source failure writes nothing. not_found records the lookup but no status.

Needs a decision

  • Retention of raw responses. They contain public personal data (inventors, attorneys). Retention is a PIPL/GDPR question for compliance. Until it's decided, raw responses go to object storage only, under a prefix that can be deleted wholesale.
  • Default refresh interval. Not set yet; it should be measured from a first full run's quota use.
  • Not in this PR: job queue and triggering, tenant reference table, API read endpoints.
  • source_data_as_of timezone: ODP gives no UTC offset, so UTC is assumed for the column. The verbatim string in extract is authoritative.

Integration test coverage gap found while doing this (pre-existing, not fixed here)

scripts/test-postgres.sh runs only 5 of the 14 integration files, and CI skips all Postgres tests. When I ran all 14 against Postgres:

  • 6 business-flow tests (cases/documents, features, search, comparison, reports, full e2e) are skipped even under the script: they read PE_MIGRATION_DATABASE_URL, which the script doesn't export. With it set, all 6 fail at seeding: global_identities.display_name is NOT NULL, and their seed data omits it.
  • test_p0_02_e2e_lifecycle errors on a non-existent table, organization_audit_events.
  • 3 test_database_context tests fail when run after other files: an FK error, i.e. an order or seed dependency.

Correction: #6 said its lifecycle test covered the upload checks. Those tests never actually ran. I've added only this PR's new file to the script. Repairing the rest should be its own PR, since it may surface real regressions.

Validation

  • ./scripts/test-postgres.sh: 92 passed (84 plus the 8 new), including the downgrade/upgrade cycle through 0018
  • .venv/bin/pytest apps/api/tests/unit/: 376 passed
  • pnpm test:web: 62 passed; pnpm build:web: ok
  • verify-source-lock, verify-scaffold, verify-p0-02, verify-p0-e2e, docker compose config: ok

🤖 Generated with Claude Code

…(ADR 0006) (#13)

> Stacked on #11. Merge #11 first; this diff is storage-only.

## Design (ADR 0006)

Legal status of a published patent is **public and identical for every
organization**. It's stored once, in two **global** tables with no
`organization_id` and no tenant RLS. This follows the repo's existing
global-table pattern (`global_identities`), protected by exact grants.
The integration suite requires RLS only on tenant tables.

**Which organization looks at which patent is itself sensitive** (it
reveals product direction and competitors). So these tables may contain
no requester, case or user field; a test asserts this. Tenant references
will live in tenant-scoped FORCE-RLS tables and bind `assessment_id`
plus `content_sha256` at sealing time.

| Table | Contents |
|---|---|
| `legal_status_source_records` | One row per lookup, **including
`not_found`**: `request_ref` (no credentials), `retrieved_at`, the
source's own data time (ODP `lastIngestionDateTime`), the rule-relevant
`extract`, and the SHA-256, size and storage key of the **exact response
bytes** |
| `legal_status_assessments` | status, term, countries, evidence, review
reasons, `rules_version`, `content_sha256`; unique per
`(source_record_id, as_of, rules_version)` |

| Role | Grants |
|---|---|
| `patent_evidence_worker` | SELECT, INSERT |
| `patent_evidence_app` | SELECT |
| `patent_evidence_platform` | none |

There are no UPDATE/DELETE grants, so both tables are **append-only**: a
new lookup or a new rules version adds rows and never rewrites history.

- **Raw responses** are content-addressed in object storage
(`public/legal-status/<source>/<sha256>.json`). The DB keeps only the
fields the rules read.
- **Worker flow** (`LegalStatusRefresher`): fetch → archive exact bytes
→ record → **assess from the stored extract** (so any stored assessment
can be reproduced from the DB) → record. A source failure writes
**nothing**. `not_found` records the lookup but no status.

## Needs a decision

- **Retention of raw responses.** They contain public personal data
(inventors, attorneys). Retention is a PIPL/GDPR question for
compliance. Until it's decided, raw responses go to object storage only,
under a prefix that can be deleted wholesale.
- **Default refresh interval.** Not set yet; it should be measured from
a first full run's quota use.
- **Not in this PR:** job queue and triggering, tenant reference table,
API read endpoints.
- **`source_data_as_of` timezone:** ODP gives no UTC offset, so UTC is
*assumed* for the column. The verbatim string in `extract` is
authoritative.

## Integration test coverage gap found while doing this (pre-existing,
not fixed here)

`scripts/test-postgres.sh` runs only **5 of the 14** integration files,
and CI skips all Postgres tests. When I ran all 14 against Postgres:

- **6 business-flow tests** (cases/documents, features, search,
comparison, reports, full e2e) are skipped even under the script: they
read `PE_MIGRATION_DATABASE_URL`, which the script doesn't export. With
it set, **all 6 fail at seeding**: `global_identities.display_name` is
NOT NULL, and their seed data omits it.
- `test_p0_02_e2e_lifecycle` errors on a non-existent table,
`organization_audit_events`.
- 3 `test_database_context` tests fail when run after other files: an FK
error, i.e. an order or seed dependency.

**Correction:** #6 said its lifecycle test covered the upload checks.
Those tests never actually ran. I've added only this PR's new file to
the script. Repairing the rest should be its own PR, since it may
surface real regressions.

## Validation
- `./scripts/test-postgres.sh`: **92 passed** (84 plus the 8 new),
including the downgrade/upgrade cycle through `0018`
- `.venv/bin/pytest apps/api/tests/unit/`: 376 passed
- `pnpm test:web`: 62 passed; `pnpm build:web`: ok
- `verify-source-lock`, `verify-scaffold`, `verify-p0-02`,
`verify-p0-e2e`, `docker compose config`: ok

🤖 Generated with [Claude Code](https://claude.com/claude-code)
@corlin
corlin merged commit ce5ed4e into main Oct 4, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant