Repository navigation
feat(legal-status): global append-only store for public legal status … - #17
Merged
Merged
Conversation
…(ADR 0006) (#13) > Stacked on #11. Merge #11 first; this diff is storage-only. ## Design (ADR 0006) Legal status of a published patent is **public and identical for every organization**. It's stored once, in two **global** tables with no `organization_id` and no tenant RLS. This follows the repo's existing global-table pattern (`global_identities`), protected by exact grants. The integration suite requires RLS only on tenant tables. **Which organization looks at which patent is itself sensitive** (it reveals product direction and competitors). So these tables may contain no requester, case or user field; a test asserts this. Tenant references will live in tenant-scoped FORCE-RLS tables and bind `assessment_id` plus `content_sha256` at sealing time. | Table | Contents | |---|---| | `legal_status_source_records` | One row per lookup, **including `not_found`**: `request_ref` (no credentials), `retrieved_at`, the source's own data time (ODP `lastIngestionDateTime`), the rule-relevant `extract`, and the SHA-256, size and storage key of the **exact response bytes** | | `legal_status_assessments` | status, term, countries, evidence, review reasons, `rules_version`, `content_sha256`; unique per `(source_record_id, as_of, rules_version)` | | Role | Grants | |---|---| | `patent_evidence_worker` | SELECT, INSERT | | `patent_evidence_app` | SELECT | | `patent_evidence_platform` | none | There are no UPDATE/DELETE grants, so both tables are **append-only**: a new lookup or a new rules version adds rows and never rewrites history. - **Raw responses** are content-addressed in object storage (`public/legal-status/<source>/<sha256>.json`). The DB keeps only the fields the rules read. - **Worker flow** (`LegalStatusRefresher`): fetch → archive exact bytes → record → **assess from the stored extract** (so any stored assessment can be reproduced from the DB) → record. A source failure writes **nothing**. `not_found` records the lookup but no status. ## Needs a decision - **Retention of raw responses.** They contain public personal data (inventors, attorneys). Retention is a PIPL/GDPR question for compliance. Until it's decided, raw responses go to object storage only, under a prefix that can be deleted wholesale. - **Default refresh interval.** Not set yet; it should be measured from a first full run's quota use. - **Not in this PR:** job queue and triggering, tenant reference table, API read endpoints. - **`source_data_as_of` timezone:** ODP gives no UTC offset, so UTC is *assumed* for the column. The verbatim string in `extract` is authoritative. ## Integration test coverage gap found while doing this (pre-existing, not fixed here) `scripts/test-postgres.sh` runs only **5 of the 14** integration files, and CI skips all Postgres tests. When I ran all 14 against Postgres: - **6 business-flow tests** (cases/documents, features, search, comparison, reports, full e2e) are skipped even under the script: they read `PE_MIGRATION_DATABASE_URL`, which the script doesn't export. With it set, **all 6 fail at seeding**: `global_identities.display_name` is NOT NULL, and their seed data omits it. - `test_p0_02_e2e_lifecycle` errors on a non-existent table, `organization_audit_events`. - 3 `test_database_context` tests fail when run after other files: an FK error, i.e. an order or seed dependency. **Correction:** #6 said its lifecycle test covered the upload checks. Those tests never actually ran. I've added only this PR's new file to the script. Repairing the rest should be its own PR, since it may surface real regressions. ## Validation - `./scripts/test-postgres.sh`: **92 passed** (84 plus the 8 new), including the downgrade/upgrade cycle through `0018` - `.venv/bin/pytest apps/api/tests/unit/`: 376 passed - `pnpm test:web`: 62 passed; `pnpm build:web`: ok - `verify-source-lock`, `verify-scaffold`, `verify-p0-02`, `verify-p0-e2e`, `docker compose config`: ok 🤖 Generated with [Claude Code](https://claude.com/claude-code)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…(ADR 0006) (#13)
Design (ADR 0006)
Legal status of a published patent is public and identical for every organization. It's stored once, in two global tables with no
organization_idand no tenant RLS. This follows the repo's existing global-table pattern (global_identities), protected by exact grants. The integration suite requires RLS only on tenant tables.Which organization looks at which patent is itself sensitive (it reveals product direction and competitors). So these tables may contain no requester, case or user field; a test asserts this. Tenant references will live in tenant-scoped FORCE-RLS tables and bind
assessment_idpluscontent_sha256at sealing time.legal_status_source_recordsnot_found:request_ref(no credentials),retrieved_at, the source's own data time (ODPlastIngestionDateTime), the rule-relevantextract, and the SHA-256, size and storage key of the exact response byteslegal_status_assessmentsrules_version,content_sha256; unique per(source_record_id, as_of, rules_version)patent_evidence_workerpatent_evidence_apppatent_evidence_platformThere are no UPDATE/DELETE grants, so both tables are append-only: a new lookup or a new rules version adds rows and never rewrites history.
public/legal-status/<source>/<sha256>.json). The DB keeps only the fields the rules read.LegalStatusRefresher): fetch → archive exact bytes → record → assess from the stored extract (so any stored assessment can be reproduced from the DB) → record. A source failure writes nothing.not_foundrecords the lookup but no status.Needs a decision
source_data_as_oftimezone: ODP gives no UTC offset, so UTC is assumed for the column. The verbatim string inextractis authoritative.Integration test coverage gap found while doing this (pre-existing, not fixed here)
scripts/test-postgres.shruns only 5 of the 14 integration files, and CI skips all Postgres tests. When I ran all 14 against Postgres:PE_MIGRATION_DATABASE_URL, which the script doesn't export. With it set, all 6 fail at seeding:global_identities.display_nameis NOT NULL, and their seed data omits it.test_p0_02_e2e_lifecycleerrors on a non-existent table,organization_audit_events.test_database_contexttests fail when run after other files: an FK error, i.e. an order or seed dependency.Correction: #6 said its lifecycle test covered the upload checks. Those tests never actually ran. I've added only this PR's new file to the script. Repairing the rest should be its own PR, since it may surface real regressions.
Validation
./scripts/test-postgres.sh: 92 passed (84 plus the 8 new), including the downgrade/upgrade cycle through0018.venv/bin/pytest apps/api/tests/unit/: 376 passedpnpm test:web: 62 passed;pnpm build:web: okverify-source-lock,verify-scaffold,verify-p0-02,verify-p0-e2e,docker compose config: ok🤖 Generated with Claude Code