Repository navigation
Commit 5d0e4e2
fix(metadata-protocol)!: one stored-metadata filter collector and search narrowing, owned by the door and called by the reader seam; cross-field and deep family reads refused (#21619)
Fixes #21544
Clause-②: yes (narrowing)
The generic data door now owns the stored-metadata family's one
filter-field collector and its one default-search narrowing. Both are
exported module functions, and the door and the in-process
reader-context seam (`@objectstack/runtime`) call the same two. The
card's measure-first step found a door gap: two filter shapes read a
family column at the generic data door without the family's refusal ever
seeing them. Per the ruling, the door therefore adopts the stricter
collector in this landing, and under the card's raise rule the card is
p1. Measurement below.
## The door-gap measurement (measured before any collector was chosen)
Measured at `b610eabf72` with a composed kernel (ObjectQL, a default
datasource, HTTP server, platform objects, auth, security, sharing, REST
and the dispatcher) and the administrator signed in. The family
credential was stored by the production writer (`PUT
/meta/datasource/NAME`). Every request went through three doors: `POST
/api/v1/data/OBJECT/query` (HTTP), the in-process door
(`protocol.findData`) and the seam (`serveStoredMetadataReadsThrough`
over the engine). Both family tables were covered, on
`driver-sqlite-wasm` and on `driver-memory`.
| shape at the door | request (body of `POST
/api/v1/data/sys_metadata/query`) | column read | answer before this PR
|
|---|---|---|---|
| positive control: direct reference |
`{"where":{"metadata":{"$ne":"zzz"}}}` | `metadata` | 400
`INVALID_FIELD` (family refusal) |
| **cross-field comparand** |
`{"where":{"type":"datasource","name":{"$ne":{"$field":"metadata"}}}}` |
`metadata` | **200, 1 row**; the `$eq` twin answers 0 rows; `type` `$lt`
`$field metadata` answers every row and `$gt` answers none (SQL). The
family column is evaluated. |
| **cross-field comparand** | same, with `checksum` (and
`previous_checksum` / `change_note` on `sys_metadata_history`) | the
hash column / the note | **200**, partitioned the same way (SQL) |
| **cross-field comparand in an aggregation filter** |
`{"groupBy":["type"],"aggregations":[{"function":"count","alias":"n","filter":{"name":{"$ne":{"$field":"metadata"}}}}]}`
| `metadata` | **200**, `n` = 1 (the `$eq` twin gives `n` = 0) |
| **direct predicate below the depth backstop** | the body filter
`{"metadata":{"$contains":"STORED_CREDENTIAL"}}` wrapped in 33 nested
one-armed `$and` levels | `metadata` | **200, the row** for the stored
credential, **0 rows** for a wrong guess, **the row** for a prefix. A
credential oracle on both drivers, both tables, over HTTP and
in-process, in `where` and in an aggregation filter. At 32 levels the
same filter is refused (control). |
| dotted key | `{"where":{"metadata.x":"zzz"}}` | none | 400
`INVALID_FIELD` from the door's dotted-path verdict (`param: where`).
Moot: never evaluated. |
| `having` | `having` naming `metadata` / `checksum`, or `{ "$field": …
}` to one | none | 400 `INVALID_FILTER`: the engine judges every
`having` name against the aggregated row's columns. Grouping by a family
column is refused, and `min` / `max` of the `textarea` / `text` family
columns is refused by the engine's aggregate-field-type door. Moot. |
The two bold shapes are the door gap. The cross-field one is a
SQL-driver reach. On `driver-memory` the reference is not resolved at
all (see Acceptance notes). The depth one is a reach on both drivers.
## What this changes
### Public surface: `@objectstack/metadata-protocol` (additive, `minor`)
- `collectStoredMetadataFilterFields(object, query): string[]` is the
family's one filter-field collector. It returns every column a read
query's filters read, across `where`, the engine's `filter` alias and
each `aggregations[i].filter`. That means each key's head plus each
cross-field `{ $field }` comparand's head, at any depth: the walk is
iterative and cycle-safe, with no depth backstop. Anything beneath an
unrecognised `$` key is read as a condition, and a `FilterArray` is
lowered first. It is `[]` outside the family. It does **not** read
`having`, whose names are the aggregated row's.
- `narrowStoredMetadataSearch(object, query, schema, wireSpelling?):
string[] | undefined` is the family's one default-search narrowing. It
moved, unchanged in its answers, from the door's private method of the
same name:
- an explicit list naming the body or a hash column is refused under the
caller's wire spelling;
- a default search returns the narrowed field set for the caller to run
as `searchFields`;
- an emptied set is refused;
- `undefined` means "run as is".
- `type StoredMetadataSearchSchema` is the definition slice the
narrowing reads.
### Accept-set changes: the generic data door, `sys_metadata` /
`sys_metadata_history` only
This applies to `GET /api/v1/data/:object`, `POST
/api/v1/data/:object/query` and in-process `findData`.
1. A cross-field comparand naming the body, `checksum`,
`previous_checksum` or `change_note` changes from **200 to 400
`INVALID_FIELD`**, with `param: filter` and `field` set to the column,
before the engine is asked. This covers `where`, `$not` and an
aggregation filter.
2. A family-column predicate, key or comparand, nested more than 32
combinators deep changes from **200 to 400 `INVALID_FIELD`**, in the
same envelope.
3. Some shapes were refused before and stay refused, with a different
refusal, in-process only. An unrecognised `$` key wrapping a family
column, an array-form aggregation filter naming one, and a malformed
`$field` reference to one were the engine's `INVALID_FILTER`. They now
get the family's `INVALID_FIELD`. Over HTTP, the array-form aggregation
filter is still refused earlier by REST validation.
Unchanged:
- **Every other object.** The collector answers `[]` outside the family,
so the door collects nothing there.
- **Dotted keys.** The dotted-path verdict still answers first.
- **Every search answer** (explicit list, default narrowing, emptied
set). These are pinned identical at the door and the seam.
### `@objectstack/runtime` (`patch`)
The seam (`stored-metadata-reader-seam.ts`) changes as follows:
- `narrowFamilySearch` is **deleted**, along with the `filterHeadFields`
wrapper and the `@objectstack/plugin-security` `collectConditionFields`
import. The seam now calls the door's two exports.
- `count` runs the query the guard returns (H2).
The seam's accept set is unchanged: everything it refused before it
still refuses. Two seam refusals change code, from the engine's
`INVALID_FILTER` to the family's `INVALID_FIELD`: an unrecognised `$`
key wrapping a family column, and a malformed `$field` reference to one.
## Readings on the dispatch's hypotheses
- **H1 (dotted / cross-field), confirmed in part.** The dotted half is
moot at the door: the dotted-path verdict refuses it. The cross-field
half is a measured gap. The depth backstop was a second gap the
measurement found.
- **H2 (`count` discards the guard's return), confirmed and corrected.**
`count` now consumes the guard's return. The pin: a default search
through `count` arrives narrowed.
- **H3 (`having`), moot.** See the table. The collector deliberately
does not read `having`: an aggregation alias spelled like a family
column is a legitimate count, and a parity control pins it as run.
- **H4 (the door's own answers unchanged), confirmed.**
- The existing door suites pass unchanged: `#21207` search, hash and
note, `#21120` body.
- The parity table pins explicit list, default search and emptied set
identical at the door and the seam.
- **H5 (other `collectConditionFields` readers).** After this PR its one
reader is `@objectstack/plugin-security`'s own FLS predicate guard
(`collectQueryFields` / `assertReadableQueryFields`). That guard does
not judge the family. `@objectstack/runtime` still depends on
`@objectstack/plugin-security` through
`security/resolve-execution-context.ts`. No `plugin-security` edit.
## Tests
- New in
`packages/metadata-protocol/src/protocol.data-door-stored-metadata-filter-reads.test.ts`:
- collector cases (16 shapes on both tables, plus non-family, cycle and
shared-node pins);
- narrowing cases;
- door pins: 6 gap shapes × 5 family columns, each refused with `code` /
`status` / `param` / `field` / `object` and the engine never asked;
- scalar-column and non-family controls.
- New in `packages/runtime/src/stored-metadata-reader-seam.test.ts`: the
door / seam parity table. It is one table of 24 cases: 7 search cases
(explicit list ×4, default ×2, emptied); 14 collector cases (direct,
dotted key ×2, cross-field comparand ×3, dotted reference, list
reference, `$not`, unknown `$` key, depth 33 ×2, aggregation filter ×2);
and 3 controls. Each case runs through `findData` and through the seam,
and the two outcomes must be equal. There is also a narrowed-default pin
and the `count` pin.
- `pnpm --filter @objectstack/metadata-protocol exec vitest run`: 208
files passed, 3 skipped; 3266 tests passed, 19 skipped (at `5513190ca5`,
after merging `main`).
- `pnpm --filter @objectstack/runtime exec vitest run --project local`:
318 files passed; 4514 passed, 19 skipped (at `5513190ca5`). `--project
repo`: 3 files, 751 passed (at `9be38c5987`).
- At the final head `0ac5749662`:
- the three family door suites: 99 passed;
- the seam, reader-contexts, body-writes and boundary suites: 86 passed.
- `typecheck`: both packages green (at `5513190ca5`). `--listFiles`
confirms both new tests are inside each package's tsc program.
## Reverse verification (both at `be99ceee6a`, through
`scripts/ablation-replace.mjs`, mutation and restore proven on disk)
1. **The seam reverted to its own narrowing and collector.** The file
was swapped to its base blob `27efc3412999`; on disk,
`narrowFamilySearch` = 1, `collectConditionFields` = 3 and the new
collector = 0. Result: **2 red** (the parity case for an unrecognised
`$` key wrapping a body filter, and the `count` pin) and 41 green.
Restored with `git checkout HEAD -- PATH`: blob == HEAD `51fe56bb5e73`,
`git diff HEAD` empty.
2. **The door's collector reverted** to the ingress key collector
expression. On disk the ablation marker = 1 and the new call = 0.
Result: **exactly the 30 door-gap pins red** (6 shapes × 5 columns); the
69 others stayed green (collector and narrowing units, controls, the
existing `#21207` / `#21120` door suites). Restored: blob == HEAD
`fa64d2b26cd9`, `git diff HEAD` empty.
Both are src-resolved: each subject is imported by relative path, so no
`dist/` leg was needed.
## Gates (at `0ac5749662`)
- `node scripts/pm/dispatch-gates.mjs --commands` derived 64 families;
all 64 were run and exited 0. Reconciled with `--ran`: 64 run, 0
NOT-MEASURED, 0 UNRUN.
- Two needed a step first:
- `check:dual-build-cjs-loads` printed `PREREQUISITE NOT MET` before a
full `turbo build`, then exited 0.
- `check:engine-double-contract` flagged the new door doubles'
`findOne`. `findData` never reads `findOne`, so the doubles carry none,
and the pinned ledger is untouched.
- Lint is a proven narrowing. ESLint ran with `--no-inline-config
--format json` on the 6 touched TS files at `0ac5749662`: 6 files, 0
errors, 0 warnings. The checked population is read from
`eslint.config.mjs` (`--print-config` per file). Type-aware linting is
not enabled anywhere: no `parserOptions.project` or `projectService`,
`project=null` per file. So this diff cannot move any untouched file's
verdict.
## File surface
- Declared:
- the data door's read regions of
`packages/metadata-protocol/src/protocol.ts`. The hydration region is
untouched; PR #21603 was merged in from `main`.
- `packages/metadata-protocol/src/index.ts`
- `packages/runtime/src/stored-metadata-reader-seam.ts`
- tests in both packages
- `.changeset/21544-door-narrowing-export.md`
- Beyond the claim's list, one sentence of comment in
`packages/metadata-protocol/src/metadata-redaction.ts`. It is the
`storedMetadataBodyPredicateRefusal` docblock's parenthetical, which
named the old collector as the source of `filterFields` and would have
been false after this change. No code.
- The changeset carries `Clause-②: yes (narrowing)` and an ADR-0087
`not-required (no-migration-prescription)` disposition marker.
`check:adr-0087-registration` reads it.
## Acceptance notes (observations, not filed)
- **`driver-memory` does not resolve a cross-field reference in
`where`.** It compares against the literal reference object, so on a
non-family object `{ "name": { "$eq": { "$field": "name" } } }` answered
0 rows (SQL: every row). It was measured at `b610eabf72` through the
generic data door on a memory-backed composition. Public reach is not
measured after `9a4182a752` (the in-memory engine is no longer a boot
store), so it is not filed. Carrier: none.
- **`count` / `count_distinct` over a family column is still served,**
at the door and the seam alike. It discloses equality only, which the
keyed content hash already serves per row.
- **The ingress gate's `collectFilterFieldKeys` keeps its 32-level
backstop for the existence question.** That question is not the
family's. The backstop's reach on an unknown field nested below it is an
unmeasured inference.
---
_Generated by [Claude
Code](https://claude.ai/code/session_017ErfyP2Rx7XWHJA27QjyUi)_
---------
Co-authored-by: Claude <noreply@anthropic.com>1 parent 36ad321 commit 5d0e4e2
7 files changed
Lines changed: 690 additions & 161 deletions
File tree
- .changeset
- packages
- metadata-protocol/src
- runtime/src
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
220 | 220 | | |
221 | 221 | | |
222 | 222 | | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
223 | 230 | | |
224 | 231 | | |
225 | 232 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
742 | 742 | | |
743 | 743 | | |
744 | 744 | | |
745 | | - | |
746 | | - | |
| 745 | + | |
| 746 | + | |
| 747 | + | |
747 | 748 | | |
748 | 749 | | |
749 | 750 | | |
| |||
Lines changed: 234 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
0 commit comments