Repository navigation
Commit 55382dc
fix(objectql,metadata-protocol,spec): a paged search reports the counted total (#22857)
Fixes #22790
Clause-②: yes (widening)
## What this does
Under a search, a paged list's `total` is now the counted number of
matches, and `hasMore` agrees with it. This is the triage direction
(6106471041), executed as ruled: one expander serves both verbs. There
is no second expander, no omitted `total` and no estimate marker.
- **`@objectstack/objectql`** (`packages/objectql/src/engine.ts`):
`ENGINE_COUNT_OPTION_KEYS` admits `search` / `searchFields`. `count()`
runs them through the existing `expandSearchOnAst`, via the carrier
helper the aggregate verb already used, now named
`expandSearchOnOptions` and typed for both bags. That is the expansion
`find` runs, with its `internal: true` exclusion, its `searchableFields`
/ `nameField` resolution and its `searchFields` narrowing. It sits at
the same point in the sequence as on `find` and `aggregate`: after the
`where` doors, before the AST is built, tokens resolve and the security
middlewares run.
- **`@objectstack/metadata-protocol`**
(`packages/metadata-protocol/src/protocol.ts`): `findData`'s paged arm
passes `search` / `searchFields` to `engine.count()`. The `countable`
exception and its page-local estimate arm are deleted. `$count=false` is
unchanged. The degraded-count `catch` is untouched (out of scope per
triage).
- **`@objectstack/spec`** (`packages/spec/src/data/data-engine.zod.ts`):
`EngineCountOptionsSchema` declares `search` and `searchFields` with the
same Zod expressions as `EngineQueryOptionsSchema`. The structured
`search` arm carries flag defaults, so the author and parsed states
part. `EngineCountOptionsParsed` is therefore declared (ADR-0122,
`check:spec-parsed-alias`), and the count schema's isomorphism pin
leaves `type-alias-convention.pin.test.ts` (776 to 775).
`check:api-surface`: 1 added, 0 removed or narrowed.
## Before / after at the public door
`packages/rest/src/list-search-total-door.test.ts` (new) drives the
`RestServer` route handlers, `findData`, `ObjectQL` and a sqlite
`SqlDriver` over 90 rows. 75 rows match `Line`: 60 in `title` and 15 in
`notes`. 15 more match only in the `internal: true` column `hint`. A
middleware scopes the member to every third row, which is 25 of the 75
matches.
| request | before | this PR |
|:--|:--|:--|
| admin `GET ?$top=25&$search=Line` | `total: 26`, `hasMore: true` |
`total: 75`, `hasMore: true` |
| admin `GET ?$top=25&$skip=50&$search=Line` (last page) | `total: 76`,
`hasMore: true` | `total: 75`, `hasMore: false` |
| admin `POST …/query` `{ search: 'Line', limit: 25 }` | `total: 26` |
`total: 75` |
| admin `POST …/query` with `searchFields: ['title']` | not reached |
`total: 60`; the unpaged rows are 60 |
| member `GET ?$top=10&$search=Line` | `total: 11` | `total: 25` |
| CONTROL: admin / member `GET ?$top`, no search | 90 / 30 | 90 / 30 |
| CONTROL: `$count=false` with a search | `total` omitted | `total`
omitted |
The first-page, POST and member "before" figures were measured on base
`c4e7fa5a31`. The last-page figure comes from ablation A1 below, which
restores the protocol's estimate arm on this branch. That is the base
door's behaviour, because the engine change is inert when the count is
sent no search.
## Mechanism hypotheses, measured
1. The PM's reading held on `c4e7fa5a31`: `protocol.ts` had `const
countable = options.search == null;` and the estimate arm (line 13321).
`ObjectQL.count` was gated by `rejectUnknownEngineOptions(…,
ENGINE_COUNT_OPTION_KEYS)` over `{ context, where }` (line 18696).
`find`'s expansion was `expandSearchOnAst` (line 12632), with
`collectInternalReadFields`.
2. **The Clause-② arm is `yes (widening)`.** No route exists without the
spec. The drift pin in `engine-unknown-option.test.ts` holds
`ENGINE_OPTION_KEY_SETS.count` equal to `EngineCountOptionsSchema`'s
shape, with no documented extras on `count`. Admitting `search`
engine-side without declaring it reds that pin. The remaining routes are
a bypass of `rejectUnknownEngineOptions` or a private side channel, and
both are ruled out.
3. Both verbs use the same expansion, measured. The driver's COUNT
receives a `where` deep-equal to the driver's FIND `where` for the same
request. The count equals the unpaged row count on the default set (75,
with the internal column excluded), on the narrowed set (60) and in the
member's scope (25).
4. Engine doubles: `check:engine-double-contract` and
`check:objectql-double-limit` exit 0. No double changed, because the new
pin uses the real engine.
`searchFields` is declared beside `search`, deliberately. `findData`
hands `find` both keys, and on stored-metadata tables it narrows
`searchFields` itself (`narrowStoredMetadataSearch`). A count that took
`search` alone would count over the wider default set, so `total` would
exceed the rows paged. On a stored-metadata table it would also count
matches in the withheld body and hash columns.
## Landing beyond the declared file surface
The claim named three source files. The diff also touches the following
files, and the widening forces each one:
- `packages/spec/src/data/hook-api.ts` (TSDoc only) and
`hook-api.test.ts`. `HookCountQuery` is an `Omit` of
`EngineCountOptions`, so it gains the pair by derivation. Its TSDoc said
"`where` and nothing else" and quoted `ENGINE_COUNT_OPTION_KEYS` as `{
context, where }`. Both became false with this change, and both are
corrected. The test's drift pin ("each derived shape equals KEPT ∪
OMITTED") forces the decision: the pair is KEPT, because
`ctx.api.object(n).count()` forwards its bag to `engine.count()` and now
honours it. **This is a spec source file beyond the #6017 declaration
(6108690316), and it is flagged to the PM in the report.**
- `packages/spec/src/type-alias-convention.pin.test.ts`: the isomorphism
pin leaves (776 to 775).
- `packages/spec/src/data/data-engine.test.ts`: a new case checks that
the count options declare the pair exactly as the find options do
(JSON-schema equality, both `search` forms).
- Two tests pinned the old estimate and are flipped:
`protocol.zero-limit-total.test.ts` (metadata-protocol) and
`protocol-data.test.ts` (objectql).
- Four artifacts were regenerated with the repo's tooling, the ones
`check:generated` named: `api-surface/data.json`,
`authorable-surface/data.json`, `export-origins/data.json` and
`content/docs/references/data/data-engine.mdx`.
- The public-door pin,
`packages/rest/src/list-search-total-door.test.ts`.
## Tests
- **Base:** `c4e7fa5a31`. `origin/main` is not merged in: neither
in-flight neighbour (#22819, #15206) has landed, and `origin/main`
`d8c7d38648` touches none of these files.
- **Commits:** the objectql and metadata-protocol suites ran on
`e85d2a4d13`. The one commit after it, `4e057d908e`, adds the changeset
and makes the door pin's page assertions soft. Everything else ran on
`4e057d908e`: rest, spec, the typechecks, the ablations and the gates.
- **Runner:** every vitest run used `--maxWorkers=2`.
| package | run | result |
|:--|:--|:--|
| `@objectstack/objectql` | `vitest run --project local` | 399 files,
7802 passed |
| `@objectstack/objectql` | `vitest run --project repo` | 1 file, 5
passed |
| `@objectstack/objectql` | `typecheck` | exit 0 |
| `@objectstack/metadata-protocol` | `vitest run` | 225 files (+3
skipped), 28129 passed, 19 skipped |
| `@objectstack/metadata-protocol` | `typecheck` | exit 0 |
| `@objectstack/rest` | `vitest run --project local` | 273 files, 5251
passed, 326 skipped |
| `@objectstack/rest` | `vitest run --project repo` | 5 files, 209
passed, 1 skipped |
| `@objectstack/rest` | `typecheck` | exit 0 |
| `@objectstack/spec` | `vitest run --project local` | 648 files, 19343
passed, 1 todo |
| `@objectstack/spec` | `vitest run --project repo` | 55 files, 971
passed |
| `@objectstack/spec` | `typecheck` (tsc, scripts and test layer; the
three edited spec tests are in the test-layer program) | exit 0 |
The filter direction is the changed packages themselves, plus
`@objectstack/rest`, which hosts the door pin and compiles against the
widened type.
**Ablations** (one leg per negative pin, plus the two halves of the
fix). In each leg, `scripts/ablation-replace.mjs` makes the mutation:
the anchor must hit, the blob change is verified, and the restore checks
that the blob equals HEAD and that `git diff HEAD` is empty. The package
is then rebuilt and `ablation-dist-preflight.mjs` finds the marker in
`dist/`. The door pin runs, the package is rebuilt again, and the
preflight confirms with `--absent` that the marker is gone and the tree
is clean.
| leg | mutation | what went red |
|:--|:--|:--|
| A1 | protocol: the estimate arm restored under a search | 4 of 6: GET
`total` 26 (want 75), last page 76 / `hasMore: true`; POST 26;
one-expander (no COUNT issued); member 11 (want 25). Both controls
stayed green. |
| A2 | protocol: `$count=false` ignored | the `$count=false` control
(`total` present) |
| A3 | engine: the `internal: true` exclusion removed from the shared
expander | unpaged rows 90 (want 75), POST `total` 90, member 30 |
| A4 | engine: the count's driver call built from the caller's `where`
instead of the middleware-scoped AST | member 75 (want 25), member
control 90 (want 30) |
| A5 | engine: the count's search expansion removed | GET / POST totals
90, last page `hasMore: true`, one-expander, member |
Two first attempts did not run, and both were re-run:
- The first A1 attempt was a no-op. Its replacement contained its
anchor, so `ablation-replace.mjs` refused it (anchor count 1 to 1) and
restored the file. It was re-run with a distinct anchor.
- The first A3 attempt never ran its test. The marker's source spelling
carries a `string[]` annotation that `dist/` drops, so the preflight's
tree reading refused (exit 91). It was re-run with `--source-marker`.
**Gates.** `node scripts/pm/dispatch-gates.mjs --commands --repo
objectstack-ai/objectstack` re-derived 116 commands for this diff: the
dispatch's 78, plus 38 from the changeset, docs, engine-double,
query-options-erasure and type-check-coverage families, among others.
Every one was run with its exit code recorded. The `--ran`
reconciliation answers 116 derived, 115 run, 1 NOT MEASURED, 0 unrun.
- **NOT MEASURED:** `pnpm check:dual-build-cjs-loads`, exit 3,
PREREQUISITE NOT MET. 44 packages outside this diff's build closure have
no `dist/`. A narrowed probe loaded every CJS entry of the three changed
packages with 0 failures: spec 19, objectql 2, metadata-protocol 1.
- `pnpm --filter @objectstack/spec run check:skill-examples` first
exited 3, because the client SDK was not built. After building
`@objectstack/client-react`'s closure it exits 0 (262 examples).
- The derivation printed STALE TREE: the tree is 5 commits behind
`origin/main`, and 7 derivation inputs changed there. The gates ran on
this branch's tree.
**Lint** is narrowed here; CI owns the repo-wide run. `eslint
--no-inline-config --format json` over the 10 touched `.ts` files
reports 10 files, 0 errors and 0 warnings. Population: all 10 resolve a
config (`--print-config`). Invariance: `eslint.config.mjs` enables no
type-aware linting (no `parserOptions.project`), so this diff cannot
move a verdict on an untouched file.
**Line budget:** +301 / -62 across 15 files, 25 of the added lines
generated. The four source files account for +61 / -42, mostly corrected
or replaced comment text. The door pin (157 lines) exceeds the suggested
300 / 60 on its own: it carries a real-chain harness, a member scope and
the driver capture.
## Acceptance notes
- The degraded-count `catch` in `findData`, which reports a failed COUNT
as `offset + page`, is untouched per triage. It was not measured
reachable here.
- `DataEngineCountRequestSchema` (the RPC request shape) extends
`EngineCountOptionsSchema`, so its generated doc rows gained the pair
too. `git grep` finds no runtime reader of that schema outside
`packages/spec`.
- objectui#12123's footer check needs no change. Once this lands, its
page-labelled fallback no longer fires under a search.
- Clause-② is `yes`, so this PR owes one contract-review-tier review
before the queue. The seat arranges it.
---
_Generated by [Claude
Code](https://claude.ai/code/session_01JfJfBUC3cQ6hhgm9MQK76T)_
---------
Co-authored-by: Claude <noreply@anthropic.com>1 parent 6364375 commit 55382dc
15 files changed
Lines changed: 301 additions & 62 deletions
File tree
- .changeset
- content/docs/references/data
- packages
- metadata-protocol/src
- objectql/src
- rest/src
- spec
- api-surface
- authorable-surface
- export-origins
- src
- data
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
221 | 221 | | |
222 | 222 | | |
223 | 223 | | |
224 | | - | |
| 224 | + | |
225 | 225 | | |
226 | 226 | | |
227 | 227 | | |
228 | 228 | | |
229 | 229 | | |
230 | 230 | | |
231 | 231 | | |
| 232 | + | |
| 233 | + | |
232 | 234 | | |
233 | 235 | | |
234 | 236 | | |
| |||
684 | 686 | | |
685 | 687 | | |
686 | 688 | | |
687 | | - | |
| 689 | + | |
688 | 690 | | |
689 | 691 | | |
690 | 692 | | |
691 | 693 | | |
692 | 694 | | |
693 | 695 | | |
694 | 696 | | |
| 697 | + | |
| 698 | + | |
695 | 699 | | |
696 | 700 | | |
697 | 701 | | |
| |||
994 | 998 | | |
995 | 999 | | |
996 | 1000 | | |
| 1001 | + | |
| 1002 | + | |
997 | 1003 | | |
998 | 1004 | | |
999 | 1005 | | |
| |||
1033 | 1039 | | |
1034 | 1040 | | |
1035 | 1041 | | |
| 1042 | + | |
| 1043 | + | |
| 1044 | + | |
| 1045 | + | |
| 1046 | + | |
| 1047 | + | |
| 1048 | + | |
| 1049 | + | |
| 1050 | + | |
| 1051 | + | |
| 1052 | + | |
| 1053 | + | |
| 1054 | + | |
1036 | 1055 | | |
1037 | 1056 | | |
1038 | 1057 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
13292 | 13292 | | |
13293 | 13293 | | |
13294 | 13294 | | |
13295 | | - | |
13296 | | - | |
13297 | | - | |
13298 | | - | |
| 13295 | + | |
| 13296 | + | |
| 13297 | + | |
| 13298 | + | |
| 13299 | + | |
13299 | 13300 | | |
13300 | 13301 | | |
13301 | 13302 | | |
| |||
13306 | 13307 | | |
13307 | 13308 | | |
13308 | 13309 | | |
13309 | | - | |
13310 | | - | |
13311 | | - | |
| 13310 | + | |
| 13311 | + | |
13312 | 13312 | | |
13313 | 13313 | | |
13314 | 13314 | | |
| |||
13318 | 13318 | | |
13319 | 13319 | | |
13320 | 13320 | | |
13321 | | - | |
13322 | 13321 | | |
13323 | 13322 | | |
13324 | 13323 | | |
| |||
13330 | 13329 | | |
13331 | 13330 | | |
13332 | 13331 | | |
13333 | | - | |
13334 | | - | |
13335 | | - | |
13336 | | - | |
| 13332 | + | |
| 13333 | + | |
13337 | 13334 | | |
13338 | 13335 | | |
13339 | 13336 | | |
13340 | | - | |
13341 | | - | |
13342 | | - | |
| 13337 | + | |
| 13338 | + | |
13343 | 13339 | | |
13344 | 13340 | | |
13345 | | - | |
| 13341 | + | |
13346 | 13342 | | |
13347 | 13343 | | |
13348 | 13344 | | |
| |||
13351 | 13347 | | |
13352 | 13348 | | |
13353 | 13349 | | |
| 13350 | + | |
| 13351 | + | |
13354 | 13352 | | |
13355 | 13353 | | |
13356 | 13354 | | |
| |||
13360 | 13358 | | |
13361 | 13359 | | |
13362 | 13360 | | |
13363 | | - | |
13364 | | - | |
13365 | | - | |
13366 | 13361 | | |
13367 | 13362 | | |
13368 | 13363 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
30 | 30 | | |
31 | 31 | | |
32 | 32 | | |
33 | | - | |
34 | | - | |
35 | | - | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
36 | 36 | | |
37 | 37 | | |
38 | 38 | | |
| |||
149 | 149 | | |
150 | 150 | | |
151 | 151 | | |
152 | | - | |
153 | | - | |
| 152 | + | |
| 153 | + | |
154 | 154 | | |
155 | | - | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
156 | 158 | | |
157 | 159 | | |
158 | 160 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
623 | 623 | | |
624 | 624 | | |
625 | 625 | | |
626 | | - | |
627 | | - | |
628 | | - | |
629 | | - | |
630 | | - | |
631 | | - | |
| 626 | + | |
| 627 | + | |
| 628 | + | |
| 629 | + | |
| 630 | + | |
| 631 | + | |
| 632 | + | |
632 | 633 | | |
633 | 634 | | |
634 | 635 | | |
| |||
12729 | 12730 | | |
12730 | 12731 | | |
12731 | 12732 | | |
12732 | | - | |
12733 | | - | |
12734 | | - | |
12735 | | - | |
12736 | | - | |
12737 | | - | |
12738 | | - | |
| 12733 | + | |
| 12734 | + | |
| 12735 | + | |
| 12736 | + | |
| 12737 | + | |
| 12738 | + | |
| 12739 | + | |
| 12740 | + | |
| 12741 | + | |
12739 | 12742 | | |
12740 | 12743 | | |
12741 | 12744 | | |
| |||
12744 | 12747 | | |
12745 | 12748 | | |
12746 | 12749 | | |
12747 | | - | |
12748 | | - | |
12749 | | - | |
12750 | | - | |
| 12750 | + | |
12751 | 12751 | | |
12752 | 12752 | | |
12753 | 12753 | | |
12754 | 12754 | | |
12755 | | - | |
12756 | | - | |
| 12755 | + | |
| 12756 | + | |
12757 | 12757 | | |
12758 | 12758 | | |
12759 | 12759 | | |
| |||
18757 | 18757 | | |
18758 | 18758 | | |
18759 | 18759 | | |
| 18760 | + | |
| 18761 | + | |
| 18762 | + | |
| 18763 | + | |
18760 | 18764 | | |
18761 | 18765 | | |
18762 | 18766 | | |
| |||
18892 | 18896 | | |
18893 | 18897 | | |
18894 | 18898 | | |
18895 | | - | |
| 18899 | + | |
18896 | 18900 | | |
18897 | 18901 | | |
18898 | 18902 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
287 | 287 | | |
288 | 288 | | |
289 | 289 | | |
290 | | - | |
291 | | - | |
292 | | - | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
293 | 293 | | |
| 294 | + | |
294 | 295 | | |
295 | 296 | | |
296 | 297 | | |
297 | | - | |
| 298 | + | |
| 299 | + | |
298 | 300 | | |
299 | 301 | | |
300 | 302 | | |
| |||
0 commit comments