Skip to content

Commit 55382dc

Browse files
fix(objectql,metadata-protocol,spec): a paged search reports the counted total (#22857)
Fixes #22790 Clause-②: yes (widening) ## What this does Under a search, a paged list's `total` is now the counted number of matches, and `hasMore` agrees with it. This is the triage direction (6106471041), executed as ruled: one expander serves both verbs. There is no second expander, no omitted `total` and no estimate marker. - **`@objectstack/objectql`** (`packages/objectql/src/engine.ts`): `ENGINE_COUNT_OPTION_KEYS` admits `search` / `searchFields`. `count()` runs them through the existing `expandSearchOnAst`, via the carrier helper the aggregate verb already used, now named `expandSearchOnOptions` and typed for both bags. That is the expansion `find` runs, with its `internal: true` exclusion, its `searchableFields` / `nameField` resolution and its `searchFields` narrowing. It sits at the same point in the sequence as on `find` and `aggregate`: after the `where` doors, before the AST is built, tokens resolve and the security middlewares run. - **`@objectstack/metadata-protocol`** (`packages/metadata-protocol/src/protocol.ts`): `findData`'s paged arm passes `search` / `searchFields` to `engine.count()`. The `countable` exception and its page-local estimate arm are deleted. `$count=false` is unchanged. The degraded-count `catch` is untouched (out of scope per triage). - **`@objectstack/spec`** (`packages/spec/src/data/data-engine.zod.ts`): `EngineCountOptionsSchema` declares `search` and `searchFields` with the same Zod expressions as `EngineQueryOptionsSchema`. The structured `search` arm carries flag defaults, so the author and parsed states part. `EngineCountOptionsParsed` is therefore declared (ADR-0122, `check:spec-parsed-alias`), and the count schema's isomorphism pin leaves `type-alias-convention.pin.test.ts` (776 to 775). `check:api-surface`: 1 added, 0 removed or narrowed. ## Before / after at the public door `packages/rest/src/list-search-total-door.test.ts` (new) drives the `RestServer` route handlers, `findData`, `ObjectQL` and a sqlite `SqlDriver` over 90 rows. 75 rows match `Line`: 60 in `title` and 15 in `notes`. 15 more match only in the `internal: true` column `hint`. A middleware scopes the member to every third row, which is 25 of the 75 matches. | request | before | this PR | |:--|:--|:--| | admin `GET ?$top=25&$search=Line` | `total: 26`, `hasMore: true` | `total: 75`, `hasMore: true` | | admin `GET ?$top=25&$skip=50&$search=Line` (last page) | `total: 76`, `hasMore: true` | `total: 75`, `hasMore: false` | | admin `POST …/query` `{ search: 'Line', limit: 25 }` | `total: 26` | `total: 75` | | admin `POST …/query` with `searchFields: ['title']` | not reached | `total: 60`; the unpaged rows are 60 | | member `GET ?$top=10&$search=Line` | `total: 11` | `total: 25` | | CONTROL: admin / member `GET ?$top`, no search | 90 / 30 | 90 / 30 | | CONTROL: `$count=false` with a search | `total` omitted | `total` omitted | The first-page, POST and member "before" figures were measured on base `c4e7fa5a31`. The last-page figure comes from ablation A1 below, which restores the protocol's estimate arm on this branch. That is the base door's behaviour, because the engine change is inert when the count is sent no search. ## Mechanism hypotheses, measured 1. The PM's reading held on `c4e7fa5a31`: `protocol.ts` had `const countable = options.search == null;` and the estimate arm (line 13321). `ObjectQL.count` was gated by `rejectUnknownEngineOptions(…, ENGINE_COUNT_OPTION_KEYS)` over `{ context, where }` (line 18696). `find`'s expansion was `expandSearchOnAst` (line 12632), with `collectInternalReadFields`. 2. **The Clause-② arm is `yes (widening)`.** No route exists without the spec. The drift pin in `engine-unknown-option.test.ts` holds `ENGINE_OPTION_KEY_SETS.count` equal to `EngineCountOptionsSchema`'s shape, with no documented extras on `count`. Admitting `search` engine-side without declaring it reds that pin. The remaining routes are a bypass of `rejectUnknownEngineOptions` or a private side channel, and both are ruled out. 3. Both verbs use the same expansion, measured. The driver's COUNT receives a `where` deep-equal to the driver's FIND `where` for the same request. The count equals the unpaged row count on the default set (75, with the internal column excluded), on the narrowed set (60) and in the member's scope (25). 4. Engine doubles: `check:engine-double-contract` and `check:objectql-double-limit` exit 0. No double changed, because the new pin uses the real engine. `searchFields` is declared beside `search`, deliberately. `findData` hands `find` both keys, and on stored-metadata tables it narrows `searchFields` itself (`narrowStoredMetadataSearch`). A count that took `search` alone would count over the wider default set, so `total` would exceed the rows paged. On a stored-metadata table it would also count matches in the withheld body and hash columns. ## Landing beyond the declared file surface The claim named three source files. The diff also touches the following files, and the widening forces each one: - `packages/spec/src/data/hook-api.ts` (TSDoc only) and `hook-api.test.ts`. `HookCountQuery` is an `Omit` of `EngineCountOptions`, so it gains the pair by derivation. Its TSDoc said "`where` and nothing else" and quoted `ENGINE_COUNT_OPTION_KEYS` as `{ context, where }`. Both became false with this change, and both are corrected. The test's drift pin ("each derived shape equals KEPT ∪ OMITTED") forces the decision: the pair is KEPT, because `ctx.api.object(n).count()` forwards its bag to `engine.count()` and now honours it. **This is a spec source file beyond the #6017 declaration (6108690316), and it is flagged to the PM in the report.** - `packages/spec/src/type-alias-convention.pin.test.ts`: the isomorphism pin leaves (776 to 775). - `packages/spec/src/data/data-engine.test.ts`: a new case checks that the count options declare the pair exactly as the find options do (JSON-schema equality, both `search` forms). - Two tests pinned the old estimate and are flipped: `protocol.zero-limit-total.test.ts` (metadata-protocol) and `protocol-data.test.ts` (objectql). - Four artifacts were regenerated with the repo's tooling, the ones `check:generated` named: `api-surface/data.json`, `authorable-surface/data.json`, `export-origins/data.json` and `content/docs/references/data/data-engine.mdx`. - The public-door pin, `packages/rest/src/list-search-total-door.test.ts`. ## Tests - **Base:** `c4e7fa5a31`. `origin/main` is not merged in: neither in-flight neighbour (#22819, #15206) has landed, and `origin/main` `d8c7d38648` touches none of these files. - **Commits:** the objectql and metadata-protocol suites ran on `e85d2a4d13`. The one commit after it, `4e057d908e`, adds the changeset and makes the door pin's page assertions soft. Everything else ran on `4e057d908e`: rest, spec, the typechecks, the ablations and the gates. - **Runner:** every vitest run used `--maxWorkers=2`. | package | run | result | |:--|:--|:--| | `@objectstack/objectql` | `vitest run --project local` | 399 files, 7802 passed | | `@objectstack/objectql` | `vitest run --project repo` | 1 file, 5 passed | | `@objectstack/objectql` | `typecheck` | exit 0 | | `@objectstack/metadata-protocol` | `vitest run` | 225 files (+3 skipped), 28129 passed, 19 skipped | | `@objectstack/metadata-protocol` | `typecheck` | exit 0 | | `@objectstack/rest` | `vitest run --project local` | 273 files, 5251 passed, 326 skipped | | `@objectstack/rest` | `vitest run --project repo` | 5 files, 209 passed, 1 skipped | | `@objectstack/rest` | `typecheck` | exit 0 | | `@objectstack/spec` | `vitest run --project local` | 648 files, 19343 passed, 1 todo | | `@objectstack/spec` | `vitest run --project repo` | 55 files, 971 passed | | `@objectstack/spec` | `typecheck` (tsc, scripts and test layer; the three edited spec tests are in the test-layer program) | exit 0 | The filter direction is the changed packages themselves, plus `@objectstack/rest`, which hosts the door pin and compiles against the widened type. **Ablations** (one leg per negative pin, plus the two halves of the fix). In each leg, `scripts/ablation-replace.mjs` makes the mutation: the anchor must hit, the blob change is verified, and the restore checks that the blob equals HEAD and that `git diff HEAD` is empty. The package is then rebuilt and `ablation-dist-preflight.mjs` finds the marker in `dist/`. The door pin runs, the package is rebuilt again, and the preflight confirms with `--absent` that the marker is gone and the tree is clean. | leg | mutation | what went red | |:--|:--|:--| | A1 | protocol: the estimate arm restored under a search | 4 of 6: GET `total` 26 (want 75), last page 76 / `hasMore: true`; POST 26; one-expander (no COUNT issued); member 11 (want 25). Both controls stayed green. | | A2 | protocol: `$count=false` ignored | the `$count=false` control (`total` present) | | A3 | engine: the `internal: true` exclusion removed from the shared expander | unpaged rows 90 (want 75), POST `total` 90, member 30 | | A4 | engine: the count's driver call built from the caller's `where` instead of the middleware-scoped AST | member 75 (want 25), member control 90 (want 30) | | A5 | engine: the count's search expansion removed | GET / POST totals 90, last page `hasMore: true`, one-expander, member | Two first attempts did not run, and both were re-run: - The first A1 attempt was a no-op. Its replacement contained its anchor, so `ablation-replace.mjs` refused it (anchor count 1 to 1) and restored the file. It was re-run with a distinct anchor. - The first A3 attempt never ran its test. The marker's source spelling carries a `string[]` annotation that `dist/` drops, so the preflight's tree reading refused (exit 91). It was re-run with `--source-marker`. **Gates.** `node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack` re-derived 116 commands for this diff: the dispatch's 78, plus 38 from the changeset, docs, engine-double, query-options-erasure and type-check-coverage families, among others. Every one was run with its exit code recorded. The `--ran` reconciliation answers 116 derived, 115 run, 1 NOT MEASURED, 0 unrun. - **NOT MEASURED:** `pnpm check:dual-build-cjs-loads`, exit 3, PREREQUISITE NOT MET. 44 packages outside this diff's build closure have no `dist/`. A narrowed probe loaded every CJS entry of the three changed packages with 0 failures: spec 19, objectql 2, metadata-protocol 1. - `pnpm --filter @objectstack/spec run check:skill-examples` first exited 3, because the client SDK was not built. After building `@objectstack/client-react`'s closure it exits 0 (262 examples). - The derivation printed STALE TREE: the tree is 5 commits behind `origin/main`, and 7 derivation inputs changed there. The gates ran on this branch's tree. **Lint** is narrowed here; CI owns the repo-wide run. `eslint --no-inline-config --format json` over the 10 touched `.ts` files reports 10 files, 0 errors and 0 warnings. Population: all 10 resolve a config (`--print-config`). Invariance: `eslint.config.mjs` enables no type-aware linting (no `parserOptions.project`), so this diff cannot move a verdict on an untouched file. **Line budget:** +301 / -62 across 15 files, 25 of the added lines generated. The four source files account for +61 / -42, mostly corrected or replaced comment text. The door pin (157 lines) exceeds the suggested 300 / 60 on its own: it carries a real-chain harness, a member scope and the driver capture. ## Acceptance notes - The degraded-count `catch` in `findData`, which reports a failed COUNT as `offset + page`, is untouched per triage. It was not measured reachable here. - `DataEngineCountRequestSchema` (the RPC request shape) extends `EngineCountOptionsSchema`, so its generated doc rows gained the pair too. `git grep` finds no runtime reader of that schema outside `packages/spec`. - objectui#12123's footer check needs no change. Once this lands, its page-labelled fallback no longer fires under a search. - Clause-② is `yes`, so this PR owes one contract-review-tier review before the queue. The seat arranges it. --- _Generated by [Claude Code](https://claude.ai/code/session_01JfJfBUC3cQ6hhgm9MQK76T)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent 6364375 commit 55382dc

15 files changed

Lines changed: 301 additions & 62 deletions

File tree

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
---
2+
"@objectstack/spec": minor
3+
"@objectstack/objectql": minor
4+
"@objectstack/metadata-protocol": patch
5+
---
6+
7+
A paged list under a search now reports the real number of matching records as `total`, and `hasMore` agrees with it on every page.
8+
9+
Clause-②: yes (widening) — `EngineCountOptionsSchema` gains two OPTIONAL keys, `search` and `searchFields`, so the accept set of the count options grows. Nothing previously admitted is refused, no key is renamed or retired, and no producer is required to write them.
10+
11+
Until now `GET /api/v1/data/:object?$top=25&$search=Line` over 75 matching records answered `total: 26` (offset + page + 1, a page-local estimate), and the last page claimed `hasMore: true`; `POST /api/v1/data/:object/query` with a `search` did the same. Nothing on the response marked the number as an estimate, so the Console rendered it as the record count and offered the wrong number of pages.
12+
13+
- **`@objectstack/spec`** — `EngineCountOptionsSchema` declares `search` (the bare string, or the structured `FullTextSearchSchema` form) and `searchFields`, identically to `EngineQueryOptionsSchema`. FROM `count(object, { where })` only → TO `count(object, { where, search, searchFields })`. The structured `search` form carries flag defaults, so the parsed state is named `EngineCountOptionsParsed` (ADR-0122). `HookCountQuery` (`ctx.api.object(name).count()`) derives from it and gains the same two keys.
14+
- **`@objectstack/objectql`** — `engine.count()` (and `ctx.api.object(name).count()`) accepts the two keys it used to refuse as unknown options, and expands them through the same ADR-0061 expansion `find()` and `aggregate()` use: the same server-resolved searchable fields, the same `internal: true` exclusion, the same `searchFields` narrowing, AND-ed with `where` before the security middlewares run. There is one expander, not two, so a count under a search is the number of rows `find()` returns for the same bag.
15+
- **`@objectstack/metadata-protocol`** — `findData`'s paged branch passes `search` / `searchFields` to `engine.count()`, and the page-local estimate branch is gone. `$count=false` still skips the count and omits `total`.
16+
17+
Nothing to migrate. A client that worked around the estimate, for example by labelling the count as page-local under a search, can read `total` as the match count again.

‎content/docs/references/data/data-engine.mdx‎

Lines changed: 21 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -221,14 +221,16 @@ Options for DataEngine.count operations
221221
| :--- | :--- | :--- | :--- |
222222
| **method** | `'count'` | ✅ | |
223223
| **object** | `string` | ✅ | |
224-
| **query** | `{ context?: object; where?: Record<string, any> \| any; filter?: Record<string, any> \| any }` | optional | |
224+
| **query** | `{ context?: object; where?: Record<string, any> \| any; search?: string \| object; searchFields?: string[]; … }` | optional | |
225225

226226
### Nested Shape: `DataEngineCountRequest.query`
227227

228228
| Property | Type | Required | Description |
229229
| :--- | :--- | :--- | :--- |
230230
| **context** | `{ userId?: string; actor?: string; attributedUserId?: string; email?: string; … }` | optional | |
231231
| **where** | `Record<string, any> \| any` | optional | |
232+
| **search** | `string \| { query: string; fields?: string[]; fuzzy?: boolean; operator?: Enum<'and' \| 'or'>; … }` | optional | |
233+
| **searchFields** | `string[]` | optional | |
232234
| **filter** | `Record<string, any> \| any` | optional | Data Engine query filter conditions |
233235

234236

@@ -684,14 +686,16 @@ This schema accepts one of the following structures:
684686
| :--- | :--- | :--- | :--- |
685687
| **method** | `'count'` | ✅ | |
686688
| **object** | `string` | ✅ | |
687-
| **query** | `{ context?: object; where?: Record<string, any> \| any; filter?: Record<string, any> \| any }` | optional | |
689+
| **query** | `{ context?: object; where?: Record<string, any> \| any; search?: string \| object; searchFields?: string[]; … }` | optional | |
688690

689691
### Nested Shape: `DataEngineRequest[method='count'].query`
690692

691693
| Property | Type | Required | Description |
692694
| :--- | :--- | :--- | :--- |
693695
| **context** | `{ userId?: string; actor?: string; attributedUserId?: string; email?: string; … }` | optional | |
694696
| **where** | `Record<string, any> \| any` | optional | |
697+
| **search** | `string \| { query: string; fields?: string[]; fuzzy?: boolean; operator?: Enum<'and' \| 'or'>; … }` | optional | |
698+
| **searchFields** | `string[]` | optional | |
695699
| **filter** | `Record<string, any> \| any` | optional | Data Engine query filter conditions |
696700

697701
---
@@ -994,6 +998,8 @@ QueryAST-aligned options for DataEngine.count operations
994998
| :--- | :--- | :--- | :--- |
995999
| **context** | `{ userId?: string; actor?: string; attributedUserId?: string; email?: string; … }` | optional | |
9961000
| **where** | `Record<string, any> \| any` | optional | |
1001+
| **search** | `string \| { query: string; fields?: string[]; fuzzy?: boolean; operator?: Enum<'and' \| 'or'>; … }` | optional | |
1002+
| **searchFields** | `string[]` | optional | |
9971003

9981004
### Nested Shape: `EngineCountOptions.context`
9991005

@@ -1033,6 +1039,19 @@ QueryAST-aligned options for DataEngine.count operations
10331039
| **transaction** | `any` | optional | |
10341040
| **traceId** | `string` | optional | |
10351041

1042+
### Nested Shape: `EngineCountOptions.search`
1043+
1044+
| Property | Type | Required | Description |
1045+
| :--- | :--- | :--- | :--- |
1046+
| **query** | `string` | ✅ | Search query text |
1047+
| **fields** | `string[]` | optional | Fields to search in (if not specified, searches all text fields) |
1048+
| **fuzzy** | `boolean` | optional (default: `false`) | [EXPERIMENTAL — not enforced] Fuzzy matching (tolerate typos). The ADR-0061 expansion reads only `query` + `fields`; no executor receives this flag. |
1049+
| **operator** | `Enum<'and' \| 'or'>` | optional (default: `"or"`) | [EXPERIMENTAL — not enforced] Logical operator between terms. The ADR-0061 expansion applies its own term semantics; no executor receives this flag. |
1050+
| **boost** | `Record<string, number>` | optional | [EXPERIMENTAL — not enforced] Field-specific relevance boosting (field name -> boost factor). No executor scores results. |
1051+
| **minScore** | `number` | optional | [EXPERIMENTAL — not enforced] Minimum relevance score threshold. No executor scores results. |
1052+
| **language** | `string` | optional | [EXPERIMENTAL — not enforced] Language for text analysis (e.g., "en", "zh", "es"). No executor selects an analyzer. |
1053+
| **highlight** | `boolean` | optional (default: `false`) | [EXPERIMENTAL — not enforced] Search result highlighting. No executor emits highlights. |
1054+
10361055

10371056
---
10381057

‎packages/metadata-protocol/src/protocol.ts‎

Lines changed: 14 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -13292,10 +13292,11 @@ export class ObjectStackProtocolImplementation implements
1329213292
// whether more pages remain (true server-side pagination). Without a limit
1329313293
// the full result set is returned, so its length already IS the total.
1329413294
//
13295-
// engine.count() only honors `where`; a `search`/`distinct` query can't be
13296-
// reproduced by it, so for those we skip the count and fall back to a
13297-
// page-local estimate (a full page implies there may be more) rather than
13298-
// reporting a wrong total.
13295+
// [#22790] The count takes the query's `search` / `searchFields` too: the
13296+
// engine expands them through the one ADR-0061 expander `find` uses, so
13297+
// `total` counts exactly the rows this query pages. A search used to skip
13298+
// the count and report a page-local estimate (`offset + page + 1`) that
13299+
// the Console rendered as the record count.
1329913300
//
1330013301
// [#22588] `limit: 0` IS a limit. It asks for no rows (objectstack#6485,
1330113302
// the drivers' pagination conformance), so `records.length` is 0 and was
@@ -13306,9 +13307,8 @@ export class ObjectStackProtocolImplementation implements
1330613307
// the full set is the total" arm, which is true of every limit except
1330713308
// this one. So `>= 0`: a zero-row request is a paged request like any
1330813309
// other, and the three arms below keep their meaning for it — the count
13309-
// runs (`total` is the filtered count, whatever the page size), `$count=false`
13310-
// still omits `total` instead of reporting the page's 0, and `search`
13311-
// still reports its page-local estimate rather than a count.
13310+
// runs (`total` is the filtered count, whatever the page size), and
13311+
// `$count=false` still omits `total` instead of reporting the page's 0.
1331213312
const pageLimit = typeof options.limit === 'number' && options.limit >= 0 ? options.limit : undefined;
1331313313
const pageOffset = typeof options.offset === 'number' && options.offset > 0 ? options.offset : 0;
1331413314
let total: number | undefined = records.length;
@@ -13318,7 +13318,6 @@ export class ObjectStackProtocolImplementation implements
1331813318
// the flag's ONLY observable effect platform-wide, on a capability
1331913319
// that never deduplicated a row. Removed with `query.distinct`
1332013320
// (tombstoned in spec 17); `total`/`hasMore` are truthful again.
13321-
const countable = options.search == null;
1332213321
if (countOptOut) {
1332313322
// [#10757] The caller said it does not need `total`, so the
1332413323
// COUNT query is not issued at all — that is the whole point of
@@ -13330,19 +13329,16 @@ export class ObjectStackProtocolImplementation implements
1333013329
// filter (IF REQUESTED)",
1333113330
// `packages/spec/src/api/protocol.zod.ts`), so absent is the
1333213331
// declared shape for "not requested" — and it is the only
13333-
// honest one: the `search` branch below reports an estimate
13334-
// because it has no better number to give, while here a real
13335-
// number was available and the caller declined it. Handing back
13336-
// a plausible-looking guess under those circumstances is how a
13332+
// honest one: a real number was available and the caller
13333+
// declined it. Handing back a plausible-looking guess is how a
1333713334
// page-local estimate ends up rendered as a record count.
1333813335
//
1333913336
// `hasMore` is still answered, from the page alone: a FULL page
13340-
// means there may be more. Same page-local rule the search
13341-
// branch uses, and it never over-reports the data — it can only
13342-
// say "maybe more" on an exactly-full last page.
13337+
// means there may be more. It never over-reports the data — it
13338+
// can only say "maybe more" on an exactly-full last page.
1334313339
hasMore = records.length === pageLimit;
1334413340
total = undefined;
13345-
} else if (countable) {
13341+
} else {
1334613342
// [#10757] `counted` is a separate, always-assigned local so
1334713343
// `hasMore` below compares against a `number`: `total` became
1334813344
// optional when `$count=false` gained the right to omit it, and
@@ -13351,6 +13347,8 @@ export class ObjectStackProtocolImplementation implements
1335113347
try {
1335213348
counted = await this.engine.count(request.object, {
1335313349
where: options.where,
13350+
search: options.search,
13351+
searchFields: options.searchFields,
1335413352
context: options.context,
1335513353
} as any);
1335613354
} catch {
@@ -13360,9 +13358,6 @@ export class ObjectStackProtocolImplementation implements
1336013358
}
1336113359
total = counted;
1336213360
hasMore = pageOffset + records.length < counted;
13363-
} else {
13364-
hasMore = records.length === pageLimit;
13365-
total = pageOffset + records.length + (hasMore ? 1 : 0);
1336613361
}
1336713362
}
1336813363
return {

‎packages/metadata-protocol/src/protocol.zero-limit-total.test.ts‎

Lines changed: 8 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -30,9 +30,9 @@
3030
* count for a zero-row page would pass (1) and fail here.
3131
* 3. The two neighbouring arms keep their meaning for a zero-row page:
3232
* `$count=false` still issues no COUNT and omits `total` (it no longer
33-
* reports the page's `0` as if it were a count), and `search` still issues
34-
* no COUNT — `engine.count()` cannot reproduce a search, which is why that
35-
* arm reports a page-local estimate at every page size.
33+
* reports the page's `0` as if it were a count), and a `search` is counted
34+
* like any other filter (#22790: the COUNT carries the search the rows
35+
* were read with).
3636
* 4. Absent a limit nothing moves: no COUNT, `total` is the set's length.
3737
*/
3838

@@ -149,10 +149,12 @@ describe('[#22588] the neighbouring arms keep their meaning for a zero-row page'
149149
expect('total' in result).toBe(false);
150150
});
151151

152-
it('search still issues no COUNT — engine.count() cannot reproduce a search', async () => {
153-
const { result, count } = await list({ $top: '0', $search: 't' });
152+
it('a search is counted, with the search the rows were read with (#22790)', async () => {
153+
const { result, count, find } = await list({ $top: '0', $search: 't' });
154154

155-
expect(count).not.toHaveBeenCalled();
155+
expect(count).toHaveBeenCalledTimes(1);
156+
expect((count.mock.calls[0] as unknown[])[1]).toMatchObject({ search: 't' });
157+
expect((find.mock.calls[0] as unknown[])[1]).toMatchObject({ search: 't' });
156158
expect(result.records).toEqual([]);
157159
});
158160

‎packages/objectql/src/engine.ts‎

Lines changed: 24 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -623,12 +623,13 @@ const ENGINE_DELETE_OPTION_KEYS: ReadonlySet<string> = new Set([
623623
'context', 'where', 'multi',
624624
...ENGINE_DRIVER_PASSTHROUGH_KEYS,
625625
]);
626-
const ENGINE_COUNT_OPTION_KEYS: ReadonlySet<string> = new Set(['context', 'where']);
627-
// `search` / `searchFields` are READ by `aggregate` through the same ADR-0061
628-
// expander `find` uses ({@link ObjectQL.expandSearchOnAggregateOptions}), so a
629-
// grouped answer under a search is the grouping of the searched rows. Before
630-
// `EngineAggregateOptionsSchema` declared them they were refused here, and the
631-
// one wire path to this verb left them out — the unsearched groups answered.
626+
// `search` / `searchFields` are READ by `count` and `aggregate` through the
627+
// same ADR-0061 expander `find` uses ({@link ObjectQL.expandSearchOnOptions}),
628+
// so a count or a grouped answer under a search is over the searched rows.
629+
// Before their schemas declared them they were refused here, and the wire paths
630+
// left them out: the unsearched groups answered, and a paged list's `total`
631+
// under a search was a page-local estimate (#22790).
632+
const ENGINE_COUNT_OPTION_KEYS: ReadonlySet<string> = new Set(['context', 'where', 'search', 'searchFields']);
632633
const ENGINE_AGGREGATE_OPTION_KEYS: ReadonlySet<string> = new Set([
633634
'context', 'where', 'groupBy', 'aggregations', 'having', 'timezone',
634635
'search', 'searchFields',
@@ -12729,13 +12730,15 @@ export class ObjectQL implements IObjectQLEngine {
1272912730
}
1273012731

1273112732
/**
12732-
* The aggregate verb's door into {@link expandSearchOnAst} — ⛔ not a second
12733-
* expander. `QuerySchema.search` (ADR-0061 D1) is declared beside `groupBy` /
12734-
* `aggregations` with no carve-out, so a grouped read under a search groups
12735-
* the SEARCHED rows: the same cross-field `$or`, resolved from the same
12736-
* server-side searchable set and the same `searchFields` narrowing, ANDed
12737-
* with `where` exactly as on `find` — before the security middlewares run,
12738-
* so an injected RLS predicate composes with it the same way on both verbs.
12733+
* The aggregate and count verbs' door into {@link expandSearchOnAst} — ⛔ not
12734+
* a second expander. `QuerySchema.search` (ADR-0061 D1) is declared beside
12735+
* `groupBy` / `aggregations` with no carve-out, so a grouped read under a
12736+
* search groups the SEARCHED rows, and a count under one counts them (#22790:
12737+
* `findData`'s paged `total`): the same cross-field `$or`, resolved from the
12738+
* same server-side searchable set, `internal: true` exclusion and
12739+
* `searchFields` narrowing, ANDed with `where` exactly as on `find` — before
12740+
* the security middlewares run, so an injected RLS predicate composes with it
12741+
* the same way on every verb.
1273912742
*
1274012743
* The expansion runs on a carrier AST holding only the three keys it reads,
1274112744
* and the result is a COPY of `query` with `where` replaced and the two
@@ -12744,16 +12747,13 @@ export class ObjectQL implements IObjectQLEngine {
1274412747
* `opCtx.options` unexpanded. The common path — no search key — returns the
1274512748
* same reference and allocates nothing.
1274612749
*/
12747-
private expandSearchOnAggregateOptions(
12748-
object: string,
12749-
query: EngineAggregateOptions,
12750-
): EngineAggregateOptions {
12750+
private expandSearchOnOptions<T extends EngineAggregateOptions | EngineCountOptions>(object: string, query: T): T {
1275112751
if (query.search === undefined && query.searchFields === undefined) return query;
1275212752
const { search, searchFields, ...rest } = query;
1275312753
const carrier = { object, where: rest.where, search, searchFields } as QueryAST;
1275412754
this.expandSearchOnAst(carrier, this._registry.getObject(object));
12755-
if (carrier.where === undefined) return rest;
12756-
return { ...rest, where: carrier.where };
12755+
if (carrier.where === undefined) return rest as T;
12756+
return { ...rest, where: carrier.where } as T;
1275712757
}
1275812758

1275912759
/**
@@ -18757,6 +18757,10 @@ export class ObjectQL implements IObjectQLEngine {
1875718757
query = foldEngineOptionAliases(object, 'count', query, ENGINE_WHERE_SLOTS);
1875818758
rejectUnknownEngineOptions(object, 'count', query, ENGINE_COUNT_OPTION_KEYS);
1875918759
query = lowerWhereFilterArray(object, 'count', query, this._registry.getObject(object), this.relatedSchemaOf);
18760+
// [#22790] ADR-0061 `search` → the rows are searched before they are
18761+
// counted, by the one expander `find` runs, at the same point in the
18762+
// sequence — so the count is the number of rows `find` returns for the bag.
18763+
if (query) query = this.expandSearchOnOptions(object, query);
1876018764
const driver = this.getDriver(object);
1876118765

1876218766
// The AST must ride on the opCtx so the security/sharing middlewares can
@@ -18892,7 +18896,7 @@ export class ObjectQL implements IObjectQLEngine {
1889218896
// ADR-0061 `search` → the rows are searched BEFORE they are grouped, by
1889318897
// the one expander `find` runs, at the same point in the sequence (after
1889418898
// the `where` doors above, before the AST is built and tokens resolve).
18895-
query = this.expandSearchOnAggregateOptions(object, query);
18899+
query = this.expandSearchOnOptions(object, query);
1889618900
this.rejectCredentialAggregation(object, query);
1889718901
// [#20783] …and a `groupBy` entry naming a structured-JSON field (`json`,
1889818902
// `composite`, `address`, …) is refused `INVALID_FIELD` / 400 here, before

‎packages/objectql/src/protocol-data.test.ts‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -287,14 +287,16 @@ describe('ObjectStackProtocolImplementation - Data Operations', () => {
287287
expect(result.hasMore).toBe(false);
288288
});
289289

290-
it('skips count for search queries and estimates hasMore from a full page', async () => {
291-
// engine.count() can't reproduce a $search, so we must not call it; a
292-
// full page (length === limit) implies there may be more.
290+
it('counts a search query under the same search, and reports that total (#22790)', async () => {
291+
// The engine expands `search` through the one expander `find` uses,
292+
// so the total is a count, never a page-local estimate.
293293
mockEngine.find.mockResolvedValue(Array.from({ length: 10 }, (_, i) => ({ id: `r${i}` })));
294+
mockEngine.count.mockResolvedValue(75);
294295

295296
const result = await protocol.findData({ object: 'task', query: { $top: 10, $search: 'foo' } });
296297

297-
expect(mockEngine.count).not.toHaveBeenCalled();
298+
expect(mockEngine.count).toHaveBeenCalledWith('task', expect.objectContaining({ search: 'foo' }));
299+
expect(result.total).toBe(75);
298300
expect(result.hasMore).toBe(true);
299301
});
300302
});

0 commit comments

Comments
 (0)