diff --git a/.changeset/typed-collection-filters.md b/.changeset/typed-collection-filters.md new file mode 100644 index 00000000..a8fcc330 --- /dev/null +++ b/.changeset/typed-collection-filters.md @@ -0,0 +1,7 @@ +--- +'@haverstack/core': minor +--- + +A collection's `filter.content`, `filter.contentPresent` and `sort.contentField` are typed from its content type: a typo'd field, a value outside an enum, an object value (which never matches) or a nested sort field no longer compiles. Adds `ContentPathOf` and `SortableFieldOf`. + +A collection query with a content filter or sort reaches only records stored at the handle's version, since a predicate in the handle's shape cannot be translated into another version's. Its first page sets `skipped.older` or `skipped.newer` when records at another version match the rest of the query. diff --git a/docs/spec/data-model.md b/docs/spec/data-model.md index f47dbac8..af8e6df9 100644 --- a/docs/spec/data-model.md +++ b/docs/spec/data-model.md @@ -418,7 +418,7 @@ A read types an `enum` field as its listed values or any other string (`'want' | **A misfit is a record of the family the handle cannot represent**: one stored at a version newer than the handle's (`reason: 'newer-version'`), which cannot be migrated down. A misfit is reported, never thrown for a page: -- `query()` returns `{ records, misfits, cursor }`. Misfits do not count as records, so a page can hold fewer than `limit` records with a cursor still set: `cursor === null` ends the results, a short page does not. The page and its cursor survive one record the app cannot read. +- `query()` returns `{ records, misfits, cursor }`, and `skipped` on the first page. Misfits do not count as records, so a page can hold fewer than `limit` records with a cursor still set: `cursor === null` ends the results, a short page does not. The page and its cursor survive one record the app cannot read. - `get()` is a read the app asked for by ID, so it throws `MisfitError`, carrying the `reason` and the record as stored. - `subscribe()` delivers a change whose record does not fit with `misfit` in place of `record`. @@ -426,7 +426,7 @@ A record at an older version that no passed migration carries to the handle's is **A subscription covers the family.** A change's `record`, when present, is migrated to the handle's version; the change's own `typeId` stays the stored one, as `RecordChange.typeId` documents. A change whose record has `deletedAt` set carries neither `record` nor `misfit`, on either surface, and neither does a change that [migrates the record out](#type-migrations) of the family, which the filter matches by its previous `typeId`. The change filter takes every key but `typeId` and `baseId`. -**Content filters and sorts apply to content as stored.** The adapter runs them before any record is migrated, so a field a migration renamed or derived is matched and ordered under its stored name: filtering an `@2` collection on a field `@1` spelled differently does not find `@1` records. `query()` takes neither `typeId` nor `presentAt`, which would change which records a page holds or the shape they arrive in, and drops either if an untyped caller passes it; `subscribe()` drops a filter's `typeId` the same way. +**A content filter or sort reaches only records stored at the handle's version.** The adapter matches and orders content as stored, before any record is migrated, and a migration is an opaque function, so a predicate written in the handle's shape cannot be translated into an older version's. Run across the family, a field a migration renamed, derived or remapped would silently miss the older records or misplace them. So a `query()` whose `filter.content` or `filter.contentPresent` is non-empty, or whose sort is a `contentField`, reads only records whose `typeId` is the handle's, even for a field no migration touched, since core cannot tell which fields those are. The untyped `query({ filter: { baseId, content } })` still matches across the family, returning records as stored. The first page of such a query sets `skipped.older` or `skipped.newer` when records at an older or newer version match the rest of the query: the results may then be missing records the filter was meant to find. A later page carries no `skipped`, since the flags depend on the query, not the page. Each is one `limit: 1` read, run beside the first page's own, and a scoped read that comes back empty with a cursor still set counts as a match rather than reading on, so a flag can be set when nothing was skipped but is never clear when something was. Mixed versions are a lasting state, not a backlog, since an app at an older version may keep writing its own, so the app decides what `skipped.older` means to it; `migrateAll()` clears it only until the next such write. `skipped.newer` says the app is behind, which an unfiltered page reports as misfits; a narrowed page carries none. A query with no content keys reads the whole family. The keys are typed against the handle's version: `filter.content` and `filter.contentPresent` take the content's paths (`ContentPathOf`), through declared objects and arrays, with any suffix below an `open` object or array, and a value of the field's type as a write takes it (an array's element type, since matching is element-wise) or `null`. A path reaching an object takes only `null`, since an adapter compares the value a path reaches and an object never equals one; a value below an `open` node is any scalar. Typing values as writes take them means an enum value the handle does not list is filtered for with a cast, the price of refusing a misspelled one; `sort.contentField` takes only a top-level scalar (`SortableFieldOf`). Path typing stops six segments deep and accepts any suffix below that, a limit of the compiler, not of the [32-segment path](#nested-content-paths). `query()` takes neither `typeId` nor `presentAt`, which would change which records a page holds or the shape they arrive in, and drops either if an untyped caller passes it; `subscribe()` drops a filter's `typeId` the same way. **Every verb that returns a record returns it as `get()` would.** The write already returned the record, so narrowing it costs no round trip, and a `restoreVersion()` to a pre-migration snapshot reads back migrated, like any older record. A write is checked against the read it already makes, before anything lands: a record outside the family is refused with `StackNotFoundError`, the answer `get()` gives, and so is a `restoreVersion()` to a snapshot taken before the record migrated into the family, which is not a version of a record in the collection. A content write to a record stored at a newer version is refused with `MisfitError` before anything is written, since its patch would be validated against a schema the handle does not know. Any other write that leaves a misfit lands, then throws `MisfitError` carrying the stored record, so the app knows the write landed. A deleted record, or a stale `ifVersion`, is refused first, as the untyped write refuses it. diff --git a/packages/core/src/collection.ts b/packages/core/src/collection.ts index 9ddf9676..44118ba7 100644 --- a/packages/core/src/collection.ts +++ b/packages/core/src/collection.ts @@ -33,11 +33,14 @@ import type { ChangeFilter, DataAssociation, IfVersionOptions, + NativeSortField, RecordChange, RecordChangeSet, RecordFilter, RecordId, StackQuery, + StackType, + TypeId, StackRecord, SubscribeOptions, Unsubscribe, @@ -95,16 +98,88 @@ export type CollectionPage = { records: CollectionRecord[]; misfits: Misfit[]; cursor: string | null; + /** + * A content filter or sort reaches only records at the handle's version. + * Each flag is set when records at an older or newer version match, or + * may match, the rest of the query: the results may be missing records + * the filter was meant to find. On the first page only, since it depends + * on the query rather than the page. See docs/spec/data-model.md § Type + * handles. + */ + skipped?: { older: boolean; newer: boolean }; }; +/** + * Path typing stops at this depth and accepts any suffix below it, since a + * recursive path type otherwise exceeds the compiler's instantiation limit. + * A typing limit, not a query limit: see docs/spec/data-model.md § Nested + * content paths. + */ +type PathDepth = [never, 0, 1, 2, 3, 4]; + +/** What a filter value can equal: an adapter compares the value a path reaches. */ +type Leaf = string | number | boolean; + +/** + * Each content path of `V` reached at `P`, paired with the value a filter + * compares there. An array is matched element-wise, so it is its element. + * An object never equals a filter value, so its path takes only `null`. + */ +type PathEntry

= [D] extends [never] + ? [P | `${P}.${string}`, Leaf] + : V extends readonly (infer E)[] + ? unknown extends E + ? [P | `${P}.${string}`, Leaf] + : PathEntry + : V extends object + ? string extends keyof V + ? [P, null] | [`${P}.${string}`, Leaf] + : [P, null] | FieldEntry + : [P, V]; + +/** `Required[K]` rather than `NonNullable`, which would leak into errors. */ +type FieldEntry = [D] extends [never] + ? [`${Prefix}${string}`, Leaf] + : { + [K in keyof C & string]-?: PathEntry<`${Prefix}${K}`, Required[K], D>; + }[keyof C & string]; + +/** + * Every content path of `C`: its fields, and dotted paths through declared + * objects and arrays. Below an `open` object or array, or six segments + * deep, any suffix is a path. + */ +export type ContentPathOf = FieldEntry[0]; + +/** The top-level scalar fields of `C`, the ones a content sort reaches. */ +export type SortableFieldOf = string extends keyof C + ? string + : { + [K in keyof C & string]-?: Required[K] extends string | number | boolean ? K : never; + }[keyof C & string]; + /** * A collection reads live records of its own family only, as stored, so * none of `typeId`, `baseId`, `includeDeleted` or `presentAt` is offered. - * `C` is reserved for typing content filters and sorts. + * Content filter keys and the sort field are checked against `C`, and + * reach only records stored at the handle's version. See + * docs/spec/data-model.md § Type handles. */ -// eslint-disable-next-line @typescript-eslint/no-unused-vars -export type CollectionQuery = Omit & { - filter?: Omit; +export type CollectionQuery = Omit & { + filter?: Omit< + RecordFilter, + 'typeId' | 'baseId' | 'includeDeleted' | 'content' | 'contentPresent' + > & { + content?: { [E in FieldEntry as E[0]]?: E[1] | null }; + contentPresent?: ContentPathOf[]; + }; + sort?: + | { field: NativeSortField; contentField?: never; direction?: 'asc' | 'desc' } + | { + field?: never; + contentField: SortableFieldOf; + direction?: 'asc' | 'desc'; + }; }; /** The family is the collection's, so neither `typeId` nor `baseId` is a filter key. */ @@ -202,15 +277,26 @@ const withoutTypeId = (filter: F | undefined): Omit + Object.keys(filter?.content ?? {}).length > 0 || + (filter?.contentPresent?.length ?? 0) > 0 || + sort?.contentField !== undefined; + /** * The collection over `handle`, shared by `Stack.collection()` and * `ScopedStack.collection()`. `registry` is the opening `Stack`'s, which - * the app's own process filled. + * the app's own process filled. `listTypes` names the family's versions, + * which `StackClient` does not offer. */ export const createCollection = ( client: StackClient, registry: MigrationRegistry, handle: TypeHandle, + listTypes: () => Promise, ): Collection> => { type C = ContentOf; if (handle.migratesFrom !== undefined && !registry.path(handle.migratesFrom, handle.id)) { @@ -293,6 +379,34 @@ export const createCollection = ( return narrow(written, true); }; + /** + * Whether a record stored at one of `typeIds` matches, or may match, + * `filter` without its content keys. A scoped page can be empty with a + * cursor set, and reading on is unbounded, so that page counts as a match. + */ + const anyAt = async (typeIds: TypeId[], filter: RecordFilter): Promise => { + if (typeIds.length === 0) return false; + const { content: _content, contentPresent: _present, ...rest } = filter; + const page = await client.query({ filter: { ...rest, typeId: typeIds }, limit: 1 }); + return page.records.length > 0 || page.cursor !== null; + }; + + const skippedVersions = async ( + filter: RecordFilter, + ): Promise['skipped']>> => { + const older: TypeId[] = []; + const newer: TypeId[] = []; + for (const type of await listTypes()) { + if (type.baseId !== handle.baseId || type.version === handleVersion) continue; + (type.version < handleVersion ? older : newer).push(type.id); + } + const [skippedOlder, skippedNewer] = await Promise.all([ + anyAt(older, filter), + anyAt(newer, filter), + ]); + return { older: skippedOlder, newer: skippedNewer }; + }; + const mutate = async ( id: RecordId, changes: RecordChangeSet, @@ -326,11 +440,28 @@ export const createCollection = ( // Stripped as well as untyped: either would change which records a // page holds, or the shape they arrive in. const { presentAt: _presentAt, filter, ...rest } = query as StackQuery; - const { records, cursor } = await client.query({ - ...rest, - filter: { ...withoutTypeId(filter), baseId: handle.baseId }, - }); - const page: CollectionPage = { records: [], misfits: [], cursor }; + const narrowed = readsContent(filter, rest.sort); + // A concrete typeId needs no family lookup; baseId does. + const { baseId: _baseId, ...unversioned } = withoutTypeId(filter); + const [{ records, cursor }, skipped] = await Promise.all([ + client.query({ + ...rest, + filter: narrowed + ? { ...unversioned, typeId: handle.id } + : { ...unversioned, baseId: handle.baseId }, + }), + rest.cursor !== undefined + ? undefined + : narrowed + ? skippedVersions(unversioned) + : { older: false, newer: false }, + ]); + const page: CollectionPage = { + records: [], + misfits: [], + cursor, + ...(skipped && { skipped }), + }; for (const stored of records) { const result = classify(stored); if ('misfit' in result) page.misfits.push(result.misfit); diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts index 52a0963f..df53481f 100644 --- a/packages/core/src/index.ts +++ b/packages/core/src/index.ts @@ -61,7 +61,9 @@ export type { CollectionChangeSet, CollectionSubscribeOptions, ContentPatch, + ContentPathOf, Misfit, + SortableFieldOf, StoredContent, } from './collection.js'; diff --git a/packages/core/src/scoped-stack/scoped-stack.ts b/packages/core/src/scoped-stack/scoped-stack.ts index 93e105c6..1fd28a4c 100644 --- a/packages/core/src/scoped-stack/scoped-stack.ts +++ b/packages/core/src/scoped-stack/scoped-stack.ts @@ -186,7 +186,7 @@ export class ScopedStack implements StackClient { collection(handle: TypeHandle): Collection> { this.assertStackOpen(); - return createCollection(this, this.migrations, handle); + return createCollection(this, this.migrations, handle, () => this.stack.listTypes()); } /** diff --git a/packages/core/src/stack/stack.ts b/packages/core/src/stack/stack.ts index 618b3bef..e2cf121a 100644 --- a/packages/core/src/stack/stack.ts +++ b/packages/core/src/stack/stack.ts @@ -395,7 +395,7 @@ export class Stack implements StackClient { collection(handle: TypeHandle): Collection> { this.assertOpen(); - return createCollection(this, this.migrations, handle); + return createCollection(this, this.migrations, handle, () => this.listTypes()); } // ------------------------------------------------------- diff --git a/packages/core/tests/collection.test.ts b/packages/core/tests/collection.test.ts index 62ae42e7..e9c45a26 100644 --- a/packages/core/tests/collection.test.ts +++ b/packages/core/tests/collection.test.ts @@ -2,14 +2,17 @@ import { describe, test, expect, expectTypeOf, beforeEach, vi } from 'vitest'; import { Stack } from '../src/stack/stack.js'; import { MemoryAdapter } from '../src/testing.js'; import { migration, typeHandle } from '../src/type-handle.js'; -import type { ContentOf, UnlistedValue } from '../src/type-handle.js'; +import type { ContentOf, ReadonlyTypeSchema, UnlistedValue } from '../src/type-handle.js'; import { MisfitError } from '../src/collection.js'; import type { Collection, CollectionChange, CollectionPage, + CollectionQuery, CollectionRecord, ContentPatch, + ContentPathOf, + SortableFieldOf, StoredContent, } from '../src/collection.js'; import { @@ -266,6 +269,66 @@ describe.each([ expect(records.map((r) => r.content.title)).toEqual(['Emma']); }); + test("a content filter reaches only the handle's version and flags both sides", async () => { + const book = await books.create({ title: 'Dune', status: 'finished' }); + await createV1('Emma', 'done'); + await stack.create(BookV3.id, { title: 'Ulysses', status: 'finished' }); + const page = await books.query({ filter: { content: { status: 'finished' } } }); + expect(page.records.map((r) => r.id)).toEqual([book.id]); + expect(page.misfits).toEqual([]); + expect(page.skipped).toEqual({ older: true, newer: true }); + }); + + test('a content sort and a presence filter narrow the same way', async () => { + const book = await books.create({ title: 'Dune', status: 'want' }); + await createV1(); + const sorted = await books.query({ sort: { contentField: 'title' } }); + expect(sorted.records.map((r) => r.id)).toEqual([book.id]); + expect(sorted.skipped).toEqual({ older: true, newer: false }); + const present = await books.query({ filter: { contentPresent: ['title'] } }); + expect(present.records.map((r) => r.id)).toEqual([book.id]); + expect(present.skipped).toEqual({ older: true, newer: false }); + }); + + test('skipped is reported on the first page only', async () => { + await books.create({ title: 'Dune', status: 'want' }); + await books.create({ title: 'Emma', status: 'want' }); + await createV1(); + const first = await books.query({ filter: { content: { status: 'want' } }, limit: 1 }); + expect(first.skipped).toEqual({ older: true, newer: false }); + const second = await books.query({ + filter: { content: { status: 'want' } }, + limit: 1, + cursor: first.cursor!, + }); + expect(second.records).toHaveLength(1); + expect(second).not.toHaveProperty('skipped'); + }); + + test('skipped applies the rest of the filter', async () => { + await books.create({ title: 'Dune', status: 'want' }); + await createV1(); + await stack.create(BookV3.id, { title: 'Ulysses', status: 'want' }); + const page = await books.query({ filter: { content: { status: 'want' }, tags: ['fantasy'] } }); + expect(page.skipped).toEqual({ older: false, newer: false }); + }); + + test('a query without content keys reads the whole family and flags nothing', async () => { + await books.create({ title: 'Dune', status: 'want' }); + await createV1(); + const page = await books.query({ filter: { content: {}, contentPresent: [] } }); + expect(page.records).toHaveLength(2); + expect(page.skipped).toEqual({ older: false, newer: false }); + }); + + test('an undefined content value is a filter, as an adapter reads it', async () => { + await books.create({ title: 'Dune', status: 'want' }); + await createV1(); + const page = await books.query({ filter: { content: { pages: undefined } } }); + expect(page.records).toHaveLength(1); + expect(page.skipped).toEqual({ older: true, newer: false }); + }); + test('query() drops typeId and presentAt from an untyped caller', async () => { const book = await books.create({ title: 'Dune', status: 'want' }); const newer = await stack.create(BookV3.id, { title: 'Emma', status: 'want' }); @@ -580,3 +643,152 @@ describe('a collection under delegation', () => { expect((await books.patchContent(old.id, { pages: 412 })).content.pages).toBe(412); }); }); + +describe('content filter and sort typing', () => { + const Contact = typeHandle({ + id: 'com.example.people/contact@1', + name: 'Contact', + schema: { + name: { kind: 'string', required: true }, + status: { kind: 'enum', values: ['active', 'archived'], required: true }, + age: { kind: 'number' }, + address: { + kind: 'object', + properties: { + city: { kind: 'string' }, + geo: { kind: 'object', properties: { lat: { kind: 'number' } } }, + }, + }, + emails: { + kind: 'array', + items: { kind: 'object', properties: { value: { kind: 'string', required: true } } }, + }, + tags: { kind: 'array', items: { kind: 'string' } }, + extra: { kind: 'object', open: true }, + bag: { kind: 'array', open: true }, + }, + }); + type Q = CollectionQuery>; + + test('derives paths and sortable fields from the content type', () => { + type C = ContentOf; + expectTypeOf>().toEqualTypeOf<'name' | 'status' | 'age'>(); + expectTypeOf>().toEqualTypeOf< + | 'name' + | 'status' + | 'age' + | 'address' + | 'address.city' + | 'address.geo' + | 'address.geo.lat' + | 'emails' + | 'emails.value' + | 'tags' + | 'extra' + | `extra.${string}` + | 'bag' + | `bag.${string}` + >(); + }); + + test('accepts declared paths, element values, any suffix below an open node, and null', async () => { + const stack = await Stack.open( + await MemoryAdapter.open({ ownerEntityId: OWNER, timezone: 'UTC' }), + ); + await stack.defineType(Contact); + const contacts = stack.collection(Contact); + const ada = await contacts.create({ + name: 'Ada', + status: 'active', + address: { city: 'London', geo: { lat: 51.5 } }, + emails: [{ value: 'ada@example.com' }], + tags: ['starred'], + extra: { source: { kind: 'import' } }, + bag: [1], + }); + await contacts.create({ name: 'Bea', status: 'archived', age: 40 }); + const { records } = await contacts.query({ + filter: { + content: { + status: 'active', + 'address.city': 'London', + 'address.geo.lat': 51.5, + 'emails.value': 'ada@example.com', + tags: 'starred', + 'extra.source.kind': 'import', + bag: 1, + age: null, + }, + contentPresent: ['address.city', 'extra.source'], + }, + sort: { contentField: 'name', direction: 'desc' }, + }); + expect(records.map((r) => r.id)).toEqual([ada.id]); + }); + + test('refuses a typo, a value outside an enum, a wrong nested type and a path through a scalar', () => { + // @ts-expect-error a typo'd field + const typo: Q = { filter: { content: { stauts: 'active' } } }; + // @ts-expect-error a value outside the enum + const badEnum: Q = { filter: { content: { status: 'actvie' } } }; + // @ts-expect-error a wrong value type at a nested path + const badNested: Q = { filter: { content: { 'address.geo.lat': 'north' } } }; + // @ts-expect-error a path through a scalar + const throughScalar: Q = { filter: { content: { 'name.first': 'Ada' } } }; + // @ts-expect-error an object never equals a filter value + const object: Q = { filter: { content: { address: { city: 'Lisbon' } } } }; + // @ts-expect-error nor does an element of an array of objects + const element: Q = { filter: { content: { emails: { value: 'ada@example.com' } } } }; + // @ts-expect-error nor an open object itself + const open: Q = { filter: { content: { extra: {} } } }; + // @ts-expect-error a typo'd presence path + const badPresent: Q = { filter: { contentPresent: ['adress.city'] } }; + expect([ + typo, + badEnum, + badNested, + throughScalar, + object, + element, + open, + badPresent, + ]).toHaveLength(8); + }); + + test('sorts by top-level scalars only', () => { + // @ts-expect-error a nested field + const nested: Q = { sort: { contentField: 'address.city' } }; + // @ts-expect-error an object field + const object: Q = { sort: { contentField: 'address' } }; + // @ts-expect-error a typo'd field + const typo: Q = { sort: { contentField: 'nmae' } }; + expect([nested, object, typo]).toHaveLength(3); + }); + + test('types paths deeper than the typing limit without exceeding the compiler', () => { + type Obj

= { kind: 'object'; properties: P }; + type DeepQ = CollectionQuery< + ContentOf<{ + a: Obj<{ + b: Obj<{ + c: Obj<{ d: Obj<{ e: Obj<{ f: Obj<{ g: Obj<{ h: { kind: 'number' } }> }> }> }> }>; + }>; + }>; + }> + >; + const ok: DeepQ = { filter: { content: { 'a.b.c.d.e': null, 'a.b.c.d.e.f.anything': 1 } } }; + // @ts-expect-error the sixth segment is still typed + const sixth: DeepQ = { filter: { content: { 'a.b.c.d.e.x': 1 } } }; + // @ts-expect-error a typo above the limit is still caught + const typo: DeepQ = { filter: { content: { 'a.x': 1 } } }; + expect([ok, sixth, typo]).toHaveLength(3); + }); + + test('an untyped handle takes any path and sort field', () => { + const ok: CollectionQuery> = { + filter: { content: { anything: 1, 'a.b': 'x' }, contentPresent: ['q.r'] }, + sort: { contentField: 'x' }, + }; + expect(ok).toBeDefined(); + }); +});