feat(core): type a collection's content filters and sorts from its content type

.changeset/typed-collection-filters.md

@@ -0,0 +1,7 @@
+---
+'@haverstack/core': minor
+---
+
+A collection's `filter.content`, `filter.contentPresent` and `sort.contentField` are typed from its content type: a typo'd field, a value outside an enum, an object value (which never matches) or a nested sort field no longer compiles. Adds `ContentPathOf<C>` and `SortableFieldOf<C>`.
+
+A collection query with a content filter or sort reaches only records stored at the handle's version, since a predicate in the handle's shape cannot be translated into another version's. Its first page sets `skipped.older` or `skipped.newer` when records at another version match the rest of the query.

docs/spec/data-model.md

@@ -418,15 +418,15 @@ A read types an `enum` field as its listed values or any other string (`'want' |
 
 **A misfit is a record of the family the handle cannot represent**: one stored at a version newer than the handle's (`reason: 'newer-version'`), which cannot be migrated down. A misfit is reported, never thrown for a page:
 
-- `query()` returns `{ records, misfits, cursor }`. Misfits do not count as records, so a page can hold fewer than `limit` records with a cursor still set: `cursor === null` ends the results, a short page does not. The page and its cursor survive one record the app cannot read.
+- `query()` returns `{ records, misfits, cursor }`, and `skipped` on the first page. Misfits do not count as records, so a page can hold fewer than `limit` records with a cursor still set: `cursor === null` ends the results, a short page does not. The page and its cursor survive one record the app cannot read.
 - `get()` is a read the app asked for by ID, so it throws `MisfitError`, carrying the `reason` and the record as stored.
 - `subscribe()` delivers a change whose record does not fit with `misfit` in place of `record`.
 
 A record at an older version that no passed migration carries to the handle's is not a misfit: that is the app's setup, and a read of it throws `StackMigrationError`, as it throws whatever a failing migration function throws. A subscription cannot throw at its emitter, so such a change arrives without its `record`, as it does wherever the emitter cannot supply one, and `get()` says why. `MisfitError` is raised only in the app's process and has no wire representation, so it sits [outside the taxonomy](./wire-format.md#the-taxonomy-root).
 
 **A subscription covers the family.** A change's `record`, when present, is migrated to the handle's version; the change's own `typeId` stays the stored one, as `RecordChange.typeId` documents. A change whose record has `deletedAt` set carries neither `record` nor `misfit`, on either surface, and neither does a change that [migrates the record out](#type-migrations) of the family, which the filter matches by its previous `typeId`. The change filter takes every key but `typeId` and `baseId`.
 
-**Content filters and sorts apply to content as stored.** The adapter runs them before any record is migrated, so a field a migration renamed or derived is matched and ordered under its stored name: filtering an `@2` collection on a field `@1` spelled differently does not find `@1` records. `query()` takes neither `typeId` nor `presentAt`, which would change which records a page holds or the shape they arrive in, and drops either if an untyped caller passes it; `subscribe()` drops a filter's `typeId` the same way.
+**A content filter or sort reaches only records stored at the handle's version.** The adapter matches and orders content as stored, before any record is migrated, and a migration is an opaque function, so a predicate written in the handle's shape cannot be translated into an older version's. Run across the family, a field a migration renamed, derived or remapped would silently miss the older records or misplace them. So a `query()` whose `filter.content` or `filter.contentPresent` is non-empty, or whose sort is a `contentField`, reads only records whose `typeId` is the handle's, even for a field no migration touched, since core cannot tell which fields those are. The untyped `query({ filter: { baseId, content } })` still matches across the family, returning records as stored. The first page of such a query sets `skipped.older` or `skipped.newer` when records at an older or newer version match the rest of the query: the results may then be missing records the filter was meant to find. A later page carries no `skipped`, since the flags depend on the query, not the page. Each is one `limit: 1` read, run beside the first page's own, and a scoped read that comes back empty with a cursor still set counts as a match rather than reading on, so a flag can be set when nothing was skipped but is never clear when something was. Mixed versions are a lasting state, not a backlog, since an app at an older version may keep writing its own, so the app decides what `skipped.older` means to it; `migrateAll()` clears it only until the next such write. `skipped.newer` says the app is behind, which an unfiltered page reports as misfits; a narrowed page carries none. A query with no content keys reads the whole family. The keys are typed against the handle's version: `filter.content` and `filter.contentPresent` take the content's paths (`ContentPathOf<C>`), through declared objects and arrays, with any suffix below an `open` object or array, and a value of the field's type as a write takes it (an array's element type, since matching is element-wise) or `null`. A path reaching an object takes only `null`, since an adapter compares the value a path reaches and an object never equals one; a value below an `open` node is any scalar. Typing values as writes take them means an enum value the handle does not list is filtered for with a cast, the price of refusing a misspelled one; `sort.contentField` takes only a top-level scalar (`SortableFieldOf<C>`). Path typing stops six segments deep and accepts any suffix below that, a limit of the compiler, not of the [32-segment path](#nested-content-paths). `query()` takes neither `typeId` nor `presentAt`, which would change which records a page holds or the shape they arrive in, and drops either if an untyped caller passes it; `subscribe()` drops a filter's `typeId` the same way.
 
 **Every verb that returns a record returns it as `get()` would.** The write already returned the record, so narrowing it costs no round trip, and a `restoreVersion()` to a pre-migration snapshot reads back migrated, like any older record. A write is checked against the read it already makes, before anything lands: a record outside the family is refused with `StackNotFoundError`, the answer `get()` gives, and so is a `restoreVersion()` to a snapshot taken before the record migrated into the family, which is not a version of a record in the collection. A content write to a record stored at a newer version is refused with `MisfitError` before anything is written, since its patch would be validated against a schema the handle does not know. Any other write that leaves a misfit lands, then throws `MisfitError` carrying the stored record, so the app knows the write landed. A deleted record, or a stale `ifVersion`, is refused first, as the untyped write refuses it.
 

packages/core/src/collection.ts

@@ -33,11 +33,14 @@ import type {
   ChangeFilter,
   DataAssociation,
   IfVersionOptions,
+  NativeSortField,
   RecordChange,
   RecordChangeSet,
   RecordFilter,
   RecordId,
   StackQuery,
+  StackType,
+  TypeId,
   StackRecord,
   SubscribeOptions,
   Unsubscribe,
@@ -95,16 +98,88 @@ export type CollectionPage<C> = {
   records: CollectionRecord<C>[];
   misfits: Misfit[];
   cursor: string | null;
+  /**
+   * A content filter or sort reaches only records at the handle's version.
+   * Each flag is set when records at an older or newer version match, or
+   * may match, the rest of the query: the results may be missing records
+   * the filter was meant to find. On the first page only, since it depends
+   * on the query rather than the page. See docs/spec/data-model.md § Type
+   * handles.
+   */
+  skipped?: { older: boolean; newer: boolean };
 };
 
+/**
+ * Path typing stops at this depth and accepts any suffix below it, since a
+ * recursive path type otherwise exceeds the compiler's instantiation limit.
+ * A typing limit, not a query limit: see docs/spec/data-model.md § Nested
+ * content paths.
+ */
+type PathDepth = [never, 0, 1, 2, 3, 4];
+
+/** What a filter value can equal: an adapter compares the value a path reaches. */
+type Leaf = string | number | boolean;
+
+/**
+ * Each content path of `V` reached at `P`, paired with the value a filter
+ * compares there. An array is matched element-wise, so it is its element.
+ * An object never equals a filter value, so its path takes only `null`.
+ */
+type PathEntry<P extends string, V, D extends number> = [D] extends [never]
+  ? [P | `${P}.${string}`, Leaf]
+  : V extends readonly (infer E)[]
+    ? unknown extends E
+      ? [P | `${P}.${string}`, Leaf]
+      : PathEntry<P, E, PathDepth[D]>
+    : V extends object
+      ? string extends keyof V
+        ? [P, null] | [`${P}.${string}`, Leaf]
+        : [P, null] | FieldEntry<V, `${P}.`, PathDepth[D]>
+      : [P, V];
+
+/** `Required<C>[K]` rather than `NonNullable`, which would leak into errors. */
+type FieldEntry<C, Prefix extends string, D extends number> = [D] extends [never]
+  ? [`${Prefix}${string}`, Leaf]
+  : {
+      [K in keyof C & string]-?: PathEntry<`${Prefix}${K}`, Required<C>[K], D>;
+    }[keyof C & string];
+
+/**
+ * Every content path of `C`: its fields, and dotted paths through declared
+ * objects and arrays. Below an `open` object or array, or six segments
+ * deep, any suffix is a path.
+ */
+export type ContentPathOf<C> = FieldEntry<C, '', 5>[0];
+
+/** The top-level scalar fields of `C`, the ones a content sort reaches. */
+export type SortableFieldOf<C> = string extends keyof C
+  ? string
+  : {
+      [K in keyof C & string]-?: Required<C>[K] extends string | number | boolean ? K : never;
+    }[keyof C & string];
+
 /**
  * A collection reads live records of its own family only, as stored, so
  * none of `typeId`, `baseId`, `includeDeleted` or `presentAt` is offered.
- * `C` is reserved for typing content filters and sorts.
+ * Content filter keys and the sort field are checked against `C`, and
+ * reach only records stored at the handle's version. See
+ * docs/spec/data-model.md § Type handles.
  */
-// eslint-disable-next-line @typescript-eslint/no-unused-vars
-export type CollectionQuery<C> = Omit<StackQuery, 'filter' | 'presentAt'> & {
-  filter?: Omit<RecordFilter, 'typeId' | 'baseId' | 'includeDeleted'>;
+export type CollectionQuery<C> = Omit<StackQuery, 'filter' | 'presentAt' | 'sort'> & {
+  filter?: Omit<
+    RecordFilter,
+    'typeId' | 'baseId' | 'includeDeleted' | 'content' | 'contentPresent'
+  > & {
+    content?: { [E in FieldEntry<C, '', 5> as E[0]]?: E[1] | null };
+    contentPresent?: ContentPathOf<C>[];
+  };
+  sort?:
+    | { field: NativeSortField; contentField?: never; direction?: 'asc' | 'desc' }
+    | {
+        field?: never;
+        contentField: SortableFieldOf<C>;
+        direction?: 'asc' | 'desc';
+      };
 };
 
 /** The family is the collection's, so neither `typeId` nor `baseId` is a filter key. */
@@ -202,15 +277,26 @@ const withoutTypeId = <F extends object>(filter: F | undefined): Omit<F, 'typeId
   return rest;
 };
 
+/**
+ * Whether a query reads content, which only records stored at the handle's
+ * version are guaranteed to hold in the handle's shape.
+ */
+const readsContent = (filter: RecordFilter | undefined, sort: StackQuery['sort']): boolean =>
+  Object.keys(filter?.content ?? {}).length > 0 ||
+  (filter?.contentPresent?.length ?? 0) > 0 ||
+  sort?.contentField !== undefined;
+
 /**
  * The collection over `handle`, shared by `Stack.collection()` and
  * `ScopedStack.collection()`. `registry` is the opening `Stack`'s, which
- * the app's own process filled.
+ * the app's own process filled. `listTypes` names the family's versions,
+ * which `StackClient` does not offer.
  */
 export const createCollection = <S extends ReadonlyTypeSchema>(
   client: StackClient,
   registry: MigrationRegistry,
   handle: TypeHandle<S>,
+  listTypes: () => Promise<StackType[]>,
 ): Collection<ContentOf<S>> => {
   type C = ContentOf<S>;
   if (handle.migratesFrom !== undefined && !registry.path(handle.migratesFrom, handle.id)) {
@@ -293,6 +379,34 @@ export const createCollection = <S extends ReadonlyTypeSchema>(
     return narrow(written, true);
   };
 
+  /**
+   * Whether a record stored at one of `typeIds` matches, or may match,
+   * `filter` without its content keys. A scoped page can be empty with a
+   * cursor set, and reading on is unbounded, so that page counts as a match.
+   */
+  const anyAt = async (typeIds: TypeId[], filter: RecordFilter): Promise<boolean> => {
+    if (typeIds.length === 0) return false;
+    const { content: _content, contentPresent: _present, ...rest } = filter;
+    const page = await client.query({ filter: { ...rest, typeId: typeIds }, limit: 1 });
+    return page.records.length > 0 || page.cursor !== null;
+  };
+
+  const skippedVersions = async (
+    filter: RecordFilter,
+  ): Promise<NonNullable<CollectionPage<C>['skipped']>> => {
+    const older: TypeId[] = [];
+    const newer: TypeId[] = [];
+    for (const type of await listTypes()) {
+      if (type.baseId !== handle.baseId || type.version === handleVersion) continue;
+      (type.version < handleVersion ? older : newer).push(type.id);
+    }
+    const [skippedOlder, skippedNewer] = await Promise.all([
+      anyAt(older, filter),
+      anyAt(newer, filter),
+    ]);
+    return { older: skippedOlder, newer: skippedNewer };
+  };
+
   const mutate = async (
     id: RecordId,
     changes: RecordChangeSet,
@@ -326,11 +440,28 @@ export const createCollection = <S extends ReadonlyTypeSchema>(
       // Stripped as well as untyped: either would change which records a
       // page holds, or the shape they arrive in.
       const { presentAt: _presentAt, filter, ...rest } = query as StackQuery;
-      const { records, cursor } = await client.query({
-        ...rest,
-        filter: { ...withoutTypeId(filter), baseId: handle.baseId },
-      });
-      const page: CollectionPage<C> = { records: [], misfits: [], cursor };
+      const narrowed = readsContent(filter, rest.sort);
+      // A concrete typeId needs no family lookup; baseId does.
+      const { baseId: _baseId, ...unversioned } = withoutTypeId(filter);
+      const [{ records, cursor }, skipped] = await Promise.all([
+        client.query({
+          ...rest,
+          filter: narrowed
+            ? { ...unversioned, typeId: handle.id }
+            : { ...unversioned, baseId: handle.baseId },
+        }),
+        rest.cursor !== undefined
+          ? undefined
+          : narrowed
+            ? skippedVersions(unversioned)
+            : { older: false, newer: false },
+      ]);
+      const page: CollectionPage<C> = {
+        records: [],
+        misfits: [],
+        cursor,
+        ...(skipped && { skipped }),
+      };
       for (const stored of records) {
         const result = classify(stored);
         if ('misfit' in result) page.misfits.push(result.misfit);

packages/core/src/index.ts

@@ -61,7 +61,9 @@ export type {
   CollectionChangeSet,
   CollectionSubscribeOptions,
   ContentPatch,
+  ContentPathOf,
   Misfit,
+  SortableFieldOf,
   StoredContent,
 } from './collection.js';
 

packages/core/src/scoped-stack/scoped-stack.ts

@@ -186,7 +186,7 @@ export class ScopedStack implements StackClient {
 
   collection<S extends ReadonlyTypeSchema>(handle: TypeHandle<S>): Collection<ContentOf<S>> {
     this.assertStackOpen();
-    return createCollection(this, this.migrations, handle);
+    return createCollection(this, this.migrations, handle, () => this.stack.listTypes());
   }
 
   /**

packages/core/src/stack/stack.ts

@@ -395,7 +395,7 @@ export class Stack implements StackClient {
 
   collection<S extends ReadonlyTypeSchema>(handle: TypeHandle<S>): Collection<ContentOf<S>> {
     this.assertOpen();
-    return createCollection(this, this.migrations, handle);
+    return createCollection(this, this.migrations, handle, () => this.listTypes());
   }
 
   // -------------------------------------------------------