fix(objectql)!: an unprojected read serves the declared fields, never an orphaned column

.changeset/21571-unprojected-read-declared-fields.md

@@ -0,0 +1,37 @@
+---
+'@objectstack/objectql': minor
+---
+
+fix(objectql)!: a read with no projection serves the object's declared fields and the platform's system columns, never a column no metadata declares (#21571)
+
+**BREAKING (narrowing)** — what a released read door serves shrinks. A column that
+no metadata declares, typically a field retired in an upgrade whose column additive
+schema sync leaves in the table until `os migrate apply --allow-destructive`, is no
+longer returned by any read through the engine.
+
+| read | before | now |
+| --- | --- | --- |
+| `POST /api/v1/data/:object/query` or `GET /api/v1/data/:object` with no `fields` | every column of the table, retired ones and their values included | the declared fields, the registry's system columns, `id`, `created_at`, `updated_at` |
+| `GET /api/v1/data/:object/:id`, export, search hits, the RPC dispatcher, `expand`ed records | the same whole row | the same declared set |
+| `engine.find` / `engine.findOne` in process (hooks, flows, plugins), no `fields` | the whole row | the declared set |
+| an explicit `fields` naming a declared field whose column does not exist yet (driver-sql retries `select('*')`) | the whole row, retired columns included | the declared set |
+| `POST /api/v1/data/:object/:id/clone` of a record whose table carries a retired column | refused `INVALID_FIELD` (the copy carried the retired column into the insert) | cloned |
+
+**Unchanged:** naming a retired column in `fields` still answers `400 INVALID_FIELD`
+on the data door. Declared fields keep their treatment: `internal: true` omission,
+credential masking, formula evaluation and the hidden `__search` strip run as
+before, and the registry-injected tenant, owner and audit columns are still served.
+No driver changed: the engine shapes the rows any driver returns, so the answer is
+the same on every driver and every door. Writes, and the rows a write returns, are
+not changed by this release.
+
+**If you still read a retired column's values** (for example a one-time conversion
+that copies the old columns into their replacement field): run that conversion
+BEFORE upgrading to this release, while the old field is still declared, or, once
+it lands, read the unmapped columns through the operator-only `os migrate` read
+(objectstack#21573). There is no flag that re-opens undeclared columns on a runtime
+door. An in-process reader that needs a column must declare it as a field.
+
+Clause-②: no (narrowing)
+
+<!-- adr-0087: not-required (no-migration-prescription) No authorable key, spelling, export, config field or stored shape is removed or renamed, and no stored row is read or rewritten: the change narrows which physical columns the engine's read verbs return, to the field map the metadata already declares. A retired column's values stay in the table; what an app does with them is an operator action stated above (convert before upgrading, or the operator-only os migrate read), not a FROM to TO mapping that objectstack migrate meta could apply. The other categories are closed on facts: the package publishes (not unpublished); no ADR-0087 id covers a read projection (not registered / already-registered); and the change is runtime behaviour, not a declaration (not runtime-interface-only / type-surface-only). -->

content/docs/data-modeling/queries.mdx

@@ -285,6 +285,13 @@ pattern the metadata-revision / flow-run / notification list endpoints already u
 }
 ```
 
+Without `fields`, a read returns the object's **declared** fields plus the platform's
+system columns (`id`, `created_at`, `updated_at`, and the tenant, owner and audit columns
+the registry adds). A column no metadata declares is never returned — for example one a
+retired field left in the table until `os migrate apply --allow-destructive` drops it — and
+naming it in `fields` on the data API is refused with `400 INVALID_FIELD`. To read such a column's values
+for a one-time conversion, run the conversion before the field is retired.
+
 ### Nested / Related Fields
 
 {/* os:check */}

packages/objectql/src/declared-read-columns.ts

@@ -0,0 +1,131 @@
+// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
+
+/**
+ * [#21571] The columns a read may serve: the object's DECLARED fields plus the
+ * platform-provisioned columns — one set, answered from the registry's field
+ * map, and the same set an explicit projection is judged against.
+ *
+ * ## The defect this closes
+ *
+ * A read that names no `fields` reached the driver with no projection, and
+ * every driver answers that with the whole row (`SELECT *` on SQL). A column
+ * that no metadata declares — a field retired in an upgrade, whose column
+ * additive schema sync leaves in the table until an operator runs a
+ * destructive apply — therefore rode back in every record body, on every door
+ * that reads through the engine (`POST /data/:object/query`, the list route,
+ * `GET /data/:object/:id`, export, search, the RPC dispatcher, a hook's or a
+ * flow's in-process read). Naming the same column in `fields` answered
+ * `400 INVALID_FIELD`: the platform knew it was not a field and served it
+ * anyway, outside any field-level rule, because there is no field to attach a
+ * rule to.
+ *
+ * ## Where it is decided, and why there
+ *
+ * Triage's ruling on #21571: the default projection of an unprojected read is
+ * decided ONCE, in the engine, so every driver and every door gets the same
+ * answer — not per driver, not per door. The declared field set is that
+ * projection; no allow-list of column names and no flag re-opens undeclared
+ * columns on a runtime door.
+ *
+ * The engine SHAPES THE ROWS the driver hands back, rather than pushing a
+ * projection down to the driver. Measured on driver-sql: its recovery ladder
+ * retries `select('*')` whenever a projected statement fails on an
+ * unresolvable column, so a pushed-down projection naming a declared field
+ * whose column does not exist yet (not migrated, or the read races a schema
+ * sync) would be answered by the whole row — orphaned columns included. An
+ * EXPLICIT projection reaches that rung today too. Shaping the returned rows
+ * holds whichever rung answered, and needs no driver edit.
+ *
+ * It runs on the rows as they arrive from the driver — before formula
+ * evaluation, `expand`, file-reference resolution and the `afterFind` hooks —
+ * so everything downstream of storage sees the declared record, the same scope
+ * `materializeDeclaredFields` gives every CEL surface. Keys added AFTER that
+ * (a formula's value, an expanded record, a hook's derived key) are the
+ * engine's or the hook's, not storage's, and are untouched. Declared fields
+ * keep their existing treatment: `internal: true` omission, credential
+ * masking and the `__search` companion strip all still run after the hooks.
+ *
+ * ## Where it deliberately has no opinion
+ *
+ * The rule the read and write doors already share: a door that cannot see the
+ * field map must not invent a verdict about it. A schema with no field map, an
+ * ARRAY field map (not checkable — `Object.keys` yields indices), or an EMPTY
+ * one (indistinguishable from an unpopulated map: a registered object always
+ * carries at least the injected system columns) leaves the rows exactly as the
+ * driver returned them.
+ *
+ * ## In-process readers of undeclared columns — measured before this landed
+ *
+ * The objectql, rest, runtime, plugin-auth, plugin-sharing, plugin-audit and
+ * service-automation suites ran with every undeclared key removed at this
+ * seam; no production reader needed one (the fallout was test fixtures). The
+ * operator reads that legitimately need a retired column's values go through
+ * the driver, never through this path: `os migrate plan`'s `unmapped_column`
+ * detection introspects the table, and `os migrate account-issuer` reads
+ * `sys_account` through the driver the engine routes it to.
+ */
+
+/**
+ * The columns the platform provisions on every physical table without an
+ * author declaring them. `id` is the driver's primary key; the two audit
+ * timestamps are engine/driver-stamped. The registry injects the rest of the
+ * system columns (tenant, owner, the audit actors) INTO the field map, so they
+ * need no entry here.
+ *
+ * ONE list: the read verbs' explicit-projection filter, this module's row
+ * shaping and the write path's undeclared-key door all read it, so a key a read
+ * accepts is never refused by a write and never trimmed from a row.
+ */
+export const PLATFORM_PROVISIONED_COLUMNS = ['id', 'created_at', 'updated_at'] as const;
+
+/**
+ * The declared column set of an object — its field-map keys plus
+ * {@link PLATFORM_PROVISIONED_COLUMNS} — or `undefined` when there is no
+ * checkable field map (absent, an array, or empty). `undefined` means "no
+ * opinion", never "nothing is declared".
+ */
+export function declaredColumnSet(
+  schema: { fields?: unknown } | null | undefined,
+): ReadonlySet<string> | undefined {
+  const fields = schema?.fields;
+  if (!fields || typeof fields !== 'object' || Array.isArray(fields)) return undefined;
+  const names = Object.keys(fields as Record<string, unknown>);
+  if (names.length === 0) return undefined;
+  const declared = new Set(names);
+  for (const provisioned of PLATFORM_PROVISIONED_COLUMNS) declared.add(provisioned);
+  return declared;
+}
+
+/**
+ * The row with every key outside `declared` removed — a NEW object when there
+ * was anything to remove, the same reference otherwise. Never mutates the row
+ * it is given: a driver that hands back a live reference into its own store
+ * (a contract violation, but one a test double commits) must not lose data
+ * because a read happened.
+ */
+export function withDeclaredColumnsOnly<T>(row: T, declared: ReadonlySet<string> | undefined): T {
+  if (!declared || !row || typeof row !== 'object' || Array.isArray(row)) return row;
+  const source = row as unknown as Record<string, unknown>;
+  let undeclared = false;
+  for (const key of Object.keys(source)) {
+    if (!declared.has(key)) { undeclared = true; break; }
+  }
+  if (!undeclared) return row;
+  const shaped: Record<string, unknown> = {};
+  for (const key of Object.keys(source)) {
+    if (declared.has(key)) shaped[key] = source[key];
+  }
+  return shaped as unknown as T;
+}
+
+/** {@link withDeclaredColumnsOnly} over a driver's result page. */
+export function rowsWithDeclaredColumnsOnly<T>(rows: T[], declared: ReadonlySet<string> | undefined): T[] {
+  if (!declared) return rows;
+  let changed = false;
+  const shaped = rows.map((row) => {
+    const next = withDeclaredColumnsOnly(row, declared);
+    if (next !== row) changed = true;
+    return next;
+  });
+  return changed ? shaped : rows;
+}

packages/objectql/src/engine.test.ts

@@ -2565,6 +2565,20 @@ describe('ObjectQL — file-as-reference migration flag (#3617)', () => {
   let engine: ObjectQL;
   let driver: IDataDriver;
 
+  // [#21571] The flag columns the real `sys_migration` declares
+  // (`platform-objects` sys-migration.object.ts). A read serves only declared
+  // columns, so a double declaring `id` alone would read every flag row as
+  // empty — the fixture declares what the production object declares.
+  const SYS_MIGRATION_DEF = {
+    name: 'sys_migration',
+    fields: {
+      id: { type: 'text' },
+      last_run_at: { type: 'datetime' },
+      verified_at: { type: 'datetime' },
+      blocking: { type: 'number' },
+    },
+  };
+
   const verifiedRow = {
     id: 'adr-0104-file-references',
     last_run_at: '2026-07-27T00:00:00.000Z',
@@ -2594,7 +2608,7 @@ describe('ObjectQL — file-as-reference migration flag (#3617)', () => {
   const withMediaObject = () => {
     vi.mocked(SchemaRegistry.getObject).mockImplementation((name: string) => {
       if (name === 'note') return { name: 'note', fields: { doc: { type: 'file' } } } as any;
-      if (name === 'sys_migration') return { name: 'sys_migration', fields: { id: { type: 'text' } } } as any;
+      if (name === 'sys_migration') return SYS_MIGRATION_DEF as any;
       return undefined;
     });
   };
@@ -2636,7 +2650,7 @@ describe('ObjectQL — file-as-reference migration flag (#3617)', () => {
   it('costs no query for an object that declares no media field', async () => {
     vi.mocked(SchemaRegistry.getObject).mockImplementation((name: string) => {
       if (name === 'invoice') return { name: 'invoice', fields: { amount: { type: 'number' } } } as any;
-      if (name === 'sys_migration') return { name: 'sys_migration', fields: { id: { type: 'text' } } } as any;
+      if (name === 'sys_migration') return SYS_MIGRATION_DEF as any;
       return undefined;
     });
     vi.mocked(driver.findOne).mockResolvedValue(verifiedRow as any);
@@ -2736,7 +2750,7 @@ describe('ObjectQL — file-as-reference migration flag (#3617)', () => {
       vi.mocked(SchemaRegistry.getObject).mockImplementation((name: string) => objects[name]);
       vi.mocked((SchemaRegistry as any).getAllObjects).mockImplementation(() => Object.values(objects));
     };
-    const SYS_MIGRATION = { name: 'sys_migration', fields: { id: { type: 'text' } } };
+    const SYS_MIGRATION = SYS_MIGRATION_DEF;
     const lines = (info: any) => info.mock.calls.map((c: any[]) => String(c[0])).join('\n');
 
     it('names the command that closes an open value-shape gate', async () => {

packages/objectql/src/engine.ts

@@ -290,6 +290,14 @@ import { readonlyWhenFkJudgementReadsParent } from './validation/rule-validator.
 // total over the MASTER's declared fields before it leaves this engine — the
 // same helper every other server seam materialises with (#1871/#4649/#4953).
 import { materializeDeclaredFields } from './declared-fields.js';
+// [#21571] The declared column set — the read verbs' default projection, their
+// explicit-projection filter and the write path's undeclared-key door, one list.
+import {
+  PLATFORM_PROVISIONED_COLUMNS,
+  declaredColumnSet,
+  rowsWithDeclaredColumnsOnly,
+  withDeclaredColumnsOnly,
+} from './declared-read-columns.js';
 import { applyInMemoryAggregation } from './in-memory-aggregation.js';
 import {
   resolveEngineDeleteDispatch,
@@ -1912,19 +1920,17 @@ function assertProjectionHasNoDottedPaths(
  *   because the partial-success path (`insertMany`) reports per row and must
  *   cull the bad rows instead of failing the batch around them.
  */
-const PLATFORM_PROVISIONED_COLUMNS = ['id', 'created_at', 'updated_at'] as const;
-
 function undeclaredWriteFieldErrors(
   object: string,
   schema: { fields?: unknown } | undefined,
   rows: readonly unknown[],
 ): Array<Error | undefined> {
   const out: Array<Error | undefined> = new Array(rows.length);
-  const fields = schema?.fields;
-  if (!fields || typeof fields !== 'object' || Array.isArray(fields)) return out;
-  const declared = new Set(Object.keys(fields as Record<string, unknown>));
-  if (declared.size === 0) return out;
-  for (const provisioned of PLATFORM_PROVISIONED_COLUMNS) declared.add(provisioned);
+  // [#21571] The declared set and its "no opinion" cases (no map, an array
+  // map, an empty map) are `declaredColumnSet`'s, shared with the read verbs'
+  // default projection — one answer to "is this a column of the object".
+  const declared = declaredColumnSet(schema);
+  if (!declared) return out;
   for (let i = 0; i < rows.length; i++) {
     const row = rows[i];
     if (!row || typeof row !== 'object' || Array.isArray(row)) continue;
@@ -11894,13 +11900,12 @@ export class ObjectQL implements IObjectQLEngine {
     // projection is a different fact and no longer reaches this filter via
     // the engine — `assertProjectionHasNoDottedPaths` above refused it.
     if (_findSchema?.fields && Array.isArray(ast.fields) && ast.fields.length > 0) {
-      const known = new Set(Object.keys(_findSchema.fields));
       // Always allow the primary key + audit columns even if not present in
       // schema.fields. Without this, callers requesting `select=id,name`
       // silently get the `id` projected away, breaking record navigation.
-      known.add('id');
-      known.add('created_at');
-      known.add('updated_at');
+      // [#21571] The same three the default projection and the write door
+      // admit — `PLATFORM_PROVISIONED_COLUMNS`, one list.
+      const known = new Set<string>([...Object.keys(_findSchema.fields), ...PLATFORM_PROVISIONED_COLUMNS]);
       // Whole names, no head-splitting: only plain entries reach here (the
       // dotted refusal above fired on anything carrying a '.').
       const filtered = ast.fields.filter(f => known.has(f));
@@ -11939,6 +11944,19 @@ export class ObjectQL implements IObjectQLEngine {
       try {
           let result = await driver.find(object, hookContext.input.ast as QueryAST, hookContext.input.options as any);
 
+          // [#21571] The read's default projection is the DECLARED field set:
+          // a column no metadata declares (a field retired in an upgrade,
+          // whose column additive sync leaves behind) never leaves the engine.
+          // Shaped here, on the rows as the driver returned them, so it holds
+          // whichever driver answered and whichever rung of a driver's
+          // recovery ladder answered (driver-sql retries `select('*')` when a
+          // projected statement names a missing column) — and before formulas,
+          // `expand`, file references and the hooks, so all of them see the
+          // declared record. See `declared-read-columns.ts`.
+          if (Array.isArray(result)) {
+            result = rowsWithDeclaredColumnsOnly(result, declaredColumnSet(_findSchema));
+          }
+
           // Post-process: evaluate formula virtual fields against the raw rows.
           // [#20082] With the caller's permission map when a formula calls
           // `can` — one resolution for the whole result set, never per row.
@@ -12174,12 +12192,9 @@ export class ObjectQL implements IObjectQLEngine {
     // the rationale, and for why this tolerance is plain-columns-only ([#7589]
     // refused any dotted entry above, so none reaches this filter).
     if (_findOneSchema?.fields && Array.isArray(ast.fields) && ast.fields.length > 0) {
-      const known = new Set(Object.keys(_findOneSchema.fields));
       // Always allow the primary key + audit columns even if not present
-      // in schema.fields (matches `find()` behavior).
-      known.add('id');
-      known.add('created_at');
-      known.add('updated_at');
+      // in schema.fields (matches `find()` behavior, one list).
+      const known = new Set<string>([...Object.keys(_findOneSchema.fields), ...PLATFORM_PROVISIONED_COLUMNS]);
       const filtered = ast.fields.filter(f => known.has(f));
       ast.fields = filtered.length > 0 ? filtered : undefined;
     }
@@ -12215,6 +12230,10 @@ export class ObjectQL implements IObjectQLEngine {
 
       let result = await driver.findOne(objectName, hookContext.input.ast as QueryAST, hookContext.input.options as any);
 
+      // [#21571] Same default projection as `find`, same position: the
+      // declared field set, applied to the row as the driver returned it.
+      result = withDeclaredColumnsOnly(result, declaredColumnSet(_findOneSchema));
+
       // Post-process: evaluate formula virtual fields against the raw row
       // ([#20082] with the caller's permission map when a formula calls `can`).
       if (result != null) {

packages/objectql/src/no-operator-object-door.ts

@@ -206,7 +206,7 @@ export function declaredNoOperatorObjectColumn(def: unknown): NoOperatorObjectCo
  * [#20745] The columns every record carries whether or not the declared map
  * lists them, with the type each stores: the same three names `find` /
  * `findOne` add to their known set and the write gate admits unconditionally
- * (`PLATFORM_PROVISIONED_COLUMNS` in `engine.ts`), because the platform
+ * (`PLATFORM_PROVISIONED_COLUMNS` in `declared-read-columns.ts`), because the platform
  * provisions them rather than the author declaring them.
  */
 const PLATFORM_PROVISIONED_COLUMN_TYPES: ReadonlyMap<string, string> = new Map([