fix(metadata-protocol)!: a metadata body's stored content hash is served and compared only in keyed form, never copied, never evaluated (#21207)

.changeset/21207-keyed-served-content-hash.md

@@ -0,0 +1,24 @@
+---
+'@objectstack/metadata-protocol': minor
+'@objectstack/objectql': minor
+'@objectstack/mcp': minor
+'@objectstack/plugin-audit': minor
+'@objectstack/service-analytics': minor
+'@objectstack/cli': minor
+---
+
+fix(metadata-protocol)!: a metadata body's stored content hash is served and compared only in keyed form, never copied, and never evaluated (#21207)
+
+Clause-②: yes (narrowing)
+
+<!-- adr-0087: not-required (no-migration-prescription) the stored content hash of a metadata body stays the canonical hash at rest and no metadata body, authorable key, spelling or export moves; what changes is the form a door serves the hash in (a keyed digest: the crypto provider's, or a process-scoped ephemeral key's when none is registered), the form an inbound version token is compared in, and which query shapes the doors accept over the two hash columns, so `objectstack migrate meta` has nothing to rewrite. The operator-run rewrite this release asks for is of audit, activity and decision-audit copies, not of metadata. The other categories are closed on facts: every package here publishes (not `unpublished`); no ADR-0087 id covers a served version token or a refused query shape (not `registered` / `already-registered`); and the change is runtime behaviour, not a declaration (not `runtime-interface-only` / `type-surface-only`). -->
+
+**BREAKING**: this narrows what the metadata doors serve and accept for the stored content hash of a metadata body — a hash over the whole stored body, withheld credential material included. Served beside the projected body it let a reader confirm a guess at that material offline; filtered on, it confirmed one online. It ships as `minor` under the launch-window convention for accept-set narrowings.
+
+**Three things change for callers and operators.**
+
+1. **A held version token gets one `409 METADATA_CONFLICT`.** Every door that hands out a metadata version token — the save, publish, package-publish and rollback receipts and the history read — now hands out a keyed digest of the stored hash instead of the hash itself, and the save and reset doors compare a token they are sent in that same form. The key is the crypto provider's; a host that registers none keys under a process-scoped ephemeral key instead, so a token is always issued and never empty. A token a client held from before the upgrade is refused once; take the token from the next read or receipt and retry. On a host with no provider the same happens after a restart, and on any host when a provider is first registered. An empty, withheld, raw or stale token is refused with the same `409`; it is never read as "no pin".
+2. **Filter, sort and group on the two stored content-hash columns, and on the version history's change note, now answer `400 INVALID_FIELD`** — on the generic data door, the MCP stdio reader and the analytics door, before the engine runs. The change note is included because a draft promotion that stated no message of its own recorded the draft's stored hash in it; the publish door now always states a hash-free message, and a note written before this release is served with the quoted hash in keyed form. A data-door search over the two stored-metadata tables no longer scans those columns or the stored body column, and an explicit search-field list naming one answers the same `400`. Every other column of the two tables is served, filtered, sorted and grouped as before, and every other object is unchanged.
+3. **Operators run `os migrate audit-metadata-bodies` once after upgrading, dry run first.** The audit ledger, the activity feed and the metadata decision-audit trail no longer copy the stored hash. The extended command drops it from the copies already written and withholds it in the decision-audit notes and their copies: a dry run by default, `--apply` to rewrite, idempotent. The version history stays the lineage.
+
+**What else changes.** The data door serves the two hash columns of the stored-metadata tables in keyed form, under the same key as the version tokens. The MCP stdio reader serves them keyed under the crypto provider's key, and omits them on a host with no provider. A `409` conflict refusal carries keyed values or none. The ObjectQL engine gains a read accessor for the registered provider's keyed digest; it is additive. A member's read of these tables is refused as before.

packages/cli/src/commands/migrate/audit-metadata-bodies.ts

@@ -44,16 +44,26 @@ async function confirm(question: string): Promise<boolean> {
  * NEW writes; rows copied before the fix keep their cleartext. This command
  * rewrites them, projecting each copied body through the SAME redactor.
  *
+ * [#21207] The same copies also carried the copied row's stored CONTENT HASH
+ * (`checksum`, and the history row's `previous_checksum`) — a hash over the whole
+ * stored body, withheld credential material included — and the decision-audit
+ * note of a refused optimistic-lock write (`sys_metadata_audit`, and its ledger
+ * and activity copies) named both hashes. The writers no longer copy either;
+ * this command drops the hash columns from the copies already written and
+ * withholds the hashes in those notes, in the same pass. Operators run it once
+ * after upgrading, dry run first.
+ *
  * Dry run by default (writes nothing), `--apply` to rewrite. Idempotent: a
- * second run finds nothing — a redacted copy has no credential left — so
- * re-running and reading a clean report is the verification. No `sys_migration`
- * flag is recorded: nothing gates irreversible behaviour on this rewrite (the
- * posture `os migrate summary-nulls` takes).
+ * second run finds nothing — a redacted copy has no credential and no hash
+ * left — so re-running and reading a clean report is the verification. No
+ * `sys_migration` flag is recorded: nothing gates irreversible behaviour on this
+ * rewrite (the posture `os migrate summary-nulls` takes).
  */
 export default class MigrateAuditMetadataBodies extends Command {
   static override description =
-    'Rewrite at-rest cleartext metadata-body copies the audit writer left in sys_audit_log / sys_activity, ' +
-    'projecting each copied body through the shared credential redactor. Dry run by default; --apply writes.';
+    'Rewrite the at-rest copies the audit writer left in sys_audit_log / sys_activity: project each copied ' +
+    'metadata body through the shared credential redactor and drop its stored content hash; withhold the hashes ' +
+    'a conflict note in sys_metadata_audit (and its copies) names. Dry run by default; --apply writes.';
 
   static override examples = [
     '$ os migrate audit-metadata-bodies',
@@ -117,12 +127,12 @@ export default class MigrateAuditMetadataBodies extends Command {
           this.exit(1);
           return;
         }
-        printWarning('Apply mode rewrites audit/activity rows. Re-run with --yes to confirm, or run without --apply to preview.');
+        printWarning('Apply mode rewrites audit/activity/decision rows. Re-run with --yes to confirm, or run without --apply to preview.');
         this.exit(1);
         return;
       }
       const ok = await confirm(
-        chalk.bold('\nRewrite every audit/activity row carrying a stored metadata body on this database? [y/N] '),
+        chalk.bold('\nRewrite every audit/activity/decision row carrying a stored metadata body or content hash on this database? [y/N] '),
       );
       if (!ok) {
         printInfo('Aborted — no changes made.');
@@ -188,14 +198,14 @@ export default class MigrateAuditMetadataBodies extends Command {
         printError(`${report.failures} row(s) could not be rewritten — re-run to finish them.`);
       } else if (apply && report.rewritten > 0) {
         printSuccess(
-          `Rewrote ${report.rewritten} audit/activity row(s). Re-run any time — it only revisits rows still carrying a body.`,
+          `Rewrote ${report.rewritten} audit/activity/decision row(s). Re-run any time — it only revisits rows still carrying a body or a hash.`,
         );
       } else if (apply) {
-        printSuccess('Nothing to rewrite — no audit/activity row carries a stored metadata body.');
+        printSuccess('Nothing to rewrite — no audit/activity/decision row carries a stored metadata body or content hash.');
       } else if (report.rewritten > 0) {
         printInfo(`Dry run only — ${report.rewritten} row(s) would be rewritten. Re-run with --apply.`);
       } else {
-        printSuccess('Nothing to rewrite — no audit/activity row carries a stored metadata body.');
+        printSuccess('Nothing to rewrite — no audit/activity/decision row carries a stored metadata body or content hash.');
       }
       console.log(chalk.dim(`  ${timer.display()}`));
       console.log('');

packages/cli/src/commands/migrate/preview-read-only.integration.test.ts

@@ -493,16 +493,22 @@ for (const cell of DIALECT_CELLS) {
         expect(payload.error).toContain("'sys_metadata'");
       }, cell.timeout);
 
-      it('audit-metadata-bodies without --apply on a database that does not exist exits 1 with both tables counted unread', async () => {
+      // [#21207] The audit reads a third table: the decision-audit trail, whose
+      // conflict notes named stored content hashes. The #21391 intent is
+      // unchanged — EVERY audited table is counted unread — so the expected set
+      // is the whole audited set, stated literally: a widening that is not
+      // carried here turns this case red instead of passing on a stale count.
+      it('audit-metadata-bodies without --apply on a database that does not exist exits 1 with every audited table counted unread', async () => {
         const absent = join(fixture!.dir, 'data', 'never-started.db');
         const { payload, exitCode } = await runJson(auditBodies, ['--database-url', `file:${absent}`]);
+        const audited = ['sys_activity', 'sys_audit_log', 'sys_metadata_audit'];
 
         expect(exitCode).toBe(1);
         expect(payload.apply).toBe(false);
         // `failures` counts the tables whose rows were NOT examined.
-        expect(payload.report.failures).toBe(2);
+        expect(payload.report.failures).toBe(audited.length);
         expect(payload.report.scanned).toBe(0);
-        expect(Object.keys(payload.report.byObject).sort()).toEqual(['sys_activity', 'sys_audit_log']);
+        expect(Object.keys(payload.report.byObject).sort()).toEqual(audited);
       }, cell.timeout);
     }
   });

packages/mcp/src/plugin.ts

@@ -29,7 +29,9 @@ import {
   createStdioDataBridge,
   enforceApiExposure,
   GATED_ACTIONS,
+  serveStoredMetadataHashes,
   serveStoredMetadataRow,
+  type StoredHashDigest,
 } from './stdio-data-bridge.js';
 import type { McpDataBridge } from './mcp-http-tools.js';
 import { CONNECT_AGENT_UI_BUNDLE } from './connect-ui.js';
@@ -377,7 +379,14 @@ export class MCPServerPlugin implements Plugin {
     let dataBridge: McpDataBridge | undefined;
     if (shouldStart) {
       const apiKey = readEnvWithDeprecation('OS_MCP_STDIO_API_KEY', [], { silent: true });
-      let ql: (IDataEngine & { find: (object: string, opts: unknown) => Promise<unknown> }) | undefined;
+      let ql:
+        | (IDataEngine & {
+            find: (object: string, opts: unknown) => Promise<unknown>;
+            // [#21207] The engine's keyed-digest accessor (objectql), probed
+            // per call: an engine without it serves no content hash.
+            getKeyedDigest?: () => StoredHashDigest | undefined;
+          })
+        | undefined;
       try {
         ql = ctx.getService('objectql');
       } catch {
@@ -546,6 +555,10 @@ export class MCPServerPlugin implements Plugin {
       // wall that changed mid-session must take effect on the next call rather
       // than at the next process restart. See `resolveStdioTenancyPosture` for
       // why this is not hoisted next to the localization memo.
+      // [#21207] The crypto provider's keyed digest, read at each use — the
+      // host registers the provider after the kernel starts.
+      const storedHashDigest = (): StoredHashDigest | undefined =>
+        typeof scopedQl.getKeyedDigest === 'function' ? scopedQl.getKeyedDigest() : undefined;
       const resolvePrincipal = async (): Promise<ExecutionContext> => {
         const ec = await resolveStdioExecutionContext(
           scopedQl,
@@ -561,6 +574,7 @@ export class MCPServerPlugin implements Plugin {
           engine: scopedQl,
           metadataService,
           resolvePrincipal,
+          keyedDigest: storedHashDigest,
         });
       } else {
         // Functional degradation, said once and naming the remedy: two of the
@@ -611,7 +625,12 @@ export class MCPServerPlugin implements Plugin {
         // `sys_metadata_history` row's body reaches this resource as its type's
         // read projection, never as the stored bytes — so the tool and the
         // resource cannot disagree about what a stored credential is.
-        return serveStoredMetadataRow(objectName, (row ?? null) as Record<string, unknown> | null);
+        // [#21207] …and its stored content hash keyed, or not served at all.
+        return serveStoredMetadataHashes(
+          objectName,
+          serveStoredMetadataRow(objectName, (row ?? null) as Record<string, unknown> | null),
+          storedHashDigest(),
+        );
       };
       ctx.logger.info(
         `[MCP] stdio transport principal-bound to OS_MCP_STDIO_API_KEY identity ${initial.userId} (RLS/FLS/tenant applied)`,