diff --git a/.changeset/23649-datasource-mapping-before-sync.md b/.changeset/23649-datasource-mapping-before-sync.md new file mode 100644 index 00000000000..99d006079ba --- /dev/null +++ b/.changeset/23649-datasource-mapping-before-sync.md @@ -0,0 +1,29 @@ +--- +'@objectstack/objectql': minor +'@objectstack/runtime': patch +--- + +fix(objectql,runtime): a stack's `datasourceMapping` is installed before the engine's first schema sync, and the datasources it routes platform objects to are connected then, so a served boot keeps every object in the store its rule names (#23649) + +Clause-②: yes + +A mapping rule is routing, not a hint (ADR-0062, the `datasourceMapping` half of D2): every object it matches resolves to the datasource it names. `AppPlugin.start()` installed the rules late. By then `ObjectQLPlugin.start()` had schema-synced every registered object onto the host's default store, and the auth plugin had written the Default Organization there. A host config declaring `datasourceMapping: [{ default: true, datasource: 'primary' }]` over a declared `primary` reached `Server is ready` with `OS_DATABASE_URL` set (or under `--dev`), and every platform read then failed with `no such table`, because the platform tables and the organization were in the URL store and `primary` had neither. `os start` with `OS_DATABASE_URL` set to a different database than the rule names behaved the same way. A rule that routes platform objects only (`objectPattern: 'sys_*'`) also left its datasource unconnected, because the connect gate counted the app's own objects only. Every platform read then failed with "mapped for object … is not registered". + +**What changes** + +The public surface widens by one export and one event: `@objectstack/objectql` now exports `OBJECTQL_DRIVERS_READY_HOOK`, and `ObjectQLPlugin.start()` triggers the kernel event it names, `objectql:drivers-ready`, which any plugin can subscribe to. + +- **`@objectstack/objectql`.** `ObjectQLPlugin.start()` triggers the kernel event `objectql:drivers-ready` (exported as `OBJECTQL_DRIVERS_READY_HOOK`), with the engine as its argument. It fires once per boot, after the boot drivers are adopted and connected and before the first schema sync, the `sys_metadata` hydration or any read or write. A handler that throws fails the boot. +- **`@objectstack/runtime`.** `AppPlugin` handles that event. It installs the stack's `datasourceMapping`, then connects every declared datasource the rules route an already-registered object to, platform objects included. The connect is the same `connectDeclared` `start()` uses, with the same fail-fast verdict. The first DDL, the hydration and the Default Organization then land on the mapped datasource. On a composition whose engine never fires the event, `start()` installs the rules as before. + +**Unchanged** + +- A config with no `datasourceMapping`, and a rule naming the host's `default`. +- The engine's default driver. The mapped datasource connects after the engine has adopted its `driver.*` services, so it never takes the default slot from a stack plugin's driver. +- Where an object's rows live after boot. The rules were already in force once `start()` had run. +- `os start` on the config-declared default (`datasourceMapping` `{ default: true }` and no URL). The resolver points the host default at the same database, so everything was already in one file, and still is. +- A rule routing only the app's own objects. Their rows were already written to the mapped datasource. The only difference is that the first sync no longer creates an empty copy of their tables in the default store. + +**Refused at boot, as documented.** The docs already say a rule that routes objects to a datasource that cannot be connected fails the boot. A rule that routed only platform objects to such a datasource used to reach `Server is ready` with every platform read failing. It now fails the boot with the connect error and the objects routed to the datasource. The remedy is to make the datasource reachable, or to remove the rule. + +**Still on the default store:** the package registry's own `sys_packages` table. It is not an object, so no rule routes it, and the package service creates and reads it on the engine's default driver. diff --git a/packages/cli/test/serve-datasource-mapping-before-sync.integration.test.ts b/packages/cli/test/serve-datasource-mapping-before-sync.integration.test.ts new file mode 100644 index 00000000000..7563b8e212a --- /dev/null +++ b/packages/cli/test/serve-datasource-mapping-before-sync.integration.test.ts @@ -0,0 +1,228 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. +// test-budget: 30 — spawns the real producer once per single-cause fixture: two production boots of the built CLI (a `default: true` mapping over a declared sqlite datasource, and the same config with no mapping), each fixture differing only in that rule; 17.6 s, the slower of two local runs on a shared container, x 1.25, no CI reading yet + +import { afterAll, beforeAll, describe, expect, it } from 'vitest'; +import { spawn, type ChildProcessByStdio } from 'node:child_process'; +import { existsSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join, resolve } from 'node:path'; +import type { Readable } from 'node:stream'; +import { fileURLToPath } from 'node:url'; +import Database from 'better-sqlite3'; +import { defineStackSourceFromLiteral, linkSpec } from './helpers/define-stack-fixture.js'; +import { + childEnv, + E2E_SECRET_KEY, + portContentionError, + requireBuiltCli, + reservePort, + RUN_JS_RESOLVES_FROM_DIST, +} from './helpers/serve-process.js'; + +/** + * A served host config whose `datasourceMapping` routes every object to a + * declared datasource (#23649), measured at the public door. + * + * ## The defect + * + * `objectstack serve objectstack.config.ts`, production, `OS_DATABASE_URL` + * naming one SQLite file, the config declaring a second (`primary`) and the rule + * `{ default: true, datasource: 'primary' }`. The boot printed its ready banner, + * then every platform read failed with "no such table": the engine plugin's + * schema sync and the auth plugin's Default Organization had both used the URL + * store, and only then did `AppPlugin.start()` install the rule, re-routing + * every platform object to a `primary` whose schema never synced. The URL store + * held the platform tables and the organization; `primary` held neither. + * + * ## What is pinned + * + * 1. the mapped boot reaches its banner with no "no such table" and no + * "is not registered" line, and afterwards `primary` holds the platform + * tables and the one Default Organization while the URL store holds no + * `sys_organization` table at all; + * 2. CONTROL: the same config with no mapping keeps all of it in the URL + * store, and the managed, unrouted `primary` is never opened. + * + * ## Why `bin/run.js` with `NODE_ENV` unset, and why `.integration` + * + * Production posture is where the measured boot ran (`bin/run-dev.js` pins + * development). The file guards a boot ORDER any serve or plugin edit can + * break, so it rides the queue run rather than the nightly `.e2e` tier — the + * reasoning `serve-no-storage-boot-guard.integration.test.ts` records. + */ + +const HERE = resolve(fileURLToPath(import.meta.url), '..'); +/** `bin/run.js` — the SHIPPED entrypoint. See the header for why this one. */ +const CLI = resolve(HERE, '../bin/run.js'); + +/** The complete banner — keyed on its LAST line, so the whole block is on the stream. */ +const READY_BANNER_TAIL = /Press Ctrl\+C to stop/; +/** What a platform read against an unsynced datasource prints. */ +const NO_SUCH_TABLE = 'no such table'; +/** What a routed read against a never-connected datasource prints. */ +const NOT_REGISTERED = 'is not registered'; + +const AUTH_SECRET = 'os-23649-auth-secret-not-a-real-credential'; +const URL_STORE = 'ctl.db'; +const PRIMARY_STORE = 'ds-primary.db'; + +const OBJECT = `{ name: 'fx_task', label: 'Task', pluralLabel: 'Tasks', fields: { title: { type: 'text', label: 'Title' } } }`; +const MANIFEST = `{ id: 'com.example.fx23649', namespace: 'fx', version: '1.0.0', type: 'app', name: 'Fx' }`; +const PRIMARY = `{ name: 'primary', label: 'Primary', driver: 'sqlite', config: { filename: './${PRIMARY_STORE}' } }`; +/** An instantiated plugin, which is what makes this a HOST config. */ +const HOST_PLUGIN = `{ name: 'com.example.fx23649.host-plugin', version: '1.0.0', init: async () => {}, start: async () => {} }`; + +/** What `spawn(…, { stdio: ['ignore', 'pipe', 'pipe'] })` returns — no `stdin`. */ +type BootChild = ChildProcessByStdio; + +const children: BootChild[] = []; +let root: string; +let mappedDir: string; +let controlDir: string; + +function writeFixture(dir: string, mapping: string): void { + mkdirSync(dir, { recursive: true }); + writeFileSync(join(dir, 'package.json'), JSON.stringify({ name: 'fx-23649', private: true, type: 'module' })); + const source = defineStackSourceFromLiteral( + `{ manifest: ${MANIFEST}, objects: [${OBJECT}], datasources: [${PRIMARY}],${mapping} plugins: [${HOST_PLUGIN}] }`, + ); + writeFileSync(join(dir, 'objectstack.config.ts'), source); + linkSpec(dir); +} + +interface Boot { + output: string; + ready: boolean; +} + +/** + * Spawn `os serve` in production posture with `OS_DATABASE_URL` naming the URL + * store. Resolves once the child has EXITED — after the complete banner (the + * child is then stopped) or on its own — so the stores are closed when read. + */ +async function boot(cwd: string): Promise { + const port = reservePort(); + const child = spawn(process.execPath, [CLI, 'serve', 'objectstack.config.ts', '--port', String(port)], { + cwd, + stdio: ['ignore', 'pipe', 'pipe'], + // `childEnv`, never a bare `...process.env` — see its header (commit 1ddda1d00). + env: childEnv({ + NO_COLOR: '1', + OS_LOG_LEVEL: '', + OS_DISABLE_CONSOLE: '1', + OS_CLOUD_URL: 'off', + OS_HOME: join(cwd, '.os-home'), + OS_SECRET_KEY: E2E_SECRET_KEY, + OS_AUTH_SECRET: AUTH_SECRET, + OS_DATABASE_URL: `file:./${URL_STORE}`, + DATABASE_URL: undefined, + OS_DATABASE_DRIVER: undefined, + OS_TENANCY_POSTURE: undefined, + OS_MULTI_ORG_ENABLED: undefined, + // Truly unset: production posture, and oclif resolves the command from `dist/`. + NODE_ENV: undefined, + }), + }) as BootChild; + children.push(child); + + let output = ''; + let ready = false; + return new Promise((done, fail) => { + const timer = setTimeout(() => { + child.kill('SIGKILL'); + fail(new Error(`serve neither printed its complete banner nor exited\n--- output ---\n${output}`)); + }, 150_000); + const onData = (d: unknown) => { + output += String(d); + if (!ready && READY_BANNER_TAIL.test(output)) { + ready = true; + child.kill('SIGTERM'); + } + }; + child.stdout.on('data', onData); + child.stderr.on('data', onData); + child.on('exit', () => { + clearTimeout(timer); + const contention = ready + ? null + : portContentionError(output, 'os serve (bin/run.js, NODE_ENV unset ⇒ production)', port); + if (contention) fail(contention); + else done({ output, ready }); + }); + }); +} + +/** The user tables of a SQLite file, and its `sys_organization` row count (`null` = no such table). */ +function readStore(file: string): { tables: string[]; organizations: number | null } { + const db = new Database(file, { readonly: true }); + try { + const tables = (db.prepare("SELECT name FROM sqlite_master WHERE type = 'table' AND name NOT LIKE 'sqlite_%'").all() as Array<{ name: string }>) + .map((r) => r.name); + const organizations = tables.includes('sys_organization') + ? (db.prepare('SELECT COUNT(*) AS n FROM sys_organization').get() as { n: number }).n + : null; + return { tables, organizations }; + } finally { + db.close(); + } +} + +beforeAll(() => { + requireBuiltCli(RUN_JS_RESOLVES_FROM_DIST); + root = mkdtempSync(join(tmpdir(), 'os-serve-mapping-before-sync-')); + mappedDir = join(root, 'mapped'); + writeFixture(mappedDir, ` datasourceMapping: [{ default: true, datasource: 'primary' }],`); + controlDir = join(root, 'control'); + writeFixture(controlDir, ''); +}); + +afterAll(() => { + for (const child of children) { + try { child.kill('SIGTERM'); } catch { /* already gone */ } + } + if (root) rmSync(root, { recursive: true, force: true }); +}); + +describe('os serve — a host config whose datasourceMapping routes every object to a declared datasource (#23649)', () => { + it( + 'boots over ONE store: the mapped datasource holds the platform tables and the Default Organization', + async () => { + const { output, ready } = await boot(mappedDir); + const seen = `\n--- output ---\n${output}`; + expect(ready, `the mapped boot did not reach its banner${seen}`).toBe(true); + expect(output, `a platform read hit an unsynced datasource${seen}`).not.toContain(NO_SUCH_TABLE); + expect(output, `a routed read hit a never-connected datasource${seen}`).not.toContain(NOT_REGISTERED); + + const primary = readStore(join(mappedDir, PRIMARY_STORE)); + for (const table of ['sys_organization', 'sys_metadata', 'sys_user', 'fx_task']) { + expect(primary.tables, `${table} is not on the mapped datasource${seen}`).toContain(table); + } + expect(primary.organizations, 'the Default Organization is not on the mapped datasource').toBe(1); + + // The URL store opened as the host default, and no platform object was synced + // or written there. + const url = readStore(join(mappedDir, URL_STORE)); + expect(url.tables, 'the platform schema was synced onto the URL store').not.toContain('sys_organization'); + expect(url.tables).not.toContain('sys_metadata'); + }, + 240_000, + ); + + it( + 'control: with no mapping the URL store holds everything, and the unrouted datasource is never opened', + async () => { + const { output, ready } = await boot(controlDir); + const seen = `\n--- output ---\n${output}`; + expect(ready, `the control boot did not reach its banner${seen}`).toBe(true); + expect(output, `the control boot hit a missing table${seen}`).not.toContain(NO_SUCH_TABLE); + + const url = readStore(join(controlDir, URL_STORE)); + for (const table of ['sys_organization', 'sys_metadata', 'sys_user', 'fx_task']) { + expect(url.tables, `${table} is not on the URL store${seen}`).toContain(table); + } + expect(url.organizations).toBe(1); + expect(existsSync(join(controlDir, PRIMARY_STORE)), 'the managed, unrouted datasource was opened').toBe(false); + }, + 240_000, + ); +}); diff --git a/packages/objectql/src/index.ts b/packages/objectql/src/index.ts index 06bae5ecb42..716da2668c9 100644 --- a/packages/objectql/src/index.ts +++ b/packages/objectql/src/index.ts @@ -573,7 +573,8 @@ export type { export { MetadataFacade } from './metadata-facade.js'; // Export Plugin Shim -export { ObjectQLPlugin } from './plugin.js'; +// [#23649] The pre-sync announcement the app plugin installs `datasourceMapping` on. +export { ObjectQLPlugin, OBJECTQL_DRIVERS_READY_HOOK } from './plugin.js'; // Export Kernel Factory export { createObjectQLKernel } from './kernel-factory.js'; diff --git a/packages/objectql/src/plugin.ts b/packages/objectql/src/plugin.ts index bff396ce527..a0b02f31d12 100644 --- a/packages/objectql/src/plugin.ts +++ b/packages/objectql/src/plugin.ts @@ -290,6 +290,35 @@ export interface ObjectQLPluginOptions { */ const SKIP_SCHEMA_SYNC_ENGINE_KEY = 'skipSchemaSync'; +/** + * [#23649] The kernel-bus event {@link ObjectQLPlugin.start} triggers once per + * boot, with the live engine as its one argument, at the last moment a routing + * decision can still be made for free: every boot-registered driver has been + * adopted (`driver.*` services) and connected (`ObjectQL.init()`), and nothing + * has been schema-synced, hydrated from `sys_metadata`, read or written yet. + * + * It exists for the stack's `datasourceMapping` (ADR-0062 D2 (d), #4462). A + * mapping rule is routing, not a hint: from the moment it is installed, every + * object it matches resolves to the named datasource and never falls back to + * the default store. Installed any later than this (it used to be installed in + * `AppPlugin.start()`, after this plugin's schema sync and after the auth + * plugin wrote the Default Organization), the rules re-route platform objects + * whose tables and rows already sit in the default store, and the boot reports + * ready over two stores. The app plugin subscribes in its `init()`: it installs + * the rules and connects every datasource they route a registered object to, + * so the first DDL and every boot write land where the mapping routes them. + * + * After driver adoption on purpose, never in Phase 1: the engine makes its + * first registered driver the default, so a datasource connected before a + * stack plugin's `driver.*` service is adopted would take the default slot + * from it and move every unbound object with it. A handler's throw propagates + * and fails the boot (a mapped datasource that cannot connect has no fallback). + * + * Not in `IPluginLifecycleEvents`: the bus is open to custom names, and this + * one has one emitter (here) and one subscriber (`AppPlugin`). + */ +export const OBJECTQL_DRIVERS_READY_HOOK = 'objectql:drivers-ready'; + export class ObjectQLPlugin implements Plugin { name = 'com.objectstack.engine.objectql'; type = 'objectql' as const; @@ -1016,6 +1045,16 @@ export class ObjectQLPlugin implements Plugin { // Initialize drivers (calls driver.connect() which sets up persistence) await this.ql?.init(); + // [#23649] The drivers are live and nothing has been synced, hydrated, read + // or written: the moment a subscriber can still decide WHERE objects live + // (the stack's `datasourceMapping`). See {@link OBJECTQL_DRIVERS_READY_HOOK}. + // Spelled as the literal here, not the constant: the settings bind-window + // guard derives the hooks fired before the settings bind from their call + // sites and can only read a literal name. The constant is the subscribers' + // spelling; the runtime pin `datasource-mapping-before-schema-sync.test.ts` + // subscribes through it and goes red if the two ever differ. + if (this.ql) await ctx.trigger('objectql:drivers-ready', this.ql); + // Phase 1: Sync built-in schemas so sys_metadata table exists before reading it. // // Cold-start-sensitive runtimes (Cloudflare Containers, Lambda) can diff --git a/packages/runtime/src/app-plugin.ts b/packages/runtime/src/app-plugin.ts index 29c2a35a738..64714de98bf 100644 --- a/packages/runtime/src/app-plugin.ts +++ b/packages/runtime/src/app-plugin.ts @@ -19,6 +19,7 @@ import { assertProtocolCompat, type ArtifactForwardConversionResult, } from '@objectstack/metadata-core'; +import { OBJECTQL_DRIVERS_READY_HOOK } from '@objectstack/objectql'; import { resolveTenancyPosture } from '@objectstack/types'; import { postureEnforcesWall, type TenancyPosture } from '@objectstack/spec/security'; import { SeedLoaderService } from './seed-loader.js'; @@ -185,6 +186,12 @@ export class AppPlugin implements Plugin { * the residual-owner warning prints once, at whichever read comes first. */ private codeDatasourceOwners?: Array<{ datasource: any; owner: { packageId?: string; packageVersion?: string } }>; + /** + * [#23649] Set once {@link installDatasourceMapping} has handed the stack's + * rules to the engine, so `start()` installs them only on a composition + * whose engine never announced its pre-sync moment. + */ + private datasourceMappingInstalled = false; /** Memoized backing store for {@link forwardConverted} — the boot's one conversion. */ private bootConversion?: ArtifactForwardConversionResult; /** When true, init/start become no-ops — env has no app payload. */ @@ -571,6 +578,17 @@ export class AppPlugin implements Plugin { contributeCodeDatasourceNames(ctx, () => this.codeDefinedDatasourceOwners(ctx) .map(({ datasource }) => datasource?.name) .filter((name): name is string => typeof name === 'string' && name.length > 0)); + + // [#23649] The stack's `datasourceMapping` governs the engine from its + // FIRST schema sync — see `honourDatasourceMappingBeforeSchemaSync`. + // Subscribed here and resolved there: the rules come from `collections`, + // which walks `packages[]`, and the registration above is the one thing + // in `init()` allowed to touch that list. A context with no hook bus (a + // mock) leaves `start()` to install the rules. + if (typeof (ctx as { hook?: unknown }).hook === 'function') { + ctx.hook(OBJECTQL_DRIVERS_READY_HOOK, (engine: unknown) => + this.honourDatasourceMappingBeforeSchemaSync(ctx, engine as IObjectQLEngine, appId)); + } } /** @@ -769,6 +787,128 @@ export class AppPlugin implements Plugin { ctx.logger.debug('[AppPlugin] Installed hook-metrics Server-Timing feed'); } + /** + * Hand the stack's `datasourceMapping` rules to the engine — the one + * install site, called at most once per boot. Answers whether there were + * rules to install. + */ + private installDatasourceMapping(ctx: PluginContext, ql: IObjectQLEngine, appId: string): boolean { + this.datasourceMappingInstalled = true; + const datasourceMapping = this.collections.datasourceMapping; + if (!Array.isArray(datasourceMapping)) return false; + ctx.logger.info('Configuring datasource mapping rules', { + appId, + ruleCount: datasourceMapping.length, + }); + ql.setDatasourceMapping(datasourceMapping); + return datasourceMapping.length > 0; + } + + /** + * [#23649] Make the stack's `datasourceMapping` govern the engine from its + * FIRST schema sync. Runs inside `ObjectQLPlugin.start()`, on + * `OBJECTQL_DRIVERS_READY_HOOK`: the boot's drivers are adopted and + * connected, and nothing has been synced, hydrated, read or written yet. + * + * Two steps, in this order: + * + * 1. install the rules ({@link installDatasourceMapping}); + * 2. connect every declared datasource they route an ALREADY-REGISTERED + * object to — platform objects included — through the same + * `connectDeclared` `start()` calls (ADR-0062 D2 (d): "a datasource a + * mapping rule routes at least one registered object to is + * auto-connected at boot", fail-fast on a `declared-auto` failure). The + * connect also syncs those objects' schemas onto the datasource, and the + * engine plugin's own sync that follows finds them there. + * + * Before this, the rules were installed in `start()`, after the engine + * plugin's schema sync and after the auth plugin's Default Organization + * had both used the default store, and step 2 counted this app's own + * objects only. A `{ default: true, datasource: 'primary' }` rule then + * re-routed every platform object to a `primary` whose schema never + * synced: the server reported ready while the URL store held the platform + * tables and the Default Organization, and `primary` answered "no such + * table". A rule routing platform objects to a datasource nothing else + * addressed (`objectPattern: 'sys_*'`) left it never connected at all. + * + * Which objects a rule matches is the ENGINE's answer + * ({@link resolveMappedObjects}), over the same registry the sync reads. + * Objects registered after this moment are routed by the installed rules + * as they arrive; `start()` connects for them as it always has. A + * datasource no rule routes a registered object to is left to `start()`, + * which connects it by its other D2 reasons at the moment it always did. + * + * Connecting HERE, after the engine adopted its `driver.*` services, keeps + * the default driver what it was: the engine makes its first registered + * driver the default, so a mapped datasource connected earlier (in + * `init()`) would take that slot from a stack plugin's driver. + */ + private async honourDatasourceMappingBeforeSchemaSync( + ctx: PluginContext, + ql: IObjectQLEngine, + appId: string, + ): Promise { + if (this.datasourceMappingInstalled) return; + if (!this.installDatasourceMapping(ctx, ql, appId)) return; + + const dsDefs = this.convertedCollection(ctx, 'datasources'); + const dsList: any[] = Array.isArray(dsDefs) + ? dsDefs + : dsDefs && typeof dsDefs === 'object' + ? Object.entries(dsDefs).map(([name, def]) => ({ name, ...(def as any) })) + : []; + if (dsList.length === 0) return; + + // Absent on a runtime without the datasource-admin plugin: declared + // datasources stay metadata-only there, and `start()` says so. + let connection: + | { + connectDeclared?: (input: { + datasources: any[]; + objects?: Array<{ name?: string; datasource?: string }>; + mappedObjects?: Record; + }) => Promise>; + } + | undefined; + try { + connection = ctx.getService('datasource-connection'); + } catch { + connection = undefined; + } + if (typeof connection?.connectDeclared !== 'function') return; + + const registered = ql.registry.getAllObjects() as Array<{ name?: string; datasource?: string }>; + const mappedObjects = this.resolveMappedObjects(ql, registered); + const routedTo = dsList.filter( + (ds) => typeof ds?.name === 'string' && (mappedObjects[ds.name]?.length ?? 0) > 0, + ); + if (routedTo.length === 0) return; + + const declaredObjects = this.convertedCollection(ctx, 'objects'); + try { + const results = await connection.connectDeclared({ + datasources: routedTo, + objects: Array.isArray(declaredObjects) ? declaredObjects : [], + mappedObjects, + }); + const connected = results.filter((r) => r.status === 'connected'); + if (connected.length > 0) { + ctx.logger.info('Auto-connected declared datasources', { + appId, + connected: connected.map((r) => r.name), + before: 'schema sync', + }); + } + } catch (err) { + // Same verdict and the same line as `start()`'s connect: a + // datasource a mapping routes objects to has no fallback store. + ctx.logger.error( + `[AppPlugin] declared-datasource auto-connect failed for app '${appId}': ${(err as Error)?.message ?? String(err)}`, + ); + throw err; + } + } + /** * Datasource name → the objects a `datasourceMapping` rule routes to it * (#4462), asked of the ENGINE rather than re-derived here. @@ -1080,14 +1220,15 @@ export class AppPlugin implements Plugin { ctx.logger.debug('Retrieved ObjectQL engine service', { appId }); - // Configure datasourceMapping if provided in the stack definition - const datasourceMapping = this.collections.datasourceMapping; - if (datasourceMapping && Array.isArray(datasourceMapping)) { - ctx.logger.info('Configuring datasource mapping rules', { - appId, - ruleCount: datasourceMapping.length - }); - ql.setDatasourceMapping(datasourceMapping); + // [#23649] The stack's `datasourceMapping` is normally already installed: + // `honourDatasourceMappingBeforeSchemaSync` ran inside + // `ObjectQLPlugin.start()`, before its schema sync. Installed HERE only on + // a composition that never announced that moment (no `ObjectQLPlugin` + // start on this kernel, e.g. a declaration-only boot that suppresses + // it, or a mock context with no hook bus), where nothing synced or + // wrote through that engine before this line. + if (!this.datasourceMappingInstalled) { + this.installDatasourceMapping(ctx, ql, appId); } // Surface code-defined datasources (ADR-0015 Addendum) in the metadata diff --git a/packages/runtime/src/datasource-mapping-before-schema-sync.test.ts b/packages/runtime/src/datasource-mapping-before-schema-sync.test.ts new file mode 100644 index 00000000000..290857cf7f5 --- /dev/null +++ b/packages/runtime/src/datasource-mapping-before-schema-sync.test.ts @@ -0,0 +1,206 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. +// +// #23649 — a stack's `datasourceMapping` governs the engine from its FIRST +// schema sync, platform objects included. +// +// ## The defect +// +// `AppPlugin.start()` used to install the mapping, after `ObjectQLPlugin.start()` +// had schema-synced every registered object onto the default store and after +// the auth plugin's `start()` had written the Default Organization there. A +// `{ default: true, datasource: 'primary' }` rule then re-routed every platform +// object to a `primary` whose schema never synced: the served boot reported +// ready while the default store held the platform tables and the boot rows, +// and `primary` answered "no such table". A rule that routes platform objects +// only (`objectPattern: 'sys_*'`) left `primary` never connected at all, +// because the connect gate counted the app's own objects. +// +// ## What is pinned +// +// One in-process boot per mapping shape, with the composition the served host +// config uses: a default SQL driver (`DriverPlugin`), the engine plugin, the +// app, the datasource-admin plugin with the REAL driver factory, and a fixture +// "platform" plugin that stands in for the auth plugin — it registers an object +// the app does not declare in `init()` and writes a row to it in `start()`, +// between the engine plugin's start and the app's (the Default Organization's +// slot). Every store is sqlite `:memory:`, so each datasource is its own +// database and nothing reaches the host filesystem. +// +// 1. `default: true` → `primary`: the platform object's table AND its boot +// row are on `primary`, and the default store holds neither; +// 2. `objectPattern: 'sys_*'` → `primary`: `primary` is connected and holds +// the platform object, while the app object (unmatched) stays on the +// default store; +// 3. CONTROL, no mapping: everything on the default store, and the managed, +// unrouted `primary` stays metadata-only (unchanged); +// +// and in every leg the engine's default driver is still the `DriverPlugin`'s: +// the mapped datasource connects after the engine adopted its drivers, so it +// never takes the default slot. + +import { describe, it, expect, afterEach } from 'vitest'; +import { Runtime } from './runtime.js'; +import { DriverPlugin } from './driver-plugin.js'; +import { AppPlugin } from './app-plugin.js'; + +// [#10126] Pay the first transform of these dist-resolved workspace deps at +// MODULE LOAD, not inside a clocked `it()` (`scripts/check-test-source-alias.mjs`). +import '@objectstack/driver-sql'; +import '@objectstack/objectql'; +import '@objectstack/service-datasource'; + +const BOOT_TIMEOUT = 60_000; + +/** The object the fixture platform plugin owns — no rule names it but `default`/`sys_*`. */ +const PLATFORM_OBJECT = 'sys_fx_ledger'; +/** The app's own object. */ +const APP_OBJECT = 'fx_task'; +/** The row the platform plugin writes in `start()` — the Default Organization's slot. */ +const BOOT_ROW = 'written-during-boot'; + +interface EngineView { + getDriverByName(name: string): any; + getDefaultDriverName(): string | undefined; + insert(object: string, data: Record, options?: unknown): Promise; +} + +function stack(datasourceMapping?: unknown[]) { + return { + manifest: { id: 'com.test.mapping-before-sync', name: 'Mapping before sync', version: '1.0.0' }, + objects: [{ name: APP_OBJECT, label: 'Task', fields: { title: { type: 'text' } } }], + datasources: [ + { + name: 'primary', + label: 'Primary (sqlite :memory:)', + driver: 'sqlite', + schemaMode: 'managed', + origin: 'code', + config: { filename: ':memory:' }, + active: true, + }, + ], + ...(datasourceMapping ? { datasourceMapping } : {}), + }; +} + +/** + * A plugin the app does not own, shaped like the auth plugin: its object is + * registered in `init()` and a row is written to it in `start()`, after the + * engine plugin's start (declared) and before the app's (it is registered + * ahead of the app, and the kernel breaks ties by registration order). + */ +function platformPlugin() { + return { + name: 'com.test.mapping-before-sync.platform', + version: '1.0.0', + dependencies: ['com.objectstack.engine.objectql'], + init: async (ctx: any) => { + ctx.getService('manifest').register({ + id: 'com.test.mapping-before-sync.platform', + name: 'Platform fixture', + version: '1.0.0', + type: 'plugin', + objects: [{ name: PLATFORM_OBJECT, label: 'Ledger', fields: { title: { type: 'text' } } }], + }); + }, + start: async (ctx: any) => { + await ctx.getService('objectql').insert(PLATFORM_OBJECT, { title: BOOT_ROW }, { context: { isSystem: true } }); + }, + }; +} + +async function makeDefaultDriver() { + const { SqlDriver } = await import('@objectstack/driver-sql'); + return new SqlDriver({ client: 'better-sqlite3', connection: { filename: ':memory:' }, useNullAsDefault: true }); +} + +const kernels: Array<{ stop?: () => Promise }> = []; +afterEach(async () => { + for (const kernel of kernels.splice(0)) { + try { await kernel.stop?.(); } catch { /* noop */ } + } +}); + +async function boot(datasourceMapping?: unknown[]) { + const { ObjectQLPlugin } = await import('@objectstack/objectql'); + const { DatasourceAdminServicePlugin, createDefaultDatasourceDriverFactory } = await import( + '@objectstack/service-datasource' + ); + const defaultDriver = await makeDefaultDriver(); + const runtime = new Runtime({ cluster: false }); + const kernel = runtime.getKernel(); + kernels.push(kernel as never); + await kernel.use(new DriverPlugin(defaultDriver)); + await kernel.use(new ObjectQLPlugin()); + await kernel.use(platformPlugin() as never); + await kernel.use(new AppPlugin(stack(datasourceMapping))); + await kernel.use(new DatasourceAdminServicePlugin({ driverFactory: createDefaultDatasourceDriverFactory() })); + await kernel.bootstrap(); + const engine = kernel.getService('objectql'); + return { engine, defaultDriver }; +} + +/** Rows knex returns for a raw SELECT, whichever envelope the dialect uses. */ +function rowsOf(result: any): any[] { + if (Array.isArray(result)) return result; + if (Array.isArray(result?.rows)) return result.rows; + return result == null ? [] : [result]; +} + +async function tablesOf(driver: any): Promise { + const result = await driver.execute("SELECT name FROM sqlite_master WHERE type = 'table' ORDER BY name"); + return rowsOf(result).map((r: any) => r.name); +} + +async function bootRowsOn(driver: any): Promise { + const result = await driver.execute(`SELECT title FROM ${PLATFORM_OBJECT}`); + return rowsOf(result).map((r: any) => r.title); +} + +describe('#23649 — datasourceMapping governs the engine from its first schema sync', () => { + it('a `default: true` rule puts the platform object\'s table and its boot row on the mapped datasource', async () => { + const { engine, defaultDriver } = await boot([{ default: true, datasource: 'primary' }]); + const primary = engine.getDriverByName('primary'); + expect(primary, 'the mapped datasource was never connected').toBeDefined(); + + // The default slot is the DriverPlugin's, as before the mapping existed. + expect(engine.getDriverByName(engine.getDefaultDriverName() as string)).toBe(defaultDriver); + + const onPrimary = await tablesOf(primary); + const onDefault = await tablesOf(defaultDriver); + expect(onPrimary).toContain(PLATFORM_OBJECT); + expect(onPrimary).toContain(APP_OBJECT); + expect(onDefault, 'the first schema sync still went to the default store').not.toContain(PLATFORM_OBJECT); + expect(onDefault).not.toContain(APP_OBJECT); + + // The row a platform plugin wrote in `start()` is where the object routes. + expect(await bootRowsOn(primary)).toEqual([BOOT_ROW]); + }, BOOT_TIMEOUT); + + it('an `objectPattern: \'sys_*\'` rule connects the datasource it routes platform objects to', async () => { + const { engine, defaultDriver } = await boot([{ objectPattern: 'sys_*', datasource: 'primary' }]); + const primary = engine.getDriverByName('primary'); + expect(primary, 'a datasource only platform objects route to was never connected').toBeDefined(); + expect(engine.getDriverByName(engine.getDefaultDriverName() as string)).toBe(defaultDriver); + + const onPrimary = await tablesOf(primary); + const onDefault = await tablesOf(defaultDriver); + expect(onPrimary).toContain(PLATFORM_OBJECT); + expect(onDefault).not.toContain(PLATFORM_OBJECT); + // The app object matches no rule: it stays on the default store. + expect(onDefault).toContain(APP_OBJECT); + expect(onPrimary).not.toContain(APP_OBJECT); + expect(await bootRowsOn(primary)).toEqual([BOOT_ROW]); + }, BOOT_TIMEOUT); + + it('control: with no mapping everything stays on the default store and the unrouted datasource stays metadata-only', async () => { + const { engine, defaultDriver } = await boot(); + expect(engine.getDriverByName('primary')).toBeUndefined(); + expect(engine.getDriverByName(engine.getDefaultDriverName() as string)).toBe(defaultDriver); + + const onDefault = await tablesOf(defaultDriver); + expect(onDefault).toContain(PLATFORM_OBJECT); + expect(onDefault).toContain(APP_OBJECT); + expect(await bootRowsOn(defaultDriver)).toEqual([BOOT_ROW]); + }, BOOT_TIMEOUT); +}); diff --git a/scripts/check-cli-test-child-env.mjs b/scripts/check-cli-test-child-env.mjs index 9672de77f62..9ed816d1be8 100644 --- a/scripts/check-cli-test-child-env.mjs +++ b/scripts/check-cli-test-child-env.mjs @@ -2635,7 +2635,7 @@ export function selfTest() { .filter((abs) => builtEntrypointSpawns(abs, readFileSync(abs, 'utf8')).spawns > 0) .map((abs) => relative(REPO_ROOT, abs).split(sep).join('/')) .sort(); - t('the built-entrypoint population is exactly the eight files that spawn bin/run.js', + t('the built-entrypoint population is exactly the nine files that spawn bin/run.js', JSON.stringify(builtFiles) === JSON.stringify([ // [#21733] Joined the population deliberately: the card's acceptance is // a plain `os dev` restart on the BUILT packages, and its production leg @@ -2651,6 +2651,12 @@ export function selfTest() { // DELIBERATE_REROUTE entries for why that input cannot be scrubbed and // why `bin/run-dev.js` is the wrong entry for it. 'packages/cli/test/published-entry-node-env-source-reroute.test.ts', + // [#23649] Joined the population deliberately: the split it pins was + // measured at the public door in PRODUCTION posture (a `datasourceMapping` + // default over `OS_DATABASE_URL`), and `bin/run-dev.js` sets + // `NODE_ENV=development` before argv is parsed. Every spawn hands its + // child `NODE_ENV: undefined`. + 'packages/cli/test/serve-datasource-mapping-before-sync.integration.test.ts', 'packages/cli/test/serve-mcp-capability-collision.e2e.test.ts', 'packages/cli/test/serve-mcp-stdio-answers.e2e.test.ts', // [#23573] Joined the population deliberately: the no-storage boot diff --git a/scripts/check-settings-bind-window.mjs b/scripts/check-settings-bind-window.mjs index c69ff6e4cc8..193e12aefed 100644 --- a/scripts/check-settings-bind-window.mjs +++ b/scripts/check-settings-bind-window.mjs @@ -168,7 +168,9 @@ const READY_HOOK = 'kernel:ready'; * no longer fired before the bind (a stale pin, and the anti-vacuity limb — a * scan that finds no emit at all stales every row). Each value is the emit site * the derivation found, so a reader can check the row without re-running it. - * All eight were derived from Phase 1/2 bodies at `4e4111ca0`; a hook fired + * The first eight were derived from Phase 1/2 bodies at `4e4111ca0`, and + * `objectql:drivers-ready` joined when `ObjectQLPlugin.start()` began firing it + * before its first schema sync (#23649); a hook fired * from a `kernel:ready` handler would be derived too and is judged the same * way — conservatively, since it precedes the bind only when its emitter's * handler does. @@ -190,6 +192,8 @@ const PRE_BIND_HOOKS = Object.freeze({ 'packages/services/service-datasource/src/plugin.ts — ExternalDatasourceServicePlugin.start() → ctx.trigger(\'external-datasource:ready\', this.service)', 'mcp:ready': 'packages/mcp/src/plugin.ts — MCPServerPlugin.start() → ctx.trigger(\'mcp:ready\', this.runtime)', + 'objectql:drivers-ready': + 'packages/objectql/src/plugin.ts — ObjectQLPlugin.start() → ctx.trigger(\'objectql:drivers-ready\', this.ql)', }); /** The method a plugin context fires a hook through. */