diff --git a/.changeset/21923-datasource-origin-from-provenance.md b/.changeset/21923-datasource-origin-from-provenance.md new file mode 100644 index 00000000000..955db84b499 --- /dev/null +++ b/.changeset/21923-datasource-origin-from-provenance.md @@ -0,0 +1,17 @@ +--- +"@objectstack/service-datasource": patch +--- + +The datasource admin door and the metadata door agree on a runtime datasource: a datasource saved through `/api/v1/meta/datasource/:name` is listed and editable through `/api/v1/datasources` in the same boot and after a restart, and one created through the admin door can be edited and removed through the metadata door (#21923) + +Clause-②: no + +`DatasourceSchema.origin` publishes `runtime` as "created via the Studio wizard, persisted in the runtime metadata store, environment-scoped, editable", and says `origin` is never accepted from client input. Three things broke that for a datasource the metadata door wrote, or one the admin door wrote: + +- **The admin door read `origin` from the record.** It served `origin ?? 'code'`, so after a restart a datasource saved through the metadata door (whose body carries no `origin`, or asserts `origin: 'code'`) was served as code-defined, and `PATCH /api/v1/datasources/:name` answered `400 DATASOURCE_ADMIN_ERROR` ("… is code-defined and cannot be edited at runtime."). It now serves `code` only for a name the host registers from code (the host's code-datasource set, the one the boot restore and the metadata door's refusal read), and `runtime` for every other name, whatever the record says. Boot pool rehydration follows the served origin, so such a datasource also gets its live pool after a restart. +- **A metadata-door write never reached the admin door until a restart.** The metadata door persisted the row but never registered it where the admin door lists, so in the same boot `GET /api/v1/datasources` omitted it and `PATCH` answered "not found". The datasource-admin plugin now registers the protocol's awaited `datasource` mutation projector: after a metadata-door save, publish, revert, rollback or delete, the admin door's registry follows the stored row and the live pool converges on it (opened on a create, rebuilt on a connectivity change, evicted on a delete) before the metadata door answers. A write under a name the host defines in code changes nothing here, so the metadata door's `DELETE` of a leftover row under such a name still leaves the code definition served. +- **An admin-created row could not be edited or removed through the metadata door.** The admin door stored its `sys_metadata` row with no `checksum`, the column the metadata door's optimistic lock compares, so `PUT` and `DELETE /api/v1/meta/datasource/:name` answered `409 METADATA_CONFLICT` for every admin-created datasource. The admin door now stamps the checksum the metadata door's repository stamps (`hashSpec` from `@objectstack/metadata-core`, which becomes a runtime dependency of this package). A row stored before this release has no checksum until the admin door next writes it: editing that datasource through the admin door once makes it editable through the metadata door. + +Cluster convergence (a peer replica's signal after an admin-door write) decides "code" from the same set, so a stored row under a code-defined name never opens a pool there, and every other stored row is pooled as runtime. + +**Unchanged.** A code-defined datasource stays read-only through both doors: the admin door's `PATCH` and `DELETE` still answer `400 DATASOURCE_ADMIN_ERROR`, and the metadata door's still answer `403 NOT_OVERRIDABLE`. A host that composes no code-datasource producer (neither `AppPlugin` nor `DefaultDatasourcePlugin`) registers no set, and the admin door then serves every datasource as runtime, as its boot restore already treats every stored row. diff --git a/packages/qa/dogfood/test/datasource-meta-door-reaches-admin-door.dogfood.test.ts b/packages/qa/dogfood/test/datasource-meta-door-reaches-admin-door.dogfood.test.ts new file mode 100644 index 00000000000..039d3adc57d --- /dev/null +++ b/packages/qa/dogfood/test/datasource-meta-door-reaches-admin-door.dogfood.test.ts @@ -0,0 +1,237 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. +// +// [#21923] The metadata door and the datasource admin door agree on a runtime +// datasource — in the same boot and after a restart — over the real showcase +// composition. +// +// ## What was broken +// +// Measured on this composition before the fix: +// +// - `PUT /api/v1/meta/datasource/` saved a datasource (200), and in the +// same boot `GET /api/v1/datasources` omitted it and the admin door's +// `PATCH` answered "not found": the save wrote `sys_metadata`, never the +// MetadataService slot the admin door lists, and opened no pool; +// - after a restart the boot restore registered the row, and the admin read +// served it as `origin: 'code'` — `origin ?? 'code'` read a body with no +// `origin` as a code definition, and a body asserting `origin: 'code'` as +// one too — so its `PATCH` was refused as "code-defined"; +// - the other direction: a datasource the admin door created was stored with +// no `checksum`, so the metadata door's optimistic lock never matched it, +// and its `PUT` and `DELETE` answered `409 METADATA_CONFLICT`. +// +// ## What each case pins (triage's pins) +// +// - a datasource the metadata door creates, with no `origin`, is listed and +// editable through the admin door in the same boot (with a live pool, as +// an admin create has) and after a restart; +// - a body asserting `origin: 'code'` stays runtime: the admin door's origin +// comes from the host's code-datasource set, never from the record; +// - a datasource the admin door creates is edited and removed through the +// metadata door, and the removal leaves the admin door in the same boot; +// - a code-defined datasource stays refused at the admin door. +// +// The verify harness composes the datasource-admin service but not its REST +// routes, so this file mounts `registerDatasourceAdminRoutes` the way +// `packages/cli/src/commands/serve.ts` does. The working directory is a +// temporary one because the showcase's external datasource and its fixture +// both name a cwd-relative SQLite file. + +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import showcaseStack, { onEnable } from '@objectstack/example-showcase'; +import { registerDatasourceAdminRoutes } from '@objectstack/service-datasource'; +import { bootStack, type VerifyStack } from '@objectstack/verify'; +import { mkdtempSync, rmSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; + +/** The showcase's code-defined datasource. */ +const EXTERNAL = 'showcase_external'; +/** Created through the metadata door with no `origin` in the body. */ +const META_NONE = 'dogfood_meta_none_21923'; +/** Created through the metadata door with a body asserting `origin: 'code'`. */ +const META_CODE = 'dogfood_meta_code_21923'; +/** Created through the admin door, then edited and removed through the metadata door. */ +const ADMIN_RT = 'dogfood_admin_rt_21923'; + +const SYS = { isSystem: true, positions: [], permissions: [] }; + +interface Answer { + status: number; + json: { + success?: boolean; + reset?: boolean; + error?: string | { code?: string; message?: string }; + code?: string; + item?: Record; + projectionApplied?: { success: boolean; error?: string }; + data?: { datasource?: Record; datasources?: Array> }; + }; +} + +/** `{ status, code, message }` from either door's refusal envelope. */ +const refusal = (a: Answer) => { + const { error, code } = a.json; + return typeof error === 'object' && error !== null + ? { status: a.status, code: error.code, message: String(error.message ?? '') } + : { status: a.status, code, message: String(error ?? '') }; +}; + +describe('[#21923] the metadata door and the admin door agree on a runtime datasource (showcase)', () => { + let stack: VerifyStack; + let token: string; + let prevCwd: string; + let dir: string; + + const routes = () => ({ + name: 'dogfood.datasource-admin-routes', + version: '1.0.0', + optionalDependencies: ['com.objectstack.server.hono'], + init: async (ctx: { getService?: (name: string) => unknown }) => { + const httpServer = ctx.getService?.('http.server') ?? ctx.getService?.('http-server'); + registerDatasourceAdminRoutes(httpServer as never, ctx as never, '/api/v1'); + }, + }); + const boot = async () => { + stack = await bootStack(showcaseStack, { databaseFile: join(dir, 'showcase.db'), extraPlugins: [routes()] }); + token = await stack.signIn(); + }; + const restart = async () => { + await stack.stop(); + await boot(); + }; + const call = async (method: string, path: string, body?: unknown): Promise => { + const res = await stack.apiAs(token, method, path, body); + return { status: res.status, json: (await res.json().catch(() => ({}))) as Answer['json'] }; + }; + const storedRows = async (name: string) => { + const ql = (await stack.kernel.getServiceAsync('objectql')) as { + find(object: string, query: unknown): Promise>>; + }; + return ql.find('sys_metadata', { where: { type: 'datasource', name }, context: SYS }); + }; + const adminEntry = async (name: string) => { + const listed = await call('GET', '/datasources'); + expect(listed.status, JSON.stringify(listed.json)).toBe(200); + return listed.json.data?.datasources?.find((d) => d.name === name); + }; + const connectVerdict = (name: string) => { + const connection = stack.kernel.getService('datasource-connection') as { + listConnectionStates(): Array<{ name: string; status: string }>; + }; + return connection.listConnectionStates().find((s) => s.name === name)?.status; + }; + /** The `origin` the stored row's own body asserts, if any. */ + const storedOrigin = async (name: string) => { + const [row] = await storedRows(name); + const raw = row?.metadata; + return (typeof raw === 'string' ? JSON.parse(raw) : (raw as Record | undefined))?.origin; + }; + /** A sqlite datasource body, as an author writes it for the metadata door. */ + const sqliteBody = (name: string, label: string, extra: Record = {}) => ({ + name, label, driver: 'sqlite', config: { filename: join(dir, `${name}.db`) }, ...extra, + }); + /** The metadata door's served item, without its `_`-prefixed envelope keys. */ + const servedBody = async (name: string) => { + const read = await call('GET', `/meta/datasource/${name}`); + expect(read.status, JSON.stringify(read.json)).toBe(200); + return Object.fromEntries(Object.entries(read.json.item ?? {}).filter(([k]) => !k.startsWith('_'))); + }; + + beforeAll(async () => { + prevCwd = process.cwd(); + dir = mkdtempSync(join(tmpdir(), 'dogfood-21923-')); + process.chdir(dir); + // Provision the remote tables, exactly as `os dev` does at boot. + await onEnable({ logger: { info() {}, warn() {} } } as never); + await boot(); + }, 180_000); + + afterAll(async () => { + await stack?.stop(); + if (prevCwd) process.chdir(prevCwd); + if (dir) rmSync(dir, { recursive: true, force: true }); + }); + + it('premise: the code datasource is served as code, and none of this file\'s names exists yet', async () => { + expect(await adminEntry(EXTERNAL)).toMatchObject({ origin: 'code' }); + for (const name of [META_NONE, META_CODE, ADMIN_RT]) { + expect(await adminEntry(name)).toBeUndefined(); + expect(await storedRows(name)).toEqual([]); + } + }); + + it('a datasource the metadata door creates is listed and editable through the admin door in the same boot, with a live pool', async () => { + const none = await call('PUT', `/meta/datasource/${META_NONE}`, sqliteBody(META_NONE, 'Meta None 21923')); + expect(none.status, JSON.stringify(none.json)).toBe(200); + const code = await call('PUT', `/meta/datasource/${META_CODE}`, sqliteBody(META_CODE, 'Meta Code 21923', { origin: 'code' })); + expect(code.status, JSON.stringify(code.json)).toBe(200); + expect(await storedRows(META_NONE)).toHaveLength(1); + // The stored body really asserts `origin: 'code'` — the record's own claim. + expect(await storedOrigin(META_CODE)).toBe('code'); + + // Listed, as runtime — the body asserting `origin: 'code'` included. + expect(await adminEntry(META_NONE)).toMatchObject({ origin: 'runtime', label: 'Meta None 21923' }); + expect(await adminEntry(META_CODE)).toMatchObject({ origin: 'runtime', label: 'Meta Code 21923' }); + + // A live pool, as the admin door's create opens one. + expect(connectVerdict(META_NONE)).toBe('connected'); + expect(connectVerdict(META_CODE)).toBe('connected'); + + // Editable through the admin door. + const patched = await call('PATCH', `/datasources/${META_NONE}`, { label: 'Meta None 21923 (admin edit)' }); + expect(patched.status, JSON.stringify(patched.json)).toBe(200); + expect(await adminEntry(META_NONE)).toMatchObject({ origin: 'runtime', label: 'Meta None 21923 (admin edit)' }); + }); + + it('a datasource the admin door creates is edited and removed through the metadata door, and the removal leaves the admin door', async () => { + const created = await call('POST', '/datasources', sqliteBody(ADMIN_RT, 'Admin Runtime 21923')); + expect(created.status, JSON.stringify(created.json)).toBe(201); + expect(connectVerdict(ADMIN_RT)).toBe('connected'); + + const put = await call('PUT', `/meta/datasource/${ADMIN_RT}`, { ...(await servedBody(ADMIN_RT)), label: 'Admin Runtime 21923 (meta edit)' }); + expect(put.status, JSON.stringify(put.json)).toBe(200); + expect(await adminEntry(ADMIN_RT)).toMatchObject({ origin: 'runtime', label: 'Admin Runtime 21923 (meta edit)' }); + + const del = await call('DELETE', `/meta/datasource/${ADMIN_RT}`); + expect(del.status, JSON.stringify(del.json)).toBe(200); + expect(await storedRows(ADMIN_RT)).toEqual([]); + expect(await adminEntry(ADMIN_RT)).toBeUndefined(); + expect(connectVerdict(ADMIN_RT)).toBeUndefined(); + }); + + it('after a restart, both metadata-door datasources are runtime, pooled and editable through the admin door', async () => { + await restart(); + + expect(await adminEntry(META_NONE)).toMatchObject({ origin: 'runtime', label: 'Meta None 21923 (admin edit)' }); + // Still the record's claim at this restart: `origin: 'code'`. Served runtime. + expect(await storedOrigin(META_CODE)).toBe('code'); + expect(await adminEntry(META_CODE)).toMatchObject({ origin: 'runtime', label: 'Meta Code 21923' }); + for (const name of [META_NONE, META_CODE]) { + expect(connectVerdict(name)).toBe('connected'); + const patched = await call('PATCH', `/datasources/${name}`, { label: `${name} (after restart)` }); + expect(patched.status, JSON.stringify(patched.json)).toBe(200); + expect(await adminEntry(name)).toMatchObject({ origin: 'runtime', label: `${name} (after restart)` }); + } + expect(await adminEntry(ADMIN_RT)).toBeUndefined(); + }, 180_000); + + it('the metadata door edits and removes an admin-edited datasource, and the admin door follows in the same boot', async () => { + const put = await call('PUT', `/meta/datasource/${META_NONE}`, { ...(await servedBody(META_NONE)), label: 'Meta None 21923 (meta edit)' }); + expect(put.status, JSON.stringify(put.json)).toBe(200); + expect(await adminEntry(META_NONE)).toMatchObject({ origin: 'runtime', label: 'Meta None 21923 (meta edit)' }); + + const del = await call('DELETE', `/meta/datasource/${META_CODE}`); + expect(del.status, JSON.stringify(del.json)).toBe(200); + expect(await storedRows(META_CODE)).toEqual([]); + expect(await adminEntry(META_CODE)).toBeUndefined(); + expect(connectVerdict(META_CODE)).toBeUndefined(); + }); + + it('a code-defined datasource stays refused at the admin door', async () => { + const patch = refusal(await call('PATCH', `/datasources/${EXTERNAL}`, { label: 'Admin edit 21923' })); + expect({ status: patch.status, code: patch.code }).toEqual({ status: 400, code: 'DATASOURCE_ADMIN_ERROR' }); + expect(patch.message.startsWith(`Datasource '${EXTERNAL}' is code-defined and cannot be edited at runtime`), patch.message).toBe(true); + expect(await adminEntry(EXTERNAL)).toMatchObject({ origin: 'code' }); + }); +}); diff --git a/packages/services/service-datasource/package.json b/packages/services/service-datasource/package.json index f4abdc45e71..11a289650c2 100644 --- a/packages/services/service-datasource/package.json +++ b/packages/services/service-datasource/package.json @@ -39,6 +39,7 @@ "@objectstack/core": "workspace:*", "@objectstack/driver-memory": "workspace:*", "@objectstack/driver-sql": "workspace:*", + "@objectstack/metadata-core": "workspace:*", "@objectstack/spec": "workspace:*", "@objectstack/types": "workspace:*", "pg-connection-string": "^2.14.0" @@ -62,7 +63,6 @@ "devDependencies": { "@objectstack/driver-mongodb": "workspace:*", "@objectstack/driver-sqlite-wasm": "workspace:*", - "@objectstack/metadata-core": "workspace:*", "@objectstack/plugin-hono-server": "workspace:*", "@types/node": "^26.6.3", "tsup": "^8.5.1", diff --git a/packages/services/service-datasource/src/__tests__/datasource-admin-plugin.test.ts b/packages/services/service-datasource/src/__tests__/datasource-admin-plugin.test.ts index a35d49e3f58..6d1e2801519 100644 --- a/packages/services/service-datasource/src/__tests__/datasource-admin-plugin.test.ts +++ b/packages/services/service-datasource/src/__tests__/datasource-admin-plugin.test.ts @@ -11,7 +11,7 @@ import { import { registerDatasourceAdminRoutes } from '../admin-routes.js'; import { ENTITLED_CREDENTIAL, createSessionAuthService, createGrantsEngine } from './entitled-caller.fixture.js'; -import { assertEngineFindOnePredicate } from '@objectstack/metadata-core'; +import { assertEngineFindOnePredicate, hashSpec } from '@objectstack/metadata-core'; // [#10126] Pay the first transform of these dist-resolved workspace deps at MODULE // LOAD. Each is reached below through a dynamic `import()` inside an `it()` body or a @@ -88,6 +88,23 @@ function fakeFactory(over?: Partial & { onProbe?: () = }; } +/** The admin door, mounted the way `os serve` mounts it, over a booted service. */ +function adminDoor(service: unknown) { + const server = new HonoHttpServer(0); + const authService = createSessionAuthService(); + const grantsEngine = createGrantsEngine(); + registerDatasourceAdminRoutes(server, { + getService: (name: string) => + name === 'auth' ? authService : name === 'objectql' || name === 'data' ? grantsEngine : service, + } as any, '/api/v1'); + return (method: string, path: string, body?: unknown) => + server.getRawApp().fetch(new Request(`http://local/api/v1${path}`, { + method, + headers: { 'content-type': 'application/json', authorization: ENTITLED_CREDENTIAL }, + ...(body === undefined ? {} : { body: JSON.stringify(body) }), + })); +} + describe('DatasourceAdminServicePlugin: probe', () => { it('tests a connection through the driver factory (latency + version)', async () => { const { service } = await boot({ @@ -308,8 +325,14 @@ describe('DatasourceAdminServicePlugin: boot rehydration', () => { describe('DatasourceAdminServicePlugin: persistence + bound count', () => { it('lists code (artefact) + runtime records with origin, blocks remove while bound', async () => { - const { service, registry } = await boot({ driverFactory: fakeFactory() }); - // seed an artefact (code) datasource lacking explicit origin + // [#21923] An artefact (code) datasource is code because the host registers + // its name from code — the code-datasource set the runtime fills, as it + // does for every datasource an artifact declares — never because its + // record lacks an explicit `origin`. + const { service, registry } = await boot({ + driverFactory: fakeFactory(), + services: { 'code-datasource-names': new Set(['crm_primary']) }, + }); registry.set('datasource', new Map([['crm_primary', { name: 'crm_primary', driver: 'sqlite' }]])); // seed an object bound to a runtime datasource registry.set('object', new Map([['lead', { name: 'lead', datasource: 'reporting' }]])); @@ -485,23 +508,6 @@ describe('DatasourceAdminServicePlugin: runtime datasource durability', () => { return { ...b, data, built, optionsWarn, ctxWarn }; }; - /** The admin door, mounted the way `os serve` mounts it, over this boot's service. */ - const adminDoor = (service: unknown) => { - const server = new HonoHttpServer(0); - const authService = createSessionAuthService(); - const grantsEngine = createGrantsEngine(); - registerDatasourceAdminRoutes(server, { - getService: (name: string) => - name === 'auth' ? authService : name === 'objectql' || name === 'data' ? grantsEngine : service, - } as any, '/api/v1'); - return (method: string, path: string, body?: unknown) => - server.getRawApp().fetch(new Request(`http://local/api/v1${path}`, { - method, - headers: { 'content-type': 'application/json', authorization: ENTITLED_CREDENTIAL }, - ...(body === undefined ? {} : { body: JSON.stringify(body) }), - })); - }; - it('keeps the code definition served, refuses the admin edit, keeps the row, names it, and opens no pool from it', async () => { const b = await harness({ codeNames: new Set(['crm_wh']), codeSlotFirst: true }); @@ -555,4 +561,175 @@ describe('DatasourceAdminServicePlugin: runtime datasource durability', () => { expect(b.ctxWarn).not.toHaveBeenCalled(); }); }); + + // [#21923] The admin door's origin comes from PROVENANCE — the host's + // code-datasource set — never from the record, and a datasource the `/meta` + // door writes reaches this door in the same boot. The `/meta` door's write + // is simulated the way it lands: a row in `sys_metadata`, then the protocol's + // awaited `datasource` mutation projector, which is exactly what the plugin + // registers (the real door and repository are pinned over the showcase in + // `packages/qa/dogfood/test/datasource-meta-door-reaches-admin-door.dogfood.test.ts`). + describe('[#21923] origin from provenance; the /meta door reaches the admin door in the same boot', () => { + const CODE_NAMES = 'code-datasource-names'; + const now = new Date().toISOString(); + /** A row the `/meta` door's repository writes: the author's body as given, with its checksum. */ + const metaRow = (name: string, body: Record) => ({ + id: `meta_${name}`, + name, + type: 'datasource', + metadata: JSON.stringify({ name, ...body }), + checksum: hashSpec({ name, ...body }, 'datasource'), + state: 'active', + version: 1, + created_at: now, + updated_at: now, + }); + const pg = (host: string) => ({ driver: 'postgres', config: { host, database: 'db' } }); + /** What `AppPlugin.start()` registers for a code datasource. */ + const CODE = { name: 'crm_wh', label: 'Code warehouse', driver: 'postgres', origin: 'code', config: { host: 'code-host', database: 'wh' } }; + + /** One host: `sys_metadata` + a driver registry, a recording factory, the protocol's projector slot. */ + const host = async (opts: { codeNames?: Set } = {}) => { + const data = Object.assign(fakeSysMetadataEngine(), { drivers: [] as any[], evicted: [] as string[] }); + (data as any).registerDriver = (d: any) => data.drivers.push(d); + (data as any).getDriverByName = (n: string) => data.drivers.find((d) => d.name === n); + (data as any).unregisterDriver = (n: string) => { + data.evicted.push(n); + const i = data.drivers.findIndex((d) => d.name === n); + if (i >= 0) data.drivers.splice(i, 1); + return i >= 0; + }; + const built: any[] = []; + const factory: IDatasourceDriverFactory = { + supports: (id: string) => id === 'postgres', + create: async (spec) => { + built.push(spec); + return { connect: async () => {}, disconnect: async () => {} }; + }, + }; + const projectors = new Map Promise>(); + const protocol = { + registerMutationProjector: (type: string, projector: (evt: unknown) => Promise) => { + projectors.set(type, projector); + }, + }; + const b = await boot({ + driverFactory: factory, + services: { data, protocol, ...(opts.codeNames ? { [CODE_NAMES]: opts.codeNames } : {}) }, + }); + await b.metadata.register('datasource', CODE.name, CODE); + await b.plugin.start(b.ctx); + /** The `/meta` door's post-persistence step: the awaited projection, with the event it sends. */ + const metaDoorWrote = (name: string, state: 'active' | 'draft' | 'deleted') => + projectors.get('datasource')!({ type: 'datasource', name, state, organizationId: null }); + const listed = async (name: string) => (await b.service.listDatasources()).find((d) => d.name === name); + const builtFor = (name: string) => built.filter((s) => s.name === name); + return { ...b, data, built, builtFor, projectors, metaDoorWrote, listed }; + }; + + it('serves code only for a name in the code-datasource set, and runtime for every other, whatever the record says', async () => { + const b = await host({ codeNames: new Set(['crm_wh']) }); + await b.metadata.register('datasource', 'crm_wh', { ...CODE, origin: 'runtime' }); + await b.metadata.register('datasource', 'rt_none', { name: 'rt_none', label: 'No origin', ...pg('a') }); + await b.metadata.register('datasource', 'rt_code', { name: 'rt_code', label: 'Asserts code', origin: 'code', ...pg('b') }); + + expect((await b.listed('crm_wh'))?.origin).toBe('code'); + expect((await b.listed('rt_none'))?.origin).toBe('runtime'); + expect((await b.listed('rt_code'))?.origin).toBe('runtime'); + const svc = b.service as unknown as DatasourceAdminService; + expect((await svc.getDatasource('crm_wh'))?.origin).toBe('code'); + expect((await svc.getDatasource('rt_none'))?.origin).toBe('runtime'); + expect((await svc.getDatasource('rt_code'))?.origin).toBe('runtime'); + + // Editable as runtime, both of them; the code name stays refused. + await expect(b.service.updateDatasource('rt_none', { label: 'Edited' })).resolves.toMatchObject({ origin: 'runtime' }); + await expect(b.service.updateDatasource('rt_code', { label: 'Edited' })).resolves.toMatchObject({ origin: 'runtime' }); + const patch = await adminDoor(b.service)('PATCH', '/datasources/crm_wh', { label: 'Edited at runtime' }); + const answer = (await patch.json()) as { error?: { code?: string; message?: string } }; + expect({ status: patch.status, code: answer.error?.code }).toEqual({ status: 400, code: 'DATASOURCE_ADMIN_ERROR' }); + expect(String(answer.error?.message).startsWith("Datasource 'crm_wh' is code-defined"), answer.error?.message).toBe(true); + }); + + it('registers the datasource projector on the protocol at start()', async () => { + const b = await host(); + expect([...b.projectors.keys()]).toEqual(['datasource']); + }); + + it('a /meta save is listed and editable here in the same boot, with a live pool, whatever origin its body asserts', async () => { + const b = await host({ codeNames: new Set(['crm_wh']) }); + b.data.rows.push(metaRow('meta_none', { label: 'Meta none', ...pg('h1') })); + b.data.rows.push(metaRow('meta_code', { label: 'Meta code', origin: 'code', ...pg('h2') })); + await b.metaDoorWrote('meta_none', 'active'); + await b.metaDoorWrote('meta_code', 'active'); + + expect(await b.listed('meta_none')).toMatchObject({ origin: 'runtime', label: 'Meta none' }); + expect(await b.listed('meta_code')).toMatchObject({ origin: 'runtime', label: 'Meta code' }); + expect(b.builtFor('meta_none').map((s) => s.config)).toEqual([{ host: 'h1', database: 'db' }]); + expect(b.builtFor('meta_code').map((s) => s.config)).toEqual([{ host: 'h2', database: 'db' }]); + await expect(b.service.updateDatasource('meta_code', { label: 'Edited here' })).resolves.toMatchObject({ origin: 'runtime' }); + }); + + it('a /meta edit rebuilds the pool from the new row; a /meta delete leaves this door and evicts the pool', async () => { + const b = await host(); + b.data.rows.push(metaRow('meta_ds', { label: 'v1', ...pg('h1') })); + await b.metaDoorWrote('meta_ds', 'active'); + + const i = b.data.rows.findIndex((r) => r.name === 'meta_ds'); + b.data.rows[i] = metaRow('meta_ds', { label: 'v2', ...pg('h2') }); + await b.metaDoorWrote('meta_ds', 'active'); + expect(await b.listed('meta_ds')).toMatchObject({ origin: 'runtime', label: 'v2' }); + expect(b.builtFor('meta_ds').map((s) => s.config.host)).toEqual(['h1', 'h2']); + + b.data.rows.splice(b.data.rows.findIndex((r) => r.name === 'meta_ds'), 1); + await b.metaDoorWrote('meta_ds', 'deleted'); + expect(await b.listed('meta_ds')).toBeUndefined(); + expect(b.data.evicted).toContain('meta_ds'); + }); + + it('a /meta write under a code name — the repair DELETE of a stored shadow included — leaves the code definition served and opens no pool', async () => { + const b = await host({ codeNames: new Set(['crm_wh']) }); + b.data.rows.push(metaRow('crm_wh', { label: 'Shadow', origin: 'runtime', ...pg('shadow-host') })); + await b.metaDoorWrote('crm_wh', 'active'); + expect(await b.metadata.get('datasource', 'crm_wh')).toEqual(CODE); + + b.data.rows.splice(b.data.rows.findIndex((r) => r.name === 'crm_wh'), 1); + await b.metaDoorWrote('crm_wh', 'deleted'); + expect(await b.metadata.get('datasource', 'crm_wh')).toEqual(CODE); + expect((await b.listed('crm_wh'))?.origin).toBe('code'); + expect(b.builtFor('crm_wh')).toEqual([]); + expect(b.data.evicted).toEqual([]); + }); + + it('a peer signal pools a stored row by provenance: never one under a code name, and a /meta row with no origin as runtime', async () => { + const b = await host({ codeNames: new Set(['crm_wh']) }); + const handlers: Array<(msg: { payload: unknown }) => void> = []; + const bus = { + publish: async () => {}, + subscribe: (_channel: string, handler: (msg: { payload: unknown }) => void) => { + handlers.push(handler); + return () => {}; + }, + }; + (b.service as unknown as DatasourceAdminService).attachDatasourceMutationPubSub(bus as never, 'node-b'); + b.data.rows.push(metaRow('crm_wh', { label: 'Shadow', origin: 'runtime', ...pg('shadow-host') })); + b.data.rows.push(metaRow('peer_meta', { label: 'Peer meta', ...pg('peer-host') })); + + for (const name of ['crm_wh', 'peer_meta']) for (const h of handlers) h({ payload: { originNode: 'node-a', name } }); + await new Promise((r) => setTimeout(r, 0)); + + expect(b.builtFor('crm_wh')).toEqual([]); + expect(b.builtFor('peer_meta').map((s) => s.config.host)).toEqual(['peer-host']); + }); + + it('an admin-written row carries the checksum the /meta door\'s repository stamps on the body it stores', async () => { + const b = await host(); + await b.service.createDatasource({ name: 'ck_ds', label: 'v1', driver: 'postgres', config: { host: 'h', database: 'db' } }); + const row = () => b.data.rows.find((r) => r.name === 'ck_ds')!; + expect(row().checksum).toBe(hashSpec(JSON.parse(row().metadata as string), 'datasource')); + + await b.service.updateDatasource('ck_ds', { label: 'v2' }); + expect(JSON.parse(row().metadata as string).label).toBe('v2'); + expect(row().checksum).toBe(hashSpec(JSON.parse(row().metadata as string), 'datasource')); + }); + }); }); diff --git a/packages/services/service-datasource/src/__tests__/datasource-system-context.pin.test.ts b/packages/services/service-datasource/src/__tests__/datasource-system-context.pin.test.ts index e650ed37b9b..6edad3a0cef 100644 --- a/packages/services/service-datasource/src/__tests__/datasource-system-context.pin.test.ts +++ b/packages/services/service-datasource/src/__tests__/datasource-system-context.pin.test.ts @@ -128,7 +128,7 @@ describe('[#21913] runtime-datasource sys_metadata helpers carry the explicit sy const second = await boot(data); await second.plugin.start(second.ctx); expect((await second.service.listDatasources()).map((d: any) => d.name)).toContain('pin_ds'); - await (second.plugin as any).convergePool('pin_ds', () => data); + await (second.plugin as any).convergePool(second.ctx, 'pin_ds', () => data); await second.service.removeDatasource('pin_ds'); expect(data.rows.some((r) => r.name === 'pin_ds')).toBe(false); diff --git a/packages/services/service-datasource/src/datasource-admin-plugin.ts b/packages/services/service-datasource/src/datasource-admin-plugin.ts index 3b51fcf3ff9..a7f28b52b96 100644 --- a/packages/services/service-datasource/src/datasource-admin-plugin.ts +++ b/packages/services/service-datasource/src/datasource-admin-plugin.ts @@ -1,6 +1,7 @@ // Copyright (c) 2025 ObjectStack. Licensed under the Apache-2.0 license. import type { Plugin, PluginContext } from '@objectstack/core'; +import { hashSpec } from '@objectstack/metadata-core'; import { applyConversionsToStoredItem } from '@objectstack/spec'; import { registerMetadataTypeActions } from '@objectstack/spec/kernel'; import type { @@ -70,6 +71,17 @@ type DataEngineLike = Partial< Pick >; +/** + * [#21923] The slice of the `/meta` door's post-persistence mutation event + * (`MetadataMutationEvent` in `@objectstack/metadata-protocol`) the datasource + * projector reads — the row's address and resulting state, read structurally: + * this package does not depend on the protocol. + */ +interface MetadataDoorMutation { + name?: unknown; + state?: unknown; +} + /** * Durable persistence for runtime datasource records via the `sys_metadata` * table — the same store runtime objects use (the protocol writes objects there @@ -99,9 +111,21 @@ function newMetaId(): string { : `meta_${Date.now()}_${Math.random().toString(36).slice(2)}`; } +/** + * [#21923] Write a runtime datasource's durable row — WITH the `checksum` the + * metadata door's repository stamps on every row it writes + * (`SysMetadataRepository.put`: `hashSpec(body, ref.type)`, the one + * computation, imported rather than re-spelled). That column is the `/meta` + * door's optimistic lock: its reads hand out the row's checksum as the version + * a `PUT` / `DELETE` names as its parent, and its writes compare that parent + * against the stored column. A row written here without one could never be + * matched, so every `/meta` edit or removal of an admin-created datasource + * answered `409 METADATA_CONFLICT` — the two doors disagreeing on one record. + */ async function persistDatasourceRow(engine: DataEngineLike | undefined, record: { name: string }): Promise { if (!engine?.insert || !engine.findOne) return; // no durable store — in-memory only const now = new Date().toISOString(); + const checksum = hashSpec(record, DS_META_TYPE); const existing = await engine.findOne( SYS_METADATA, { where: { type: DS_META_TYPE, name: record.name, state: 'active' } }, @@ -110,7 +134,7 @@ async function persistDatasourceRow(engine: DataEngineLike | undefined, record: if (existing) { await engine.update?.( SYS_METADATA, - { metadata: JSON.stringify(record), updated_at: now, version: ((existing.version as number) || 0) + 1, state: 'active' }, + { metadata: JSON.stringify(record), checksum, updated_at: now, version: ((existing.version as number) || 0) + 1, state: 'active' }, { where: { id: existing.id }, context: SYSTEM_CTX }, ); } else { @@ -120,6 +144,7 @@ async function persistDatasourceRow(engine: DataEngineLike | undefined, record: type: DS_META_TYPE, scope: 'platform', metadata: JSON.stringify(record), + checksum, state: 'active', version: 1, created_at: now, @@ -366,15 +391,16 @@ export class DatasourceAdminServicePlugin implements Plugin { const config: DatasourceAdminServiceConfig = { probe: (input) => this.probe(factory(), input), + // [#21923] Every record this door serves carries the origin its + // PROVENANCE gives — `servedOrigin` — whatever the record says. listDatasourceRecords: async () => { const rows = ((await metadataOf()?.list('datasource')) ?? []) as StoredDatasource[]; - // Artefact-loaded rows may omit `origin`; treat them as code-defined. - return rows.map((r) => ({ ...r, origin: r.origin ?? 'code' })); + return rows.map((r) => ({ ...r, origin: servedOrigin(ctx, r.name) })); }, getDatasourceRecord: async (name) => { const row = (await metadataOf()?.get('datasource', name)) as StoredDatasource | undefined; - return row ? { ...row, origin: row.origin ?? 'code' } : undefined; + return row ? { ...row, origin: servedOrigin(ctx, name) } : undefined; }, putDatasourceRecord: async (record) => { @@ -498,7 +524,7 @@ export class DatasourceAdminServicePlugin implements Plugin { // [#13805] The receive half of the datasource cluster bridge — see // `convergePool` below for the decision table. The engine is resolved // per call for the same reason every other seam here resolves lazily. - convergePool: (name) => this.convergePool(name, engineOf), + convergePool: (name) => this.convergePool(ctx, name, engineOf), // The admin list's `status` reads the connection service's retained // verdicts (framework#3827). Resolved lazily per call: the service exists @@ -563,9 +589,67 @@ export class DatasourceAdminServicePlugin implements Plugin { // over it (#21922, see restoreRuntimeDatasources). await this.restoreRuntimeDatasources(ctx); await this.rehydratePools(); + this.projectMetadataDoorWrites(ctx); if (this.service) await ctx.trigger('datasource-admin:ready', this.service); } + /** + * [#21923] Make a datasource the `/meta` door writes reach THIS door in the + * same boot, as a restart would make it: the door's save, publish, revert, + * rollback and delete persist to `sys_metadata` (and the SchemaRegistry) + * but never touch the MetadataService slot this door lists, so a datasource + * saved there was missing here until the next boot restore registered it. + * + * The seam is the protocol's awaited per-type mutation projector (ADR-0094), + * not its fire-and-forget `onMetadataMutation` listener: the projection runs + * inside the write, so the `/meta` answer is returned only once this door + * lists the datasource, and a projection that fails is reported on that + * answer (`projectionApplied`) as well as logged by the protocol. + * + * Resolved at `start()`, once every plugin has inited. No protocol, no + * metadata door: nothing writes a datasource this plugin does not already + * see. + */ + private projectMetadataDoorWrites(ctx: PluginContext): void { + const protocol = safeGetService<{ + registerMutationProjector?(type: string, projector: (evt: MetadataDoorMutation) => Promise): void; + }>(ctx, 'protocol'); + if (typeof protocol?.registerMutationProjector !== 'function') return; + protocol.registerMutationProjector(DS_META_TYPE, (evt) => this.applyMetadataDoorWrite(ctx, evt)); + } + + /** + * [#21923] One `/meta` door write of a datasource, applied to this door's + * two derived states — boot's {@link restoreRuntimeDatasources} and + * {@link rehydratePools}, narrowed to one name and re-run on demand: + * + * - the MetadataService slot follows the durable row: registered from it + * when an active row exists, unregistered when none does (a delete); + * - the live pool converges on the same row through + * {@link convergePool}, so a saved datasource gets the pool an admin + * create gets, an edit rebuilds it, and a delete evicts it. + * + * Read from `sys_metadata`, the single source of truth, never from the + * event (a delete carries no body). A `draft` save leaves the active row as + * it was, so it changes nothing here. ⛔ A name the host defines in code is + * never registered, unregistered or pooled from a stored row — code wins on + * a collision, as at the boot restore; the `/meta` door's `DELETE` of such a + * row is the repair, and must leave the code definition served. + */ + private async applyMetadataDoorWrite(ctx: PluginContext, evt: MetadataDoorMutation): Promise { + if (evt?.state === 'draft' || typeof evt?.name !== 'string' || evt.name === '') return; + const name = evt.name; + if (servedOrigin(ctx, name) === 'code') return; + const engineOf = (): DataEngineLike | undefined => safeGetService(ctx, 'data'); + const metadata = safeGetService(ctx, 'metadata'); + if (metadata?.register && metadata.unregister) { + const row = await loadDatasourceRow(engineOf(), name); + if (row) await metadata.register(DS_META_TYPE, name, row); + else await metadata.unregister(DS_META_TYPE, name); + } + await this.convergePool(ctx, name, engineOf); + } + /** * Reload persisted runtime datasource rows (sys_metadata) into the registry. * @@ -673,11 +757,15 @@ export class DatasourceAdminServicePlugin implements Plugin { * metadata registry (which, on the host-config boot, is per-replica state * with no cluster seam — the read a peer must not converge from): * - * - **no active runtime row** (deleted, disabled, or a name this plugin - * never pooled) → evict the pool THIS plugin built, if any — through - * `unregisterPool` → `disconnect`, the #13578 eviction door. A name it - * never pooled is left alone, so a stray signal can never reach a - * code-defined pool the host stack owns. + * - **no active runtime row** (deleted, disabled, a name this plugin + * never pooled, or — [#21923] — a name the host defines in code, read + * from the code-datasource set and never from the row's own `origin`) + * → evict the pool THIS plugin built, if any — through `unregisterPool` + * → `disconnect`, the #13578 eviction door. A name it never pooled is + * left alone, so a stray signal can never reach a code-defined pool the + * host stack owns, nor open one from a stored row under a code name. + * Every other row is runtime by provenance, and is pooled as one + * whatever its body says — a `/meta` save carries no `origin`. * - **a row this replica has no pool for** → build it (the create case, and * re-enabling after `active: false`). * - **a row whose connectivity-bearing fields differ from what the live @@ -696,16 +784,18 @@ export class DatasourceAdminServicePlugin implements Plugin { * This method converges the DRIVER registry only. */ private async convergePool( + ctx: PluginContext, name: string, engineOf: () => DataEngineLike | undefined, ): Promise { const cfg = this.config; if (!cfg?.registerPool || !cfg.unregisterPool) return; - const row = await loadDatasourceRow(engineOf(), name); + const stored = servedOrigin(ctx, name) === 'code' ? undefined : await loadDatasourceRow(engineOf(), name); + const row: StoredDatasource | undefined = stored ? { ...stored, origin: 'runtime' } : undefined; const live = this.livePoolRecords.get(name); - if (!row || row.origin !== 'runtime' || row.active === false) { + if (!row || row.active === false) { if (live) await cfg.unregisterPool(name); return; } @@ -814,6 +904,28 @@ function codeDatasourceNamesOf(ctx: PluginContext): { has(name: string): boolean return typeof names?.has === 'function' ? (names as { has(name: string): boolean }) : undefined; } +/** + * [#21923] The origin the admin door serves for a datasource, from its + * PROVENANCE: `code` when the host registers that name from code (the + * {@link CODE_DATASOURCE_NAMES_SERVICE} set the runtime fills in Phase 1, the + * one answer the boot restore and the `/meta` door's refusal read too), and + * `runtime` for every other name — a row the boot restore registered from + * `sys_metadata`, one the `/meta` door saved, one this door created. + * + * ⛔ Never the record's own `origin`, and never a default for its absence: + * the MetadataService slot holds whatever was registered last, and a stored + * row's body says whatever its writer put there — a `/meta` save carries no + * `origin`, or asserts `origin: 'code'`. Reading the record turned a datasource + * an operator created at runtime into a read-only "code-defined" one after a + * restart (`origin ?? 'code'`). + * + * A host that composes no code-datasource producer registers no set, and then + * nothing is code, as for the boot restore. + */ +function servedOrigin(ctx: PluginContext, name: string): 'code' | 'runtime' { + return codeDatasourceNamesOf(ctx)?.has(name) ? 'code' : 'runtime'; +} + function errMsg(err: unknown): string { return err instanceof Error ? err.message : String(err); } diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 0fa188c53b3..4188f8455e2 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -2561,6 +2561,9 @@ importers: '@objectstack/driver-turso': specifier: workspace:^ version: link:../../drivers/driver-turso + '@objectstack/metadata-core': + specifier: workspace:* + version: link:../../metadata-core '@objectstack/spec': specifier: workspace:* version: link:../../spec @@ -2577,9 +2580,6 @@ importers: '@objectstack/driver-sqlite-wasm': specifier: workspace:* version: link:../../drivers/driver-sqlite-wasm - '@objectstack/metadata-core': - specifier: workspace:* - version: link:../../metadata-core '@objectstack/plugin-hono-server': specifier: workspace:* version: link:../../plugins/plugin-hono-server