diff --git a/.changeset/21505-read-scope-temporal-coercion.md b/.changeset/21505-read-scope-temporal-coercion.md new file mode 100644 index 00000000000..1dd56307b33 --- /dev/null +++ b/.changeset/21505-read-scope-temporal-coercion.md @@ -0,0 +1,17 @@ +--- +"@objectstack/service-analytics": minor +--- + +fix(service-analytics)!: the analytics read scope and the draft preview compare a temporal comparand in the column's storage form, as the engine does (ADR-0053 D-A1 / D-A2) (#21505) + +Clause-②: yes (narrowing) + + + +**BREAKING**: this changes the rows two analytics faces select for a value comparison on a declared temporal column, in both directions, onto the rows `engine.find` selects for the same filter: on some filters fewer rows than before, on others more. The faces are the row-level read scope compiled into the native statement, and the draft preview (`queryDataset` with `previewDrafts`). It ships as `minor` under the launch-window convention for answer changes. No export is removed, no accepted input is refused and no error code changes. + +**The read scope.** `compileScopedFilterToSql` takes two new optional members in its options, `coerceTemporalFilterValue(field, value)` and `coerceTemporalFilterColumn(field, columnSql)`. Together they are the driver's `temporalFilterValue` / `temporalFilterColumnSql` pair, bound to the object the scope reads. After the shared lowering, every value comparison binds its comparand through the first and reads its column through the second: equality, `$ne`, the four orderings, `$in`, `$nin` and `$between`. Null tests, `$empty` and the text operators read the column as stored. An absent member is identity: the comparand and the column stay as written, which is what a host that passes neither got before. `NativeSQLStrategy` (the read scope merged into the native statement) and the `ObjectQLStrategy` echo (`/analytics/sql`) pass the context's pair, which `AnalyticsServicePlugin` wires to the driver. Before, the comparand was bound as written and the database read it by its own rules, on SQLite and on PostgreSQL whatever the server's time zone. + +**The draft preview.** It has no driver, so each value comparison on a column the host declares `datetime`, `date` or `time` now puts both sides in the storage form `@objectstack/core`'s `temporalStorageForm` gives: the comparand, and the drafted row's value, as `driver-memory` reads them. Before, it compared the two spellings as text. A column the host names no type for is compared as written, as before. + +A `date` column answered the engine's rows on both faces before and still does when both sides are spelled as days. No `@objectstack/spec` contract changes and no dependency edge is added. A host that calls `compileScopedFilterToSql` directly gets the coercion by passing the pair from its driver. diff --git a/packages/services/service-analytics/src/__tests__/analytics-faces-one-lowering.test.ts b/packages/services/service-analytics/src/__tests__/analytics-faces-one-lowering.test.ts index d5679db8264..66dd08b68af 100644 --- a/packages/services/service-analytics/src/__tests__/analytics-faces-one-lowering.test.ts +++ b/packages/services/service-analytics/src/__tests__/analytics-faces-one-lowering.test.ts @@ -409,16 +409,22 @@ for (const cell of DB_CELLS) { }); } - // F9 binds a comparand as written (no storage-form coercion), so its - // datetime cells are pinned on SQLite, whose stored form is ISO text. - // PostgreSQL reads a bare-day text bound in the session's zone; that is - // the read scope's temporal-coercion question, not its lowering. - it.skipIf(cell.id !== 'sqlite')('the read scope (F9), typed by its declared value shape, answers the same rows', async () => { + // [#21505] F9 binds each comparand through the driver's coercion pair + // (ADR-0053 D-A1 / D-A2), as both of its consumers wire it, so its + // datetime cells hold on PostgreSQL too, where a bare-day text bound + // would otherwise be read in the session's zone. + it('the read scope (F9), typed by its declared value shape, answers the same rows', async () => { const declaredValueShape = (field: string) => (DECLARED[field] ? { type: DECLARED[field], multiple: false } : undefined); for (const [label, query, expected] of CELLS) { if (!query.where) continue; - const { sql, params } = compileScopedFilterToSql(query.where as FilterCondition, OBJECT, { declaredValueShape, dialect: 'sqlite' }); - const rows = await (engine as any).execute(`select "id" from "${OBJECT}" where ${sql}`, { args: params, object: OBJECT }); + const { sql, params } = compileScopedFilterToSql(query.where as FilterCondition, OBJECT, { + declaredValueShape, + dialect: cell.id === 'pg' ? 'postgres' : 'sqlite', + coerceTemporalFilterValue: (field, value) => driver.temporalFilterValue(OBJECT, field, value), + coerceTemporalFilterColumn: (field, columnSql) => driver.temporalFilterColumnSql(OBJECT, field, columnSql), + }); + const res = await (engine as any).execute(`select "id" from "${OBJECT}" where ${sql}`, { args: params, object: OBJECT }); + const rows = Array.isArray(res) ? res : (res as { rows: Array> }).rows; expect(ids(rows as Array>), label).toBe(expected); } }); diff --git a/packages/services/service-analytics/src/__tests__/read-scope-temporal-coercion.test.ts b/packages/services/service-analytics/src/__tests__/read-scope-temporal-coercion.test.ts new file mode 100644 index 00000000000..94821df1885 --- /dev/null +++ b/packages/services/service-analytics/src/__tests__/read-scope-temporal-coercion.test.ts @@ -0,0 +1,369 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * [#21505, ADR-0053 D-A1 / D-A2] The read scope binds a temporal comparand in + * the column's storage form, through the driver's own coercion pair. + * + * `compileScopedFilterToSql` takes `coerceTemporalFilterValue` and + * `coerceTemporalFilterColumn`, the engine door's pair + * (`IDataDriver.temporalFilterValue` / `temporalFilterColumnSql`) bound to the + * object, and applies them after the shared lowering to every value + * comparison. Both of its consumers pass the context's pair. Absent members are + * identity. The draft preview, which has no driver, puts both sides of each + * value comparison in the same storage form with `@objectstack/core`'s + * `temporalStorageForm`, by the column's declared type. + * + * Measured at `d2f452b88`, before this change, with the comparand bound as + * written: the read scope differed from `engine.find` on 7 of the 12 + * `datetime` cells below on SQLite and 9 of 12 on PostgreSQL 16 under a + * non-UTC server, and on 0 of the 4 `date` cells. The native face, which runs + * the read scope, differed on the same cells end to end; the ObjectQL face + * (the engine) on none. The preview differed on 7 of the 12 `datetime` cells + * and on 0 of the 4 `date` cells. + * + * The PostgreSQL cells run where `OS_TEST_POSTGRES_URL` is set, and are a + * named skip otherwise. Each asserts its server is not on UTC, because a UTC + * server reads a bare day as UTC midnight and would pass with no coercion at + * all. Each owns its table, dropped before and after. + */ + +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import { ObjectQL } from '@objectstack/objectql'; +import { SqlDriver } from '@objectstack/driver-sql'; +import { lowerFilterCondition, type Cube, type FilterCondition } from '@objectstack/spec/data'; +import { DatasetSchema } from '@objectstack/spec/ui'; +import { AnalyticsService } from '../analytics-service.js'; +import { AnalyticsServicePlugin } from '../plugin.js'; +import { compileScopedFilterToSql, type ReadScopeCompileOptions } from '../read-scope-sql.js'; + +// ── Where the pair applies: after the lowering, on the value comparisons ───── + +const DECLARED: Record = { signed_at: 'datetime', due_on: 'date', note: 'text' }; +const declaredValueShape = (field: string) => (DECLARED[field] ? { type: DECLARED[field], multiple: false } : undefined); + +describe('[#21505] the compiler applies the coercion pair after the lowering, to every value comparison', () => { + const seen: Array<[string, unknown]> = []; + const RECORDING: ReadScopeCompileOptions = { + declaredValueShape, + dialect: 'sqlite', + coerceTemporalFilterValue: (field, value) => { seen.push([field, value]); return `C(${String(value)})`; }, + coerceTemporalFilterColumn: (_field, columnSql) => `COL(${columnSql})`, + }; + const compile = (scope: FilterCondition, opts: ReadScopeCompileOptions = RECORDING) => compileScopedFilterToSql(scope, 't', opts); + + it('the comparand the hook receives is the lowered one (ADR-0053 D-E3: widen first, convert second)', () => { + seen.length = 0; + expect(compile({ signed_at: { $lte: '2026-07-28' } })).toEqual({ sql: 'COL("t"."signed_at") < ?', params: ['C(2026-07-29)'] }); + expect(seen).toEqual([['signed_at', '2026-07-29']]); + expect(compile({ signed_at: { $between: ['2026-07-28', '2026-07-28'] } })).toEqual({ + sql: '(COL("t"."signed_at") >= ? AND COL("t"."signed_at") < ?)', + params: ['C(2026-07-28)', 'C(2026-07-29)'], + }); + }); + + it('every value comparison takes both halves', () => { + const cells: Array<[FilterCondition, string, unknown[]]> = [ + [{ due_on: '2026-07-28' }, 'COL("t"."due_on") = ?', ['C(2026-07-28)']], + [{ due_on: { $eq: '2026-07-28' } }, 'COL("t"."due_on") = ?', ['C(2026-07-28)']], + [{ due_on: { $ne: '2026-07-28' } }, '(("t"."due_on" IS NULL OR COL("t"."due_on") <> ?))', ['C(2026-07-28)']], + [ + { due_on: { $gt: 'a', $gte: 'b', $lt: 'c', $lte: 'd' } }, + '(COL("t"."due_on") > ? AND COL("t"."due_on") >= ? AND COL("t"."due_on") < ? AND COL("t"."due_on") <= ?)', + ['C(a)', 'C(b)', 'C(c)', 'C(d)'], + ], + [{ due_on: { $in: ['a', 'b'] } }, 'COL("t"."due_on") IN (?, ?)', ['C(a)', 'C(b)']], + [{ due_on: { $nin: ['a'] } }, '(("t"."due_on" IS NULL OR COL("t"."due_on") NOT IN (?)))', ['C(a)']], + [{ due_on: { $between: ['a', 'b'] } }, 'COL("t"."due_on") BETWEEN ? AND ?', ['C(a)', 'C(b)']], + ]; + for (const [scope, sql, params] of cells) expect(compile(scope), JSON.stringify(scope)).toEqual({ sql, params }); + }); + + it('a null test, `$empty` and a text arm read the column as stored and bind no coerced value', () => { + seen.length = 0; + expect(compile({ due_on: null })).toEqual({ sql: '"t"."due_on" IS NULL', params: [] }); + expect(compile({ note: { $null: true } })).toEqual({ sql: '"t"."note" IS NULL', params: [] }); + expect(compile({ note: { $empty: true } })).toEqual({ sql: '("t"."note" IS NULL OR "t"."note" = ?)', params: [''] }); + expect(compile({ note: { $contains: 'x' } }).sql).not.toContain('COL('); + expect(seen).toEqual([]); + }); + + it('absent members are identity: the same SQL and binds as identity members', () => { + const identity: ReadScopeCompileOptions = { + declaredValueShape, + dialect: 'sqlite', + coerceTemporalFilterValue: (_f, v) => v, + coerceTemporalFilterColumn: (_f, c) => c, + }; + const scopes: FilterCondition[] = [ + { signed_at: { $lte: '2026-07-28' } }, + { signed_at: { $ne: '2026-07-28' }, due_on: { $in: ['2026-07-28'] } }, + { $or: [{ note: 'n' }, { $not: { signed_at: { $between: ['2026-07-28', '9999-12-31'] } } }] }, + ]; + for (const scope of scopes) { + expect(compile(scope, { declaredValueShape, dialect: 'sqlite' }), JSON.stringify(scope)).toEqual(compile(scope, identity)); + } + }); +}); + +// ── The engine's rows, on a real engine, on SQLite and on PostgreSQL ────────── + +const OBJECT = 'os21505_coercion'; +const LEDGER = { + name: OBJECT, + label: 'Coercion', + fields: { + signed_at: { name: 'signed_at', type: 'datetime' as const }, + due_on: { name: 'due_on', type: 'date' as const }, + note: { name: 'note', type: 'text' as const }, + }, +}; +const ROWS = [ + { id: 'r1', signed_at: '2026-07-27T10:00:00.000Z', due_on: '2026-07-27', note: '2026-07-27' }, + { id: 'r2', signed_at: '2026-07-28T00:00:00.000Z', due_on: '2026-07-28', note: '2026-07-28' }, + { id: 'r3', signed_at: '2026-07-28T10:00:00.000Z', due_on: '2026-07-28', note: '2026-07-28 late' }, + { id: 'r4', signed_at: '2026-07-29T10:00:00.000Z', due_on: '2026-07-29', note: 'n' }, + { id: 'r5', signed_at: null, due_on: null, note: null }, +]; +/** The same instants and days as {@link ROWS}, drafted in other spellings an author can write. */ +const RESPELLED = [ + { id: 'r1', signed_at: new Date(Date.UTC(2026, 6, 27, 10)), due_on: '2026-07-27T23:00:00Z', note: '2026-07-27' }, + { id: 'r2', signed_at: '2026-07-28T00:00:00Z', due_on: '2026-07-28T00:00:00.000Z', note: '2026-07-28' }, + { id: 'r3', signed_at: '2026-07-28 10:00:00', due_on: '2026-07-28', note: '2026-07-28 late' }, + { id: 'r4', signed_at: '2026-07-29T18:00:00+08:00', due_on: '2026-07-29', note: 'n' }, + { id: 'r5', signed_at: null, due_on: null, note: null }, +]; +const CUBE = { + name: 'os21505_cube', + title: 'Coercion', + sql: OBJECT, + public: true, + measures: { n: { type: 'count', sql: '*', label: 'n' } }, + dimensions: { id: { type: 'string', sql: 'id', label: 'Id' } }, +} as unknown as Cube; + +/** Each cell: a label, the scope, and the rows `engine.find` answers for it. */ +const CELLS: ReadonlyArray = [ + ['datetime $between one day', { signed_at: { $between: ['2026-07-28', '2026-07-28'] } }, 'r2,r3'], + ['datetime $between to the last day', { signed_at: { $between: ['2026-07-28', '9999-12-31'] } }, 'r2,r3,r4'], + ['datetime $ne a day', { signed_at: { $ne: '2026-07-28' } }, 'r1,r3,r4,r5'], + ['datetime equality on a day', { signed_at: '2026-07-28' }, 'r2'], + ['datetime $gte a day', { signed_at: { $gte: '2026-07-28' } }, 'r2,r3,r4'], + ['datetime $lte a day', { signed_at: { $lte: '2026-07-28' } }, 'r1,r2,r3'], + ['datetime $gt a day', { signed_at: { $gt: '2026-07-28' } }, 'r3,r4'], + ['datetime $lt a day', { signed_at: { $lt: '2026-07-28' } }, 'r1'], + ['datetime $in [a day]', { signed_at: { $in: ['2026-07-28'] } }, 'r2'], + ['datetime $nin [a day]', { signed_at: { $nin: ['2026-07-28'] } }, 'r1,r3,r4,r5'], + ['datetime $gte a zone-naive time', { signed_at: { $gte: '2026-07-28 05:00' } }, 'r3,r4'], + ['datetime $ne under $not under $or', { $or: [{ note: 'n' }, { $not: { signed_at: { $ne: '2026-07-28' } } }] }, 'r2,r4'], + ['date $between one day (control)', { due_on: { $between: ['2026-07-28', '2026-07-28'] } }, 'r2,r3'], + ['date $ne a day (control)', { due_on: { $ne: '2026-07-28' } }, 'r1,r4,r5'], + ['date $lte a day (control)', { due_on: { $lte: '2026-07-28' } }, 'r1,r2,r3'], + ['date equality on a day (control)', { due_on: '2026-07-28' }, 'r2,r3'], +]; + +const quiet = { debug() {}, info() {}, warn() {}, error() {}, child() { return quiet; } }; +/** The ids of a result, sorted: a row array, or a raw `pg` result's `rows`. */ +const ids = (res: unknown): string => + ((Array.isArray(res) ? res : (res as { rows: unknown[] }).rows) as Array>) + .map((r) => String(r.id)).sort().join(','); + +interface DbCell { id: 'sqlite' | 'pg'; dialect: string; config: () => Record | null } +const DB_CELLS: readonly DbCell[] = [ + { id: 'sqlite', dialect: 'sqlite', config: () => ({ client: 'better-sqlite3', connection: { filename: ':memory:' }, useNullAsDefault: true }) }, + { + id: 'pg', + dialect: 'postgres', + config: () => (process.env.OS_TEST_POSTGRES_URL ? { client: 'pg', connection: process.env.OS_TEST_POSTGRES_URL } : null), + }, +]; + +for (const cell of DB_CELLS) { + const config = cell.config(); + describe.skipIf(!config)( + `[#21505] the read scope admits the engine's rows (${cell.id})${config ? '' : ' (skipped: set OS_TEST_POSTGRES_URL to run this cell)'}`, + () => { + let driver: SqlDriver; + let engine: ObjectQL; + let faces: { native: AnalyticsService; objectql: AnalyticsService }; + let scope: FilterCondition | null = null; + let rawStatements = 0; + const drop = async () => { + if (cell.id === 'pg') await (driver as any)?.execute(`drop table if exists ${OBJECT}`).catch(() => {}); + }; + + beforeAll(async () => { + driver = new SqlDriver(config as never); + await drop(); + engine = new ObjectQL({ logger: quiet } as never); + engine.registerDriver(driver as never, true); + await engine.init(); + engine.registry.registerObject(LEDGER as never); + await engine.syncSchemas(); + for (const row of ROWS) await engine.insert(OBJECT, { ...row } as never); + const realExecute = (engine as any).execute.bind(engine); + (engine as any).execute = (sql: string, opts?: { object?: string }) => { + if (opts?.object === OBJECT) rawStatements += 1; + return realExecute(sql, opts); + }; + // The plugin's own composition, with a host read scope: the native + // face, and the same narrowed to the engine aggregate (whose echo is + // the `/analytics/sql` statement). + const composed: Record = {}; + for (const [face, caps] of [ + ['native', undefined], + ['objectql', () => ({ nativeSql: false, objectqlAggregate: true, inMemory: false })], + ] as const) { + const registered: Record = {}; + await new AnalyticsServicePlugin({ cubes: [CUBE], getReadScope: () => scope, ...(caps ? { queryCapabilities: caps } : {}) } as never).init({ + getService: (name: string) => (name === 'data' ? engine : registered[name]), + registerService: (name: string, svc: unknown) => { registered[name] = svc; }, + replaceService: (name: string, svc: unknown) => { registered[name] = svc; }, + hook: () => {}, + logger: quiet, + } as never); + composed[face] = registered.analytics as AnalyticsService; + } + faces = composed as typeof faces; + }); + afterAll(async () => { + await drop(); + try { await engine?.destroy(); } catch { /* noop */ } + }); + + it.skipIf(cell.id !== 'pg')('the server is not on UTC (the premise of the PostgreSQL cells)', async () => { + const res = await (driver as any).execute('show timezone'); + const zone = String(Object.values((res as { rows: Array> }).rows[0])[0]); + expect(['UTC', 'Etc/UTC', 'GMT', 'Etc/GMT', 'UCT', 'Zulu']).not.toContain(zone); + }); + + it('compiled with the driver\'s pair, every cell admits the engine\'s rows, and the date control too', async () => { + const options: ReadScopeCompileOptions = { + declaredValueShape, + dialect: cell.dialect, + coerceTemporalFilterValue: (field, value) => driver.temporalFilterValue(OBJECT, field, value), + coerceTemporalFilterColumn: (field, columnSql) => driver.temporalFilterColumnSql(OBJECT, field, columnSql), + }; + for (const [label, where, expected] of CELLS) { + expect(ids(await engine.find(OBJECT, { where, fields: ['id'] } as never)), `${label}: engine.find`).toBe(expected); + const { sql, params } = compileScopedFilterToSql(where, OBJECT, options); + const rows = await (engine as any).execute(`select "id" from "${OBJECT}" where ${sql}`, { args: params, object: OBJECT }); + expect(ids(rows), `${label}: the read scope`).toBe(expected); + } + }); + + it('end to end, the native face scoped by a host getReadScope answers the engine\'s rows', async () => { + for (const [label, where, expected] of CELLS) { + scope = where; + const before = rawStatements; + const res = await faces.native.query({ cube: CUBE.name, measures: ['n'], dimensions: ['id'] } as never); + expect(ids(res.rows), `${label}: the native face`).toBe(expected); + expect(rawStatements - before, `${label}: the native strategy answered`).toBeGreaterThanOrEqual(1); + } + scope = null; + }); + + it('the ObjectQL echo prints the comparand the executed statement binds', async () => { + scope = { signed_at: { $ne: '2026-07-28' } }; + const echo = await faces.objectql.generateSql({ cube: CUBE.name, measures: ['n'], dimensions: ['id'] } as never); + scope = null; + expect(echo.params).toContain(driver.temporalFilterValue(OBJECT, 'signed_at', '2026-07-28')); + expect(echo.params).not.toContain('2026-07-28'); + }); + + it('the draft preview (queryDataset previewDrafts) answers the engine\'s rows over drafted rows in either spelling', async () => { + // Window ends written shorter than the stored instant, beside the cells. + const windows: Array<[string, [string, string]]> = [ + ['datetime window to a zone-naive minute', ['2026-07-28', '2026-07-28T10:00']], + ['datetime window to a zone-naive second', ['2026-07-28', '2026-07-28T10:00:00']], + ]; + for (const [label, [start, end]] of windows) { + expect(ids(await engine.find(OBJECT, { where: { signed_at: { $gte: start, $lte: end } }, fields: ['id'] } as never)), `${label}: engine.find`).toBe('r2,r3'); + } + const dataset = DatasetSchema.parse({ + name: 'os21505_preview', + label: 'Coercion preview', + object: OBJECT, + dimensions: [{ name: 'id', field: 'id', type: 'string' }, { name: 'signed_at', field: 'signed_at', type: 'date' }], + measures: [{ name: 'row_count', aggregate: 'count' }], + }); + for (const [spelling, drafted] of [['canonical', ROWS], ['respelled', RESPELLED]] as const) { + // The live path is not wired, so an answer can only come from the preview. + const svc = new AnalyticsService({ + sourceFieldMeta: (object: string, field: string) => (object === OBJECT && DECLARED[field] ? { type: DECLARED[field] } : undefined), + queryCapabilities: () => ({ nativeSql: false, objectqlAggregate: true, inMemory: false }), + executeAggregate: async () => { throw new Error('the live path ran: the preview did not answer'); }, + draftRowsResolver: async (object: string) => (object === OBJECT ? drafted.map((r) => ({ ...r })) : null), + } as never); + const preview = async (selection: Record) => + ids((await svc.queryDataset(dataset as never, { dimensions: ['id'], measures: ['row_count'], ...selection } as never, { tenantId: 'org_A' } as never, { previewDrafts: true })).rows); + for (const [label, where, expected] of CELLS) { + expect(await preview({ runtimeFilter: where }), `${spelling} rows, ${label}: the preview`).toBe(expected); + } + for (const [label, dateRange] of windows) { + expect(await preview({ timeDimensions: [{ dimension: 'signed_at', dateRange }] }), `${spelling} rows, ${label}: the preview`).toBe('r2,r3'); + } + } + }); + }, + ); +} + +// ── The column half: a SQLite datetime column the driver has not certified ──── + +/** + * An external object (ADR-0015) is registered with no backfill, so its SQLite + * `datetime` column is never certified canonical and may hold the forms written + * before the canonical convention. The driver reads such a column through its + * repair expression, and `temporalFilterColumnSql` hands that expression to a + * raw-SQL caller. Coercing the comparand alone keeps half the defect (D-A2). + */ +describe('[#21505] the column half reads an uncertified SQLite datetime column as the driver does', () => { + const LEGACY = 'os21505_legacy'; + let driver: SqlDriver; + + beforeAll(async () => { + driver = new SqlDriver({ client: 'better-sqlite3', connection: { filename: ':memory:' }, useNullAsDefault: true } as never); + const knex = (driver as any).knex; + await knex.schema.createTable(LEGACY, (t: any) => { + t.string('id').primary(); + t.specificType('at', 'datetime'); + }); + await knex(LEGACY).insert([ + { id: 'l1', at: Date.UTC(2026, 6, 27, 10) }, + { id: 'l2', at: '2026-07-28T00:00:00.000Z' }, + { id: 'l3', at: '2026-07-28 10:00:00' }, + { id: 'l4', at: '2026-07-29T18:00:00+08:00' }, + { id: 'l5', at: null }, + ]); + driver.registerExternalObject({ name: LEGACY, fields: { at: { name: 'at', type: 'datetime' } } }); + }); + afterAll(async () => { + try { await driver?.disconnect(); } catch { /* noop */ } + }); + + it('the fixture is the uncertified state: the driver wraps this column', () => { + expect(driver.temporalFilterColumnSql(LEGACY, 'at', '"c"')).not.toBe('"c"'); + }); + + it('with the pair, the read scope admits the driver\'s own rows', async () => { + const options: ReadScopeCompileOptions = { + declaredValueShape: (field) => (field === 'at' ? { type: 'datetime', multiple: false } : undefined), + dialect: 'sqlite', + coerceTemporalFilterValue: (field, value) => driver.temporalFilterValue(LEGACY, field, value), + coerceTemporalFilterColumn: (field, columnSql) => driver.temporalFilterColumnSql(LEGACY, field, columnSql), + }; + const cells: Array<[FilterCondition, string]> = [ + [{ at: { $gte: '2026-07-28' } }, 'l2,l3,l4'], + [{ at: { $lt: '2026-07-28' } }, 'l1'], + [{ at: { $ne: '2026-07-28' } }, 'l1,l3,l4,l5'], + [{ at: { $in: ['2026-07-28 10:00'] } }, 'l3'], + ]; + for (const [where, expected] of cells) { + const lowered = lowerFilterCondition(where, { isDatetimeColumn: (field) => field === 'at' }); + expect(ids(await driver.find(LEGACY, { where: lowered, fields: ['id'] } as never)), `${JSON.stringify(where)}: the driver`).toBe(expected); + const { sql, params } = compileScopedFilterToSql(where, LEGACY, options); + const rows = await (driver as any).knex.raw(`select "id" from "${LEGACY}" where ${sql}`, params); + expect(ids(rows), `${JSON.stringify(where)}: the read scope`).toBe(expected); + } + }); +}); diff --git a/packages/services/service-analytics/src/preview-evaluator.ts b/packages/services/service-analytics/src/preview-evaluator.ts index 89cc78eb510..4a8656492a5 100644 --- a/packages/services/service-analytics/src/preview-evaluator.ts +++ b/packages/services/service-analytics/src/preview-evaluator.ts @@ -35,7 +35,10 @@ import { resolveAnalyticsDateRangeString, utcInstantMs, compensatedSum, + temporalComparandKind, + temporalStorageForm, type BucketGranularity, + type TemporalComparandKind, } from '@objectstack/core'; import { explicitDateRangeWindow } from './date-range-array-arm.js'; // [#19810] The `where` door's refusal envelope — `INVALID_FILTER` / 400, @@ -655,6 +658,77 @@ export function declaredPreviewLowering(declaredType?: (field: string) => string }; } +/** + * [#21505, ADR-0053 D-A1] The draft preview compares a temporal value in its + * STORAGE form, on both sides, as `driver-memory` does. The preview has no + * driver, so its counterpart of the engine door is the rule that door + * applies, `@objectstack/core`'s `temporalStorageForm`, for the kind + * `temporalComparandKind` gives the column's declared type. A drafted row + * keeps whatever spelling its author wrote, so the row value is put in that + * form too, the way a driver puts it on write. + * + * Without it a drafted chart compared the two spellings as text. An instant + * against a bare day, or a window end written shorter than the stored instant, + * then counted rows the published chart does not, or missed rows it counts. + * + * A column the host names no type for stays as written on both sides. + */ +type PreviewTemporalKind = (field: string) => TemporalComparandKind | null; + +function previewTemporalKind(declaredType?: (field: string) => string | undefined): PreviewTemporalKind { + const kinds = new Map(); + return (field) => { + if (!kinds.has(field)) kinds.set(field, temporalComparandKind(declaredType?.(field))); + return kinds.get(field)!; + }; +} + +/** The value comparisons whose comparand takes the storage form: `driver-memory`'s arm set. */ +const STORAGE_FORM_OPERATORS = new Set(['$eq', '$ne', '$gt', '$gte', '$lt', '$lte', '$in', '$nin', '$between']); + +/** `value` in the storage form of `kind`; a list maps its members, as the drivers do. */ +function storageForm(value: unknown, kind: TemporalComparandKind): unknown { + return Array.isArray(value) ? value.map((v) => temporalStorageForm(v, kind)) : temporalStorageForm(value, kind); +} + +/** The condition with each value comparison's comparand on a temporal column in its storage form. */ +function previewStorageComparands(node: Row | undefined, kindOf: PreviewTemporalKind): Row | undefined { + if (!node) return node; + const out: Row = {}; + for (const [key, cond] of Object.entries(node)) { + const kind = key.startsWith('$') ? null : kindOf(key); + if ((key === '$and' || key === '$or') && Array.isArray(cond)) { + out[key] = cond.map((arm) => previewStorageComparands(arm as Row, kindOf)); + } else if (key === '$not' && cond !== null && typeof cond === 'object' && !Array.isArray(cond)) { + out[key] = previewStorageComparands(cond as Row, kindOf); + } else if (!kind || cond == null || Array.isArray(cond)) { + out[key] = cond; + } else if (typeof cond !== 'object' || cond instanceof Date) { + out[key] = temporalStorageForm(cond, kind); // implicit equality + } else { + out[key] = Object.fromEntries(Object.entries(cond as Row).map(([op, expected]) => [ + op, + STORAGE_FORM_OPERATORS.has(op) && expected != null ? storageForm(expected, kind) : expected, + ])); + } + } + return out; +} + +/** The row with every declared temporal field in its storage form; the same row when none moved. */ +function previewStorageRow(row: Row, kindOf: PreviewTemporalKind): Row { + let out: Row | undefined; + for (const [field, value] of Object.entries(row)) { + const kind = kindOf(field); + if (!kind) continue; + const stored = storageForm(value, kind); + if (stored === value) continue; + out ??= { ...row }; + out[field] = stored; + } + return out ?? row; +} + /** * Evaluate `query` over `rows` using the cube's measure/dimension specs. * Mirrors the engine strategies' output contract: rows keyed by bare @@ -703,9 +777,13 @@ export function evaluateAnalyticsQueryOverRows( // handed. Measured when the NULL guards arrived (step 3), they moved only the // rows this face read through `String()` — a row with no value against the // text `'null'` or `'undefined'` — onto every driver's answer. - const where = lowerFilterCondition(normalizeWhereComparands(query.where), lowering); - assertPreviewCanEvaluate(where); - let filtered = rows.filter((r) => matchesWhere(r, where)); + const lowered = lowerFilterCondition(normalizeWhereComparands(query.where), lowering); + assertPreviewCanEvaluate(lowered); + // [#21505] …then each value comparison in the temporal STORAGE form, the + // comparand here and the row value at the match (see {@link previewTemporalKind}). + const kindOf = previewTemporalKind(declaredType); + const where = previewStorageComparands(lowered, kindOf); + let filtered = rows.filter((r) => matchesWhere(previewStorageRow(r, kindOf), where)); const timeDims = query.timeDimensions ?? []; for (const td of timeDims) { const dim = cube.dimensions?.[td.dimension]; @@ -727,8 +805,8 @@ export function evaluateAnalyticsQueryOverRows( // 4, with a `'~'`-suffix reading of a full-timestamp end ("inclusive of // that instant's own sub-values") that no other face gives. const bounds = endExclusive ? { $gte: start, $lt: end } : { $gte: start, $lte: end }; - const window = lowerFilterCondition({ [field]: bounds }, lowering); - filtered = filtered.filter((r) => matchesWhere(r, window)); + const window = previewStorageComparands(lowerFilterCondition({ [field]: bounds }, lowering), kindOf); + filtered = filtered.filter((r) => matchesWhere(previewStorageRow(r, kindOf), window)); } // 2. Grouping keys: each selected dimension (time dims bucketed). diff --git a/packages/services/service-analytics/src/read-scope-sql.ts b/packages/services/service-analytics/src/read-scope-sql.ts index 012f458156f..fb1352e2305 100644 --- a/packages/services/service-analytics/src/read-scope-sql.ts +++ b/packages/services/service-analytics/src/read-scope-sql.ts @@ -642,6 +642,28 @@ import { * has no arm for: still `READ_SCOPE_COMPILE_FAILED` / 500, withheld, per the * #5367 section above. The note at {@link compileOperator}'s `default:` arm * records why a 400 would be the wrong class here. + * + * ## A temporal comparand binds in the column's storage form (#21505, ADR-0053 D-A1 / D-A2) + * + * D-A1 binds every surface that puts a filter comparand into raw SQL to the + * driver's dialect-aware temporal coercion. This compiler bound the comparand + * as written, so the database read it by its own rules instead of the + * engine's: PostgreSQL cast a bare day in the SESSION's zone, and SQLite + * compared it as text against the canonical instant. The read scope and the + * engine then admitted different rows for one policy, on some cells in the + * admitting direction. + * + * The caller now hands the driver's pair, bound to the object + * ({@link ReadScopeCompileOptions.coerceTemporalFilterValue} and + * {@link ReadScopeCompileOptions.coerceTemporalFilterColumn}): the engine + * door's own functions, never a second copy of the storage rule. Every value + * comparison in {@link compileOperator} and {@link compileField} binds its + * comparand through the first and reads its column through the second. The + * order is D-E3's by construction: the shared lowering at the entry widens a + * bare day first, and the arms convert the bound they are handed. The null + * tests, `$empty` and the text arms read the column as stored, as the native + * `where` face does. An absent member is identity, the contract's own reading + * for a driver whose storage form is the wire form. */ const IDENT = /^[a-z_][a-z0-9_]*$/i; @@ -746,6 +768,29 @@ export interface ReadScopeCompileOptions { * consumers fill it from the context's `declaredValueShape` hook. */ declaredValueShape?: (field: string) => ValueShapeFieldDef | undefined; + /** + * [#21505, ADR-0053 D-A1] The comparand half of the driver's temporal + * coercion, bound to the object this scope reads: `field`'s comparand in + * the form the column is STORED in. It is the engine door's own function + * (`IDataDriver.temporalFilterValue`, which `StrategyContext.coerceTemporalFilterValue` + * reaches), so a read scope and the engine compare one value. Applied after + * the shared lowering, to every comparand a value comparison binds + * (equality, `$ne`, the four orderings, `$in`, `$nin`, `$between`), never to + * a text pattern or a null test. Absent is identity: the comparand binds as + * written. Both of this compiler's consumers fill it from the context. + */ + coerceTemporalFilterValue?: (field: string, value: unknown) => unknown; + /** + * [#21505, ADR-0053 D-A2] The column half of the same coercion, and its + * required pair: given the column reference a value comparison was going + * to emit, the expression it must emit instead so the column reads in the + * form {@link ReadScopeCompileOptions.coerceTemporalFilterValue} put the + * comparand in (`IDataDriver.temporalFilterColumnSql`). It answers the + * reference unchanged for every column that needs no repair. The + * expression binds nothing: the consumers renumber every `?` in this + * compiler's output. Absent is identity: the column reads as written. + */ + coerceTemporalFilterColumn?: (field: string, columnSql: string) => string; } /** A node the compiler can walk: a plain object, not `null` and not an array. */ @@ -1310,11 +1355,11 @@ function compileField(field: string, value: unknown, qAlias: string, params: unk // "not a field reference". assertNoFieldReferenceComparand(field, value); - // Scalar / null → implicit equality. + // Scalar / null → implicit equality. [#21505] A value is compared in the + // column's storage form, like `$eq` below; the null test reads it as stored. if (value === null) return `${col} IS NULL`; if (typeof value !== 'object' || value instanceof Date) { - params.push(value); - return `${col} = ?`; + return `${comparisonColumn(col, field, opts)} = ${bindComparand(params, field, value, opts)}`; } // The implicit spelling of the equality slot {@link assertNoListInEqualitySlot} // guards under `$eq` — the shape a CEL `field == ` lowers to. @@ -1361,6 +1406,25 @@ function bind(params: unknown[], v: unknown): string { return '?'; } +/** + * [#21505] Bind a value comparison's comparand in the column's storage form, + * through the caller's {@link ReadScopeCompileOptions.coerceTemporalFilterValue} + * (identity when absent). See the module header's #21505 section. + */ +function bindComparand(params: unknown[], field: string, v: unknown, opts: ReadScopeCompileOptions): string { + return bind(params, opts.coerceTemporalFilterValue ? opts.coerceTemporalFilterValue(field, v) : v); +} + +/** + * [#21505] A value comparison's column, read in the form its comparand was + * coerced into, through {@link ReadScopeCompileOptions.coerceTemporalFilterColumn} + * (identity when absent). Null tests, `$empty` and the text arms read the + * column as stored, as the native `where` face does. + */ +function comparisonColumn(col: string, field: string, opts: ReadScopeCompileOptions): string { + return opts.coerceTemporalFilterColumn ? opts.coerceTemporalFilterColumn(field, col) : col; +} + /** * [#15684] Compile one case-EXACT text predicate for the dialect that will run * this scope — `text-match-sql.ts` picks the construct, this wrapper supplies @@ -2014,24 +2078,30 @@ function compileOperator( params: unknown[], opts: ReadScopeCompileOptions, ): string { + // [#21505] The value comparisons below compare in the column's STORAGE form: + // each comparand through {@link bindComparand}, the column through + // {@link comparisonColumn} — the driver's pair, identity when the caller + // passes none. The null tests and the text arms read the column as stored. + const vcol = (): string => comparisonColumn(col, field, opts); + const vbind = (v: unknown): string => bindComparand(params, field, v, opts); switch (op) { // [#19975] `val` is never a list here: {@link assertNoListInEqualitySlot} // refused one at {@link compileField}, before this emitter runs. - case '$eq': return val === null ? `${col} IS NULL` : `${col} = ${bind(params, val)}`; + case '$eq': return val === null ? `${col} IS NULL` : `${vcol()} = ${vbind(val)}`; // [#5298] `$ne: null` is `IS NOT NULL` — already total, and "has any // value" is false for a row that has none. A `$ne` of a value arrives // inside the NULL escape the shared lowering wrote around it (see the // module header), so the comparison compiles as written here. - case '$ne': return val === null ? `${col} IS NOT NULL` : `${col} <> ${bind(params, val)}`; - case '$gt': return `${col} > ${bind(params, val)}`; - case '$gte': return `${col} >= ${bind(params, val)}`; - case '$lt': return `${col} < ${bind(params, val)}`; - case '$lte': return `${col} <= ${bind(params, val)}`; + case '$ne': return val === null ? `${col} IS NOT NULL` : `${vcol()} <> ${vbind(val)}`; + case '$gt': return `${vcol()} > ${vbind(val)}`; + case '$gte': return `${vcol()} >= ${vbind(val)}`; + case '$lt': return `${vcol()} < ${vbind(val)}`; + case '$lte': return `${vcol()} <= ${vbind(val)}`; case '$in': { if (!Array.isArray(val)) throw readScopeCompileError(`[read-scope-sql] $in for "${field}" needs an array (fail-closed).`); if (val.length === 0) return FALSE_CLAUSE; // IN () matches nothing — safe assertCompilableMembers(op, field, val); - return `${col} IN (${val.map((v) => bind(params, v)).join(', ')})`; + return `${vcol()} IN (${val.map(vbind).join(', ')})`; } case '$nin': { if (!Array.isArray(val)) throw readScopeCompileError(`[read-scope-sql] $nin for "${field}" needs an array (fail-closed).`); @@ -2046,12 +2116,12 @@ function compileOperator( assertCompilableMembers(op, field, val); // [#5298] "Not among this list" holds vacuously for a value that is not // there: the shared lowering's NULL escape around this leaf says so. - return `${col} NOT IN (${val.map((v) => bind(params, v)).join(', ')})`; + return `${vcol()} NOT IN (${val.map(vbind).join(', ')})`; } case '$between': { if (!Array.isArray(val) || val.length !== 2) throw readScopeCompileError(`[read-scope-sql] $between for "${field}" needs [min,max] (fail-closed).`); assertCompilableMembers(op, field, val); - return `${col} BETWEEN ${bind(params, val[0])} AND ${bind(params, val[1])}`; + return `${vcol()} BETWEEN ${vbind(val[0])} AND ${vbind(val[1])}`; } // [#5567] The comparand is a LITERAL, so it is escaped and the escape // character is bound with it. See {@link textMatch}. diff --git a/packages/services/service-analytics/src/strategies/native-sql-strategy.ts b/packages/services/service-analytics/src/strategies/native-sql-strategy.ts index 64d8ec0dac1..0c4022fd163 100644 --- a/packages/services/service-analytics/src/strategies/native-sql-strategy.ts +++ b/packages/services/service-analytics/src/strategies/native-sql-strategy.ts @@ -1451,11 +1451,21 @@ export class NativeSQLStrategy implements AnalyticsStrategy { // text; one it cannot resolve is refused in the read-scope envelope. // [#20445] …and so does the declared value shape, so a policy's `$empty` // is answered by the field's row of the ruled per-type table. + // [#21505] …and so does the driver's temporal coercion pair (ADR-0053 + // D-A1 / D-A2), bound to the OBJECT (never the alias), so a temporal + // comparand binds in the column's storage form and the scope admits the + // rows the engine admits. const { sql, params: scopeParams } = compileScopedFilterToSql(filter, alias, { nonTextColumn: nonTextColumnResolver(ctx, objectName), dialect: sqlDialectFor(ctx, objectName), context: ctx.context, declaredValueShape: declaredValueShapeResolver(ctx, objectName), + coerceTemporalFilterValue: ctx.coerceTemporalFilterValue + ? (field, value) => ctx.coerceTemporalFilterValue!(objectName, field, value) + : undefined, + coerceTemporalFilterColumn: ctx.coerceTemporalFilterColumn + ? (field, columnSql) => ctx.coerceTemporalFilterColumn!(objectName, field, columnSql) + : undefined, }); // [#13926] The #13640 door guard, at THIS strategy's merge site. This is // not an echo: `execute()` runs this method's output through diff --git a/packages/services/service-analytics/src/strategies/objectql-strategy.ts b/packages/services/service-analytics/src/strategies/objectql-strategy.ts index 9839b161945..239144e623c 100644 --- a/packages/services/service-analytics/src/strategies/objectql-strategy.ts +++ b/packages/services/service-analytics/src/strategies/objectql-strategy.ts @@ -643,11 +643,19 @@ export class ObjectQLStrategy implements AnalyticsStrategy { // to, and refuses one it cannot resolve, as `execute()` does. // [#20445] …and the same declared value shape, so the echoed scope // prints the `$empty` arm the executed native statement runs. + // [#21505] …and the same temporal coercion pair, so the echo prints + // the storage-form comparand and column the executed statement binds. const { sql: scopeSql, params: scopeParams } = compileScopedFilterToSql(scope, tableName, { nonTextColumn: nonTextColumnResolver(ctx, tableName), dialect: sqlDialectFor(ctx, tableName), context: ctx.context, declaredValueShape: declaredValueShapeResolver(ctx, tableName), + coerceTemporalFilterValue: ctx.coerceTemporalFilterValue + ? (field, value) => ctx.coerceTemporalFilterValue!(tableName, field, value) + : undefined, + coerceTemporalFilterColumn: ctx.coerceTemporalFilterColumn + ? (field, columnSql) => ctx.coerceTemporalFilterColumn!(tableName, field, columnSql) + : undefined, }); // [#13926] The same door guard `execute()` trusts (`withReadScope`, // #13640), at the ECHO's own merge — so one read scope gets ONE verdict