diff --git a/.changeset/21485-date-bucket-calendar-day.md b/.changeset/21485-date-bucket-calendar-day.md new file mode 100644 index 00000000000..9091a4d45ff --- /dev/null +++ b/.changeset/21485-date-bucket-calendar-day.md @@ -0,0 +1,11 @@ +--- +'@objectstack/driver-sql': patch +--- + +A `Field.date` grouped by `day`, `week`, `month`, `quarter` or `year` buckets as its own calendar day on PostgreSQL and MySQL, whatever zone the server or the session is in (#21485). + +Clause-②: no + +- **What was wrong.** The PostgreSQL bucket cast every column to `timestamptz` and the MySQL bucket passed every column through `convert_tz`. A `date` has no instant, so both invented midnight in the session's zone, and on a session east of UTC the conversion to UTC read the previous day. On PostgreSQL with the server at `Asia/Shanghai`, `2026-06-01` grouped into month `2026-05`, and `2026-01-01` into year `2025`. MySQL did the same once the session zone was `+08:00`; the driver pins its own sessions to UTC, so there it took a host `pool.afterCreate` that sets the session zone. +- **What it does now.** A declared `Field.date` buckets its calendar day with no zone conversion. A `Field.datetime`, and a column with no declaration, keep the UTC-instant expression, byte for byte. SQLite already bucketed a `date` as its calendar day and is unchanged. +- **Where it shows.** `aggregate()` with a `dateGranularity` group, and the expression `SqlDriver.dateBucketSql()` renders for the analytics SQL echo, which reads the same expression. diff --git a/packages/drivers/driver-sql/src/sql-driver-21485-date-bucket-calendar-day.test.ts b/packages/drivers/driver-sql/src/sql-driver-21485-date-bucket-calendar-day.test.ts new file mode 100644 index 00000000000..604cb310d8a --- /dev/null +++ b/packages/drivers/driver-sql/src/sql-driver-21485-date-bucket-calendar-day.test.ts @@ -0,0 +1,272 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * [#21485] A `Field.date` buckets as its own calendar day, whatever zone the + * server or the session is in. A `Field.datetime` buckets as its UTC instant. + * + * ## The defect this pins closed + * + * `buildDateBucketExpr` used one expression per dialect for every column it + * bucketed, the instant one: `to_char((col)::timestamptz AT TIME ZONE 'UTC', …)` + * on PostgreSQL and `date_format(convert_tz(col, @@session.time_zone, '+00:00'), …)` + * on MySQL. A `date` has no instant, so the cast INVENTED one: midnight in the + * session's zone. On a session east of UTC that midnight is the previous UTC + * day, and the UTC conversion then read the day before. Measured on + * PostgreSQL 16.14 with the server's `TimeZone` at `Asia/Shanghai`, and on + * MySQL 8.0.46 with the session at `+08:00`, both before the fix: + * + * | `Field.date` value | day | week | month | quarter | year | + * |:--|:--|:--|:--|:--|:--| + * | `2026-06-01` | `2026-05-31` | `2026-W22` | `2026-05` | `2026-Q2` | `2026` | + * | `2026-01-01` | `2025-12-31` | `2026-W01` | `2025-12` | `2025-Q4` | `2025` | + * | `2024-12-30` | `2024-12-29` | `2024-W52` | `2024-12` | `2024-Q4` | `2024` | + * + * At a UTC session every one of those is the value's own calendar day, which + * is why the UTC cells stayed green: the zone decided the answer. + * + * ## The cells + * + * - **sqlite**, every run. A `date` is TEXT there and `strftime` reads it with + * no zone, so this cell is the control the live cells must agree with. + * - **live postgres** / **live mysql**, where `OS_TEST_POSTGRES_URL` / + * `OS_TEST_MYSQL_URL` are provisioned (the `Temporal Conformance (live PG + + * MySQL)` CI job, step "Run driver-sql suite against both live servers"), + * and a named skip (red under `OS_EXPECT_LIVE_DIALECT_MATRIX=1`) otherwise. + * Each runs twice: + * 1. **as provisioned** — the session the driver leaves in place. On + * PostgreSQL that is the server's `TimeZone` (`Asia/Shanghai` in CI), + * which is this card's reproduction. On MySQL the driver pins its own + * session to `+00:00` (#3942), so this run cannot see the defect there. + * Both axes are asserted, and the three-way zone skew is asserted first. + * 2. **session at +08:00** — a host `pool.afterCreate` sets the session + * zone, which the driver chains after its own hook. This is the run that + * is red-capable on any server, and the only one that is on MySQL. A + * test asserts that the session really is at `+08:00`, so a hook that + * stops taking effect is a red, not a quiet return to UTC. + * Only the `date` axis is asserted here. A host-set MySQL session also + * moves the `datetime` arm, whose `DATETIME(3)` holds the UTC wall + * clock the driver wrote; that arm is outside this card's ruling, which + * keeps it as it is, and it is recorded on the PR instead. + * + * Every cell checks both doors that render the expression: `aggregate()`'s + * GROUP BY, and `dateBucketSql()` — the text the analytics echo prints (#21441) + * — run as SQL against the same rows. + * + * ## Reverse verification + * + * Measured with all three cells provisioned (PostgreSQL 16.14 at + * `Asia/Shanghai`, MySQL 8.0.46 at a global `+08:00`, process + * `TZ=America/New_York`), this file beside `sql-driver-temporal-dialect.test.ts`: + * + * - fix in place: 61 of 61 green; + * - `sql-driver.ts` reverted to its pre-fix text: 16 red, 45 green. The red + * ones are the `date` axis at all five granularities on live postgres as + * provisioned, live postgres at `+08:00`, and live mysql at `+08:00` (15), + * plus the no-server pin in the dialect-gating file. Live mysql as + * provisioned stayed green, because the driver's UTC session pin hides the + * defect there. Every `datetime` cell and the sqlite cell stayed green. + */ + +import { afterAll, beforeAll, describe, expect, it } from 'vitest'; +import { SqlDriver, type SqlDriverConfig } from '../src/index.js'; +import { + DIALECT_CELLS, + assertThreeWayZoneSkew, + declareDialectCell, + readServerZone, + type DialectCell, + type DialectId, +} from './live-dialect-matrix.testkit.js'; + +const TABLE = 'os21485_bucket'; + +type Granularity = 'day' | 'week' | 'month' | 'quarter' | 'year'; +type Axis = 'on' | 'at'; + +const GRANULARITIES: readonly Granularity[] = ['day', 'week', 'month', 'quarter', 'year']; + +/** + * The granularities each dialect buckets in SQL. Declared rather than read off + * the driver, and checked against it below, so the set the tests iterate cannot + * shrink without a red. SQLite buckets `week` in memory (#21595 owns that). + */ +const BUCKETED_IN_SQL: Record = { + sqlite: ['day', 'month', 'quarter', 'year'], + pg: GRANULARITIES, + mysql: GRANULARITIES, +}; + +type Labels = Record; +const labels = (day: string, week: string, month: string, quarter: string, year: string): Labels => + ({ day, week, month, quarter, year }); + +/** + * Every row, and the label each value buckets under. + * + * `on` is a calendar day whose midnight at `+08:00` falls on the PREVIOUS UTC + * day, at a month, a quarter, a year and an ISO-week boundary. `at` is an + * instant whose `+08:00` wall clock is on the NEXT day, across the same + * boundaries, so a fix that bucketed a `datetime` by its local day would be + * one bucket off too. The last row is the empty bucket. + */ +const ROWS: ReadonlyArray<{ id: string; on: [string, Labels] | null; at: [string, Labels] | null }> = [ + { + id: 'r1', + on: ['2026-06-01', labels('2026-06-01', '2026-W23', '2026-06', '2026-Q2', '2026')], + at: ['2026-05-31T20:00:00.000Z', labels('2026-05-31', '2026-W22', '2026-05', '2026-Q2', '2026')], + }, + { + id: 'r2', + on: ['2026-04-01', labels('2026-04-01', '2026-W14', '2026-04', '2026-Q2', '2026')], + at: ['2026-03-31T18:30:00.000Z', labels('2026-03-31', '2026-W14', '2026-03', '2026-Q1', '2026')], + }, + { + id: 'r3', + on: ['2026-01-01', labels('2026-01-01', '2026-W01', '2026-01', '2026-Q1', '2026')], + at: ['2025-12-31T20:00:00.000Z', labels('2025-12-31', '2026-W01', '2025-12', '2025-Q4', '2025')], + }, + { + id: 'r4', + on: ['2024-12-30', labels('2024-12-30', '2025-W01', '2024-12', '2024-Q4', '2024')], + at: ['2024-12-29T18:00:00.000Z', labels('2024-12-29', '2024-W52', '2024-12', '2024-Q4', '2024')], + }, + { id: 'r5', on: null, at: null }, +]; + +/** The empty bucket, keyed out of band so it cannot collide with a real label. */ +const EMPTY = '(empty bucket)'; +const labelOf = (v: unknown): string => (v == null ? EMPTY : String(v)); + +/** label -> row count: what `field` bucketed by `g` must answer. */ +function expectedBuckets(axis: Axis, g: Granularity): [string, number][] { + const counts = new Map(); + for (const row of ROWS) { + const key = row[axis] ? row[axis]![1][g] : EMPTY; + counts.set(key, (counts.get(key) ?? 0) + 1); + } + return [...counts].sort(); +} + +function bucketsOf(rows: readonly Record[], labelColumn: string): [string, number][] { + return rows.map((r): [string, number] => [labelOf(r[labelColumn]), Number(r.n)]).sort(); +} + +/** Unwrap a raw result across knex's three dialect shapes. */ +function rowsOf(res: any): Record[] { + if (Array.isArray(res) && Array.isArray(res[0])) return res[0]; // mysql2: [rows, fields] + if (Array.isArray(res)) return res; // better-sqlite3 + return res?.rows ?? []; // pg +} + +/** The statement that sets a session to `+08:00`, per live dialect. */ +const SET_SESSION_ZONE: Partial> = { + pg: `SET TIME ZONE 'Asia/Shanghai'`, + mysql: `SET time_zone = '+08:00'`, +}; + +/** The session's own zone, read back through the driver. */ +const READ_SESSION_ZONE: Partial> = { + pg: `select current_setting('TimeZone') as tz`, + mysql: `select @@session.time_zone as tz`, +}; + +interface SessionRun { + label: string; + /** Statement a host `pool.afterCreate` runs on each new connection, if any. */ + hostSessionSql?: string; + /** The axes asserted in this run. */ + axes: readonly Axis[]; +} + +function sessionRuns(cell: DialectCell): SessionRun[] { + if (!cell.live) return [{ label: 'in process', axes: ['on', 'at'] }]; + return [ + { label: 'session as provisioned', axes: ['on', 'at'] }, + { label: 'session at +08:00 (host pool.afterCreate)', hostSessionSql: SET_SESSION_ZONE[cell.id], axes: ['on'] }, + ]; +} + +function configFor(cell: DialectCell, run: SessionRun): SqlDriverConfig { + const base = cell.config(); + const sql = run.hostSessionSql; + if (!sql) return base; + return { + ...base, + pool: { + afterCreate(connection: any, done: (err?: unknown, conn?: unknown) => void) { + connection.query(sql, (err: unknown) => done(err, connection)); + }, + }, + }; +} + +function measure(cell: DialectCell): void { + for (const run of sessionRuns(cell)) { + describe(`[#21485] date buckets on ${cell.label}, ${run.label}`, () => { + let driver: SqlDriver; + + beforeAll(async () => { + driver = new SqlDriver(configFor(cell, run)); + await driver.execute(`drop table if exists ${TABLE}`).catch(() => {}); + await driver.initObjects([{ name: TABLE, fields: { on: { type: 'date' }, at: { type: 'datetime' } } }]); + for (const row of ROWS) { + await driver.create( + TABLE, + { id: row.id, on: row.on ? row.on[0] : null, at: row.at ? row.at[0] : null }, + { bypassTenantAudit: true }, + ); + } + }); + + afterAll(async () => { + await driver?.execute(`drop table if exists ${TABLE}`).catch(() => {}); + await driver?.disconnect(); + }); + + if (cell.live && !run.hostSessionSql) { + it('runs where the server, the process and UTC are three different zones', async () => { + assertThreeWayZoneSkew(cell, await readServerZone(cell, driver)); + }); + } + + if (run.hostSessionSql) { + it('the session really is at +08:00, so this run can see a zone leak', async () => { + const rows = rowsOf(await driver.execute(READ_SESSION_ZONE[cell.id]!)); + expect(['Asia/Shanghai', '+08:00']).toContain(String(rows[0]?.tz)); + }); + } + + it('buckets in SQL exactly the granularities this file iterates', () => { + const advertised = Object.entries(driver.supports.queryDateGranularity as Record) + .filter(([, on]) => on) + .map(([g]) => g) + .sort(); + expect(advertised).toEqual([...BUCKETED_IN_SQL[cell.id]].sort()); + }); + + for (const axis of run.axes) { + const subject = axis === 'on' ? 'a Field.date buckets as its calendar day' : 'a Field.datetime buckets as its UTC instant'; + it.each(BUCKETED_IN_SQL[cell.id])(`${subject}, by %s: aggregate() and the dateBucketSql echo`, async (g) => { + const want = expectedBuckets(axis, g); + + const grouped = await driver.aggregate(TABLE, { + groupBy: [{ field: axis, dateGranularity: g }], + aggregations: [{ function: 'count', alias: 'n' }], + }); + expect(bucketsOf(grouped as Record[], axis), 'aggregate()').toEqual(want); + + const expr = driver.dateBucketSql(TABLE, axis, g); + expect(expr, 'dateBucketSql renders an expression').toBeTypeOf('string'); + const echoed = rowsOf( + await driver.execute(`select ${expr} as b, count(*) as n from ${TABLE} group by ${expr}`), + ); + expect(bucketsOf(echoed, 'b'), 'the dateBucketSql echo, run').toEqual(want); + }); + } + }); + } +} + +for (const cell of DIALECT_CELLS) { + declareDialectCell(cell, 'date-bucket calendar day', measure); +} diff --git a/packages/drivers/driver-sql/src/sql-driver-temporal-dialect.test.ts b/packages/drivers/driver-sql/src/sql-driver-temporal-dialect.test.ts index ec7a941d6a3..4fdfff96856 100644 --- a/packages/drivers/driver-sql/src/sql-driver-temporal-dialect.test.ts +++ b/packages/drivers/driver-sql/src/sql-driver-temporal-dialect.test.ts @@ -215,6 +215,35 @@ describe('buildDateBucketExpr dialect gating (#3773)', () => { } }); + /** + * [#21485] The no-server half of the live pins in + * `sql-driver-21485-date-bucket-calendar-day.test.ts`, so a regression is red + * in every job, including the ones with no PostgreSQL or MySQL attached. A + * `Field.date` is a calendar day: its bucket names no zone and converts none. + * Each spelling refused below read the session's zone, and on a session east + * of UTC moved the day into the previous bucket. + * + * On PostgreSQL the day must reach `to_char` as a `timestamp`. A bare `date` + * there resolves `to_char(timestamptz, text)` through the implicit cast, so a + * `timestamptz` is back without the text naming it. + */ + it('Postgres and MySQL bucket a declared Field.date with no zone conversion (#21485)', () => { + for (const client of ['pg', 'mysql2']) { + const d = makeDriver(client); + d.seedDate('t', 'on'); + for (const g of [...GRANULARITIES, 'week']) { + const sql = expr(d, 'on', g, 't')!.sql.toLowerCase(); + for (const zoned of ['timestamptz', 'time zone', 'convert_tz', 'time_zone']) { + expect(sql, `${client} ${g}`).not.toContain(zoned); + } + if (client === 'pg') expect(sql, `${client} ${g}`).toContain('::date::timestamp,'); + // The control: the same column with no declaration keeps the UTC-instant arm. + expect(expr(d, 'on', g)!.sql.toLowerCase(), `${client} ${g}, undeclared`) + .toContain(client === 'pg' ? `at time zone 'utc'` : 'convert_tz('); + } + } + }); + it('every emitted expression binds exactly as many identifiers as it references', () => { // The quarter expression references the column twice; an epoch-normalised // one references it six times. A mismatch here is knex silently shifting diff --git a/packages/drivers/driver-sql/src/sql-driver.ts b/packages/drivers/driver-sql/src/sql-driver.ts index 164cb6b66b3..be60fd4699d 100644 --- a/packages/drivers/driver-sql/src/sql-driver.ts +++ b/packages/drivers/driver-sql/src/sql-driver.ts @@ -6121,6 +6121,28 @@ export class SqlDriver implements IDataDriver { * {@link coercionKey} returns for the builder). It is what makes the SQLite * expression storage-aware — see {@link sqliteTemporalArg}. Omitting it yields * the plain column form, which is correct for any TEXT-stored column. + * + * ## A `Field.date` buckets as its calendar day (#21485) + * + * `table` also decides which value the PostgreSQL and MySQL arms bucket. A + * declared `Field.date` ({@link temporalFieldKind} `'date'`) is a calendar + * day with no instant (ADR-0053), so it is bucketed as that day and never + * converted between zones. Everything else, a `Field.datetime` or an + * undeclared column, keeps the UTC-instant conversion, byte for byte. + * + * The instant form cannot carry a `date`. Casting one to `timestamptz`, or + * handing it to `convert_tz`, invents an instant at midnight in the + * SESSION's zone, and the UTC conversion then reads the previous day on any + * session east of UTC. Measured on PostgreSQL 16.14 at `Asia/Shanghai` and on + * MySQL 8.0.46 with the session at `+08:00`: `2026-06-01` bucketed into month + * `2026-05`, and `2026-01-01` into year `2025`. + * + * On PostgreSQL the day goes to `to_char` as a `timestamp` (no zone), not as + * a bare `date`. A bare `date` resolves `to_char(timestamptz, text)` through + * the implicit cast, which round-trips the session zone. That round trip is + * the identity on every day except one the zone skipped: measured over + * 1900..2100, `Pacific/Apia` printed `2011-12-30` as `2011-12-31` and + * `Pacific/Kiritimati` printed `1994-12-31` as `1995-01-01`. */ protected buildDateBucketExpr( field: string, @@ -6129,23 +6151,27 @@ export class SqlDriver implements IDataDriver { ): { sql: string; bindings: any[] } | null { if (!this.dateGranularityCapabilities[granularity]) return null; + const calendarDay = this.temporalFieldKind(table, field) === 'date'; + if (this.isPostgres) { + const arg = calendarDay ? `(??)::date::timestamp` : `(??)::timestamptz AT TIME ZONE 'UTC'`; switch (granularity) { - case 'year': return { sql: `to_char((??)::timestamptz AT TIME ZONE 'UTC', 'YYYY')`, bindings: [field] }; - case 'month': return { sql: `to_char((??)::timestamptz AT TIME ZONE 'UTC', 'YYYY-MM')`, bindings: [field] }; - case 'day': return { sql: `to_char((??)::timestamptz AT TIME ZONE 'UTC', 'YYYY-MM-DD')`, bindings: [field] }; - case 'quarter': return { sql: `to_char((??)::timestamptz AT TIME ZONE 'UTC', 'YYYY"-Q"Q')`, bindings: [field] }; - case 'week': return { sql: `to_char((??)::timestamptz AT TIME ZONE 'UTC', 'IYYY"-W"IW')`, bindings: [field] }; + case 'year': return { sql: `to_char(${arg}, 'YYYY')`, bindings: [field] }; + case 'month': return { sql: `to_char(${arg}, 'YYYY-MM')`, bindings: [field] }; + case 'day': return { sql: `to_char(${arg}, 'YYYY-MM-DD')`, bindings: [field] }; + case 'quarter': return { sql: `to_char(${arg}, 'YYYY"-Q"Q')`, bindings: [field] }; + case 'week': return { sql: `to_char(${arg}, 'IYYY"-W"IW')`, bindings: [field] }; } } if (this.isMysql) { + const arg = calendarDay ? `??` : `convert_tz(??, @@session.time_zone, '+00:00')`; switch (granularity) { - case 'year': return { sql: `date_format(convert_tz(??, @@session.time_zone, '+00:00'), '%Y')`, bindings: [field] }; - case 'month': return { sql: `date_format(convert_tz(??, @@session.time_zone, '+00:00'), '%Y-%m')`, bindings: [field] }; - case 'day': return { sql: `date_format(convert_tz(??, @@session.time_zone, '+00:00'), '%Y-%m-%d')`, bindings: [field] }; - case 'quarter': return { sql: `concat(date_format(convert_tz(??, @@session.time_zone, '+00:00'), '%Y'), '-Q', quarter(convert_tz(??, @@session.time_zone, '+00:00')))`, bindings: [field, field] }; - case 'week': return { sql: `date_format(convert_tz(??, @@session.time_zone, '+00:00'), '%x-W%v')`, bindings: [field] }; + case 'year': return { sql: `date_format(${arg}, '%Y')`, bindings: [field] }; + case 'month': return { sql: `date_format(${arg}, '%Y-%m')`, bindings: [field] }; + case 'day': return { sql: `date_format(${arg}, '%Y-%m-%d')`, bindings: [field] }; + case 'quarter': return { sql: `concat(date_format(${arg}, '%Y'), '-Q', quarter(${arg}))`, bindings: [field, field] }; + case 'week': return { sql: `date_format(${arg}, '%x-W%v')`, bindings: [field] }; } } @@ -6186,7 +6212,8 @@ export class SqlDriver implements IDataDriver { * quoting. The expression binds identifiers only, so the text carries no * value placeholder. `objectName` goes in as the coercion key, as * `aggregate` passes it, so a SQLite `Field.datetime` column that may still - * hold pre-canonical values renders the repair the GROUP BY runs (#3773). + * hold pre-canonical values renders the repair the GROUP BY runs (#3773), + * and a `Field.date` renders its calendar day on every dialect (#21485). * * Read structurally by its caller, like {@link dialectName}. It is not a * member of the `IDataDriver` contract.