Skip to content

Commit 9f39094

Browse files
feat(core): import reads dotted and compact year-first days and matches option values in any case (#25596)
<!-- claude-projects-attribution: {"github_login":"os-sales"} --> _Requested by **Sales** · [project thread](https://claude.ai/code/project/chan_01D5bWFY3rFzgJcytD5gD4Uu?thread=cmsg_01D5bWFY3rFzgJcytD5gD4UuDWJsBGQpadCH8JR2xZQ5re)_ Part of #25582 Clause-②: yes (widening: two date-cell spellings and an option value in another case, each refused before with `invalid_date` / `invalid_option`, are now read; nothing read before is refused or stored differently) Metadata-protocol: none ## What Ruling 6377844879 (on #25493, letter A), proposals **1** and **2**, in `packages/core/src/utils/import-coerce.ts`: 1. **`parseDateCell` reads `YYYY.MM.DD` and `YYYYMMDD`.** A four-digit year, a two-digit month and day, nothing else in the cell. `2011.07.01` is stored as `2011-07-01`, `20090615` as `2009-06-15`. The day must exist (`namesRealCalendarDay`, the same arithmetic as the other shapes): `20090230` and `2009.02.30` stay `invalid_date`. No other form is widened: `2011.7.1`, `2011.07.01 9:00`, `11.07.01`, `01.07.2011`, `2011.07-01`, `200961`, `2009615`, `200906150` and `20090615 9:00` stay refused. - **Field kinds.** Read for a `date` and a `datetime` (midnight UTC, the reading a bare `2026/7/15` already gets on a `datetime`), never for a `time`: both spellings name a day and no clock. The `time` branch keeps its year-first-only fallback unchanged. - **A number is not a compact day.** The compact form is read only from a text cell, so a JSON number `20090615` stays refused. The existing reader treats numbers as never temporal (`outsideYears` docblock: an Excel serial, epoch ms). An xlsx cell holding that number reaches the reader as its text (`xlsxCellToString` in `import-prepare.ts`), so it is read. That path is read from the code, not measured. - An eight-digit cell on a `number` or `text` column is unaffected (`20090615` → the number `20090615` / the text `'20090615'`). This is pinned. 2. **Option values match case-insensitively.** `matchOption` now has three passes: exact value, then label case-insensitively (as before), then value case-insensitively. It stores the declared value, never the cell's spelling: `CN` stores `cn`. The value pass runs last, so a cell that matched before stores what it stored (pinned: a cell that is one option's label and, by case, another option's value keeps the label match). Single select, `multiple: true` select and multiselect parts all go through this one function. - **Ambiguity, measured.** The spec cannot declare two values that differ only by case: `FieldSchema.safeParse` on `[{value:'cn'},{value:'CN'}]` fails at `options.1.value` ("System identifier must be lowercase…"). A programmatic `engine.registry.registerObject` accepts and keeps the pair unparsed. Measured with a throwaway test against `@objectstack/objectql`, then deleted. So the pass refuses to guess. A cell naming two such values by case alone (`Cn`) gets the no-match answer it got before (`invalid_option`, or kept under `createMissingOptions`). An exact `cn` / `CN` still matches exactly. The same value declared twice counts as one value. #25582 remains open: proposal 4 (every cell error of a row) is not in this PR. It needs an `errors` member on `ImportRowResultSchema` first (the `domain:spec` seat's; the note is on the card), and `import-runner.ts` still keeps `errors[0]`. Proposal 3 (`aliases`) was not ruled in. ## Tests - `packages/core/src/utils/import-coerce.test.ts`: a `[#25582]` block covering the new forms on `date` and `datetime`, the invalid-calendar controls, the not-widened controls, `time` refusal, number refusal, the 8-digit number/text control, and the row-level `invalid_date`. `matchOption` gets value-case match storing the canonical value, label-ahead-of-value, and the ambiguous pair. `coerceRow` gets select, multiselect and `multiple: true` select parts stored canonical, with `keptOptionValues` empty and an unmatched part named. - `packages/rest/src/import-year-first-day-option-case.test.ts` (new): through the real `POST /api/v1/data/:object/import` over `SqlDriver` (better-sqlite3), a CSV row `2011.07.01,20090615,CN` imports and stores `2011-07-01` / `2009-06-15` / `cn`. The JSON dry run of the same row is clean and persists nothing. `20090230` and `DE` are still that row's `invalid_date` / `invalid_option`. **Ablation, once per proposal.** Each leg went through `scripts/ablation-replace.mjs` (anchor hit 1 → 0, blob moved). Then `@objectstack/core` was rebuilt, because the rest test reads core's `dist/`, and `ablation-dist-preflight.mjs` saw the marker in `dist/index.js` and `dist/index.cjs`. - **P1** (the `readYearFirstDay` call removed): core 9 failed / 167 passed of 176. Route 3 failed of 3 (`ok: 0, errors: 1` on the ledger row). - **P2** (the value pass returns nothing): core 3 failed / 173 passed. Route 2 failed / 1 passed (the refusal case stays green, as expected). - **Restore.** Blob == HEAD (`60af90a93041`), `git diff HEAD` empty. Core rebuilt, and `--absent` passed for both markers on all 14 built files with a clean tree. ## Gates All readings below were taken at head `c630eacca0` (after merging `origin/main`), unless a line names another commit. - `node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack` derived 70 commands. All 70 ran and exited 0. `--ran` gives: `✓ dispatch-gates --ran: 70 derived famil(ies) accounted for — 70 run, 0 NOT-MEASURED`. - `pnpm lint`, through the verify lock: `VERDICT command-exit 0`. - `pnpm --filter @objectstack/core test`: 118 files / 2757 tests passed. `pnpm --filter @objectstack/core typecheck` and `pnpm --filter @objectstack/rest typecheck` passed (each package's test-layer `check:test-typecheck: OK`). Both through the lock, `VERDICT command-exit 0`. - `pnpm --filter @objectstack/rest exec vitest run` over the new file plus `import-date-cell-iso-real-day`, `import-dryrun-parity` and `import-integration`: 4 files / 119 tests passed. - After the merge: the `@objectstack/rest` closure was rebuilt, and `pnpm --filter @objectstack/spec check:generated` gives `✓ All 14 generated artifacts are up to date`. - NOT MEASURED locally: the whole `@objectstack/rest` suite (only a test file is added there; CI `Test Core` runs it) and `test:repo` (no repo-project test is touched). ## Acceptance notes - The xlsx template's instructions sheet (`TEMPLATE_READER_CLAIMS` in `packages/rest/src/import-template.ts`) lists example date spellings and does not mention the two new ones. Every claim it makes is still true (none of its refused examples is now read). Adding them is outside this slice's file surface. carrier: none. - `2011.7.1` (dotted, one-digit month/day) stays refused, per the ruling's 「不放宽其它形式」. Widening it would be a new decision. --- _Generated by [Claude Code](https://claude.ai/code/session_01CxBKycaiUpA1yGx1XyoGqX)_ 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01CxBKycaiUpA1yGx1XyoGqX --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent a57643d commit 9f39094

4 files changed

Lines changed: 319 additions & 7 deletions

File tree

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
---
2+
'@objectstack/core': patch
3+
---
4+
5+
fix(core): `POST /api/v1/data/:object/import` reads a dotted (`2011.07.01`) or compact (`20090615`) year-first day in a `date` or `datetime` cell, and matches a select / multiselect cell against the option values case-insensitively
6+
7+
Clause-②: yes (widening: two date-cell spellings and an option value in another case, each refused before with `invalid_date` / `invalid_option`, are now read; nothing read before is refused or stored differently)
8+
Metadata-protocol: none
9+
10+
Two cell shapes a person reads without hesitation used to fail the whole row. Both are now read by the import's cell reader (`parseDateCell`, `matchOption`), on the commit and on the dry run:
11+
12+
- **A year-first day, dotted or compact.** `YYYY.MM.DD` (`2011.07.01`, the dotted date of Chinese ledgers) and `YYYYMMDD` (`20090615`, the compact date of ERP exports) are stored as `2011-07-01` / `2009-06-15` in a `date` cell, and as midnight UTC in a `datetime` cell, as `2026/7/15` already is. The day must exist: `20090230` and `2009.02.30` stay `invalid_date`, never rolled over. No other form is widened: a one-digit month or day (`2011.7.1`), a clock after either spelling, a two-digit year, a day-first dotted date (`01.07.2011`), six or seven digits, a `time` cell and a JSON number `20090615` stay refused. An eight-digit cell on a `number` or `text` column is read exactly as before.
13+
- **An option value in another case.** A select / multiselect cell `CN` matches the declared option value `cn` and stores `cn` — the declared value, never the cell's spelling — as labels already matched in any case. A cell that matched an exact value or a label before stores what it stored: the value pass runs after both. A cell naming, by case alone, two declared values that differ only by case (`cn` and `CN`, which the spec's lowercase option value refuses but a programmatically registered object can carry) is not resolved by guessing; it gets `invalid_option` as before.
14+
15+
Every cell error of a row is still reported as the first one only; that half of the same request is not in this change.

‎packages/core/src/utils/import-coerce.test.ts‎

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -212,6 +212,64 @@ describe('[#20534] parseDateCell — ISO 8601, the export shape or a year-first
212212
});
213213
});
214214

215+
/**
216+
* [#25582, maintainer ruling 6377844879 letter A] Two more year-first spellings
217+
* of a bare day: `YYYY.MM.DD` and `YYYYMMDD`, on a day that exists, for a
218+
* `date` and a `datetime` — and no other form is widened.
219+
*/
220+
describe('[#25582] parseDateCell — the dotted and compact year-first day', () => {
221+
it.each([
222+
['2011.07.01', 'date', '2011-07-01'],
223+
['20090615', 'date', '2009-06-15'],
224+
[' 2011.07.01 ', 'date', '2011-07-01'],
225+
['2028.02.29', 'date', '2028-02-29'],
226+
['20280229', 'date', '2028-02-29'],
227+
['0500.01.01', 'date', '0500-01-01'],
228+
// A `datetime` reads the bare day as midnight UTC, as it reads `2026/7/15`.
229+
['2011.07.01', 'datetime', '2011-07-01T00:00:00.000Z'],
230+
['20090615', 'datetime', '2009-06-15T00:00:00.000Z'],
231+
] as const)('reads %j as a %s cell as %j', (cell, kind, stored) => {
232+
expect(parseDateCell(cell, kind)).toBe(stored);
233+
});
234+
235+
it.each([
236+
// A day that does not exist — refused, never rolled over.
237+
['20090230', 'date'], ['2009.02.30', 'date'], ['20260229', 'datetime'],
238+
['20091301', 'date'], ['20090100', 'date'], ['2009.13.01', 'datetime'],
239+
// No other form is widened.
240+
['2011.7.1', 'date'], ['2011.07.01 9:00', 'datetime'], ['2011.07.01T09:00', 'datetime'],
241+
['11.07.01', 'date'], ['01.07.2011', 'date'], ['2011.07-01', 'date'],
242+
['200961', 'date'], ['2009615', 'date'], ['200906150', 'date'], ['20090615 9:00', 'datetime'],
243+
// A `time` holds no day, so neither spelling is a `time` cell.
244+
['2011.07.01', 'time'], ['20090615', 'time'],
245+
] as const)('refuses %j as a %s cell', (cell, kind) => {
246+
expect(parseDateCell(cell, kind)).toBeUndefined();
247+
});
248+
249+
it('reads the compact day only from a text cell, never from a number', () => {
250+
expect(parseDateCell(20090615, 'date')).toBeUndefined();
251+
expect(parseDateCell(20090615, 'datetime')).toBeUndefined();
252+
});
253+
254+
it('leaves an eight-digit cell on a number or text field as it was', async () => {
255+
const metaMap = new Map<string, ExportFieldMeta>([
256+
['n', { name: 'n', type: 'number' } as ExportFieldMeta],
257+
['code', { name: 'code', type: 'text' } as ExportFieldMeta],
258+
['d', { name: 'd', type: 'date' } as ExportFieldMeta],
259+
]);
260+
const { data, errors } = await coerceRow({ n: '20090615', code: '20090615', d: '20090615' }, metaMap, {});
261+
expect(errors).toEqual([]);
262+
expect(data).toEqual({ n: 20090615, code: '20090615', d: '2009-06-15' });
263+
});
264+
265+
it('fails the row with invalid_date for a compact day that does not exist', async () => {
266+
const metaMap = new Map<string, ExportFieldMeta>([['d', { name: 'd', type: 'date' } as ExportFieldMeta]]);
267+
const { data, errors } = await coerceRow({ d: '20090230' }, metaMap, {});
268+
expect(data).toEqual({});
269+
expect(errors.map((e) => [e.field, e.code])).toEqual([['d', 'invalid_date']]);
270+
});
271+
});
272+
215273
describe('matchOption', () => {
216274
const options = [{ label: '高', value: 'high' }, { label: '低', value: 'low' }];
217275
it('matches by option value (code)', () => {
@@ -224,6 +282,31 @@ describe('matchOption', () => {
224282
it('returns undefined when nothing matches', () => {
225283
expect(matchOption('medium', options)).toBeUndefined();
226284
});
285+
286+
// [#25582, maintainer ruling 6377844879 letter A] Values match case-insensitively too.
287+
it('[#25582] matches by option value case-insensitively and stores the declared value', () => {
288+
const countries = [{ label: '中国', value: 'cn' }, { label: '美国', value: 'us' }];
289+
expect(matchOption('CN', countries)).toBe('cn');
290+
expect(matchOption(' Us ', countries)).toBe('us');
291+
expect(matchOption('HIGH', options)).toBe('high');
292+
// Not a substring or prefix match.
293+
expect(matchOption('CHN', countries)).toBeUndefined();
294+
});
295+
it('[#25582] keeps a label match ahead of the case-insensitive value match', () => {
296+
// `CN` is this option's label and, by case alone, the other option's value:
297+
// the label match it had before the value pass existed still wins.
298+
expect(matchOption('CN', [{ label: 'Other', value: 'cn' }, { label: 'cn', value: 'china' }])).toBe('china');
299+
});
300+
it('[#25582] refuses, rather than guesses, a cell naming two values that differ only by case', () => {
301+
const pair = [{ label: 'Lower', value: 'cn' }, { label: 'Upper', value: 'CN' }];
302+
// An exact value still matches exactly.
303+
expect(matchOption('cn', pair)).toBe('cn');
304+
expect(matchOption('CN', pair)).toBe('CN');
305+
// `Cn` names both by case alone: no match, as before this pass existed.
306+
expect(matchOption('Cn', pair)).toBeUndefined();
307+
// The same value declared twice is one value, not two.
308+
expect(matchOption('CN', [{ label: 'a', value: 'cn' }, { label: 'b', value: 'cn' }])).toBe('cn');
309+
});
227310
});
228311

229312
describe('splitMulti', () => {
@@ -401,6 +484,28 @@ describe('coerceRow', () => {
401484
expect(data).toEqual({});
402485
});
403486

487+
it('[#25582] matches select and multiselect option values case-insensitively, storing the declared value', async () => {
488+
const metaMap = meta({
489+
country: { type: 'select', options: [{ label: '中国', value: 'cn' }, { label: '美国', value: 'us' }] },
490+
markets: { type: 'multiselect', options: [{ label: '中国', value: 'cn' }, { label: '美国', value: 'us' }, { label: '日本', value: 'jp' }] },
491+
regions: { type: 'select', multiple: true, options: [{ label: '东部', value: 'east' }, { label: '西部', value: 'west' }] },
492+
});
493+
const { data, errors, keptOptionValues } = await coerceRow(
494+
{ country: 'CN', markets: 'CN; Us、日本', regions: 'EAST,west' },
495+
metaMap,
496+
{},
497+
);
498+
expect(errors).toEqual([]);
499+
expect(data).toEqual({ country: 'cn', markets: ['cn', 'us', 'jp'], regions: ['east', 'west'] });
500+
// Matched, not kept: the write carries no admission for them.
501+
expect(keptOptionValues).toEqual({});
502+
503+
// A part that matches no value or label still fails, naming that part.
504+
const bad = await coerceRow({ markets: 'CN; DE' }, metaMap, {});
505+
expect(bad.errors.map((e) => [e.field, e.code])).toEqual([['markets', 'invalid_option']]);
506+
expect(bad.errors[0].message).toContain('DE');
507+
});
508+
404509
it('honours createMissingOptions by keeping the raw select value', async () => {
405510
const metaMap = meta({ s: { type: 'select', options: [{ label: 'A', value: 'a' }] } });
406511
const strict = await coerceRow({ s: 'zzz' }, metaMap, {});

‎packages/core/src/utils/import-coerce.ts‎

Lines changed: 69 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -335,8 +335,9 @@ function readTimeOfDayCell(s: string): string | undefined {
335335
* stored as `2026-07-15T14:00Z` on a New York host and `…T02:00Z` on a
336336
* Shanghai one, a `date` a day apart) and reads `07/08/2026` month-first. A
337337
* space before a zone (`2026-07-15 10:00Z`) is refused as the write door
338-
* refuses it. The one other text shape read at all is the year-first date,
339-
* {@link YEAR_FIRST_CELL}.
338+
* refuses it. The other text shapes read at all are the year-first date,
339+
* {@link YEAR_FIRST_CELL}, and the year-first day spellings
340+
* {@link YEAR_FIRST_DOTTED_DAY} and {@link YEAR_FIRST_COMPACT_DAY}.
340341
*/
341342
const ISO_TEMPORAL_CELL =
342343
/^(\d{4})-(\d{2})-(\d{2})(?:(T| )(\d{2}):(\d{2})(?::(\d{2})(?:\.(\d+))?)?(Z|[+-]\d{2}:?\d{2})?)?$/;
@@ -450,6 +451,44 @@ function readYearFirstCell(s: string): IsoTemporalCell | undefined {
450451
return { day, wall };
451452
}
452453

454+
/**
455+
* [#25582, maintainer ruling 6377844879 letter A] Two more year-first spellings
456+
* of a bare calendar day, after trimming, each with a four-digit year and a
457+
* two-digit month and day and nothing else in the cell:
458+
*
459+
* - `YYYY.MM.DD` (`2011.07.01`) — the dotted date written in Chinese
460+
* correspondence and ledgers;
461+
* - `YYYYMMDD` (`20090615`) — the compact date an ERP export writes. Read only
462+
* from a TEXT cell: a number is never a temporal cell ({@link outsideYears}),
463+
* so a JSON `20090615` stays refused. An xlsx cell holding that number reaches
464+
* here as the text it shows (`import-prepare.ts`), so it is read.
465+
*
466+
* The year comes first and every part has a fixed width, so there is no field
467+
* order to guess. The ruling widens these two forms and no other: a one-digit
468+
* month or day (`2011.7.1`), a clock after either (`2011.07.01 9:00`), a
469+
* two-digit year (`11.07.01`), six or seven digits (`200961`) and a day-first
470+
* dotted date (`01.07.2011`) stay refused. The day must exist
471+
* ({@link namesRealCalendarDay}): `20090230` is refused, never rolled over.
472+
* Read for a `date` and a `datetime` (midnight UTC, as a bare `2026/7/15` is
473+
* read there), never for a `time`, which holds no day.
474+
*/
475+
const YEAR_FIRST_DOTTED_DAY = /^(\d{4})\.(\d{2})\.(\d{2})$/;
476+
/** See {@link YEAR_FIRST_DOTTED_DAY}. */
477+
const YEAR_FIRST_COMPACT_DAY = /^(\d{4})(\d{2})(\d{2})$/;
478+
479+
/**
480+
* Read a trimmed cell as a {@link YEAR_FIRST_DOTTED_DAY} or (a text cell only)
481+
* {@link YEAR_FIRST_COMPACT_DAY} day, or `undefined`. The day is the ISO
482+
* `YYYY-MM-DD` spelled from the cell's own digits.
483+
*/
484+
function readYearFirstDay(s: string, fromText: boolean): IsoTemporalCell | undefined {
485+
const m = YEAR_FIRST_DOTTED_DAY.exec(s) ?? (fromText ? YEAR_FIRST_COMPACT_DAY.exec(s) : null);
486+
if (!m) return undefined;
487+
const [, y, mo, d] = m;
488+
if (!namesRealCalendarDay(Number(y), Number(mo), Number(d))) return undefined;
489+
return { day: `${y}-${mo}-${d}` };
490+
}
491+
453492
/** The `HH:MM:SS` UTC clock of an instant. */
454493
function utcClock(t: Date): string {
455494
return `${pad2(t.getUTCHours())}:${pad2(t.getUTCMinutes())}:${pad2(t.getUTCSeconds())}`;
@@ -472,7 +511,9 @@ function utcClock(t: Date): string {
472511
* A text cell is read only in an {@link ISO_TEMPORAL_CELL} shape — ISO 8601,
473512
* or the export's own `YYYY-MM-DD HH:mm:ss` — or as a year-first date
474513
* ({@link YEAR_FIRST_CELL}: `2026/7/15`, `2026/7/15 9:00`, stored padded, the
475-
* clock read exactly as the export shape's), and only on a calendar day that
514+
* clock read exactly as the export shape's), or — a `date` / `datetime` only —
515+
* as a dotted or compact year-first day (`2011.07.01`, `20090615`;
516+
* {@link YEAR_FIRST_DOTTED_DAY}, #25582), and only on a calendar day that
476517
* exists. A `time` cell is read by core's one `time` rule instead
477518
* ({@link readTimeOfDayCell}, #20722): a bare `HH:MM[:SS[.f…]]` or an ISO
478519
* 8601 instant, as the write door reads it, and otherwise in the year-first
@@ -561,7 +602,10 @@ export function parseDateCell(
561602
// [#20722] Core's rule is the whole verdict on an ISO 8601 `time` cell, so a
562603
// `time` cell it refuses is read in the year-first form alone — the one
563604
// reading the import has beyond the write door (#20534, maintainer ruling).
564-
const cell = (kind === 'time' ? undefined : readIsoTemporalCell(s)) ?? readYearFirstCell(s);
605+
// [#25582] The dotted and compact year-first days name a day and no clock,
606+
// so a `time` cell never reads them.
607+
const cell = (kind === 'time' ? undefined : readIsoTemporalCell(s) ?? readYearFirstDay(s, typeof raw === 'string'))
608+
?? readYearFirstCell(s);
565609
if (!cell) return undefined;
566610

567611
if (cell.wall) {
@@ -608,8 +652,19 @@ function outsideYears(raw: unknown, kind: 'date' | 'datetime'): { firstYear: str
608652

609653
/**
610654
* Match a cell against a field's options, accepting **either** the option value
611-
* (code) or its human label (case-insensitive). Returns the canonical option
612-
* value to store, or `undefined` on no match.
655+
* (code) or its human label, each case-insensitively. Returns the canonical
656+
* option value to store — the declared value, never the cell's spelling — or
657+
* `undefined` on no match.
658+
*
659+
* Three passes, in order: the exact value, then the label case-insensitively,
660+
* then [#25582, maintainer ruling 6377844879 letter A] the value
661+
* case-insensitively (`CN` stores `cn`). The value pass comes last so that a
662+
* cell matched before it existed stores what it stored. A cell that names, by
663+
* case alone, two declared values that differ only by case (`cn` and `CN`) is
664+
* not resolved by that pass — picking one would be a guess — so it gets the
665+
* no-match answer it got before. The spec cannot declare such a pair (an option
666+
* value is a lowercase `SystemIdentifierSchema`); a programmatically registered
667+
* object can.
613668
*/
614669
export function matchOption(
615670
raw: unknown,
@@ -626,7 +681,14 @@ export function matchOption(
626681
for (const o of options) {
627682
if (o && typeof o.label === 'string' && o.label.trim().toLowerCase() === lower) return o.value;
628683
}
629-
return undefined;
684+
// [#25582] Case-insensitive value match, unless it names two declared values.
685+
let hit: { value: unknown } | undefined;
686+
for (const o of options) {
687+
if (!o || o.value === undefined || String(o.value).toLowerCase() !== lower) continue;
688+
if (hit && String(hit.value) !== String(o.value)) return undefined;
689+
hit ??= { value: o.value };
690+
}
691+
return hit?.value;
630692
}
631693

632694
/** Split a multi-value cell on commas / semicolons / Chinese comma / newlines. */

0 commit comments

Comments
 (0)