Repository navigation
feat: add projection-aware reads and inferred synchronous mapping - #233
Conversation
There was a problem hiding this comment.
Let's remove this file but instead update README.md file. Also we should not have before and after because the reader doesn't know how it was before, he must know only current state of the affairs.
There was a problem hiding this comment.
Addressed in 6f7fc689. Removed READ-SCHEMAS.md and moved its consumer examples, decoding rules, polymorphic dispatch, and limitations into README.md under “Projection-aware reads,” after the prerequisites. Removed the before/after narrative; the guide describes the current API, including separate query/mapping/service files. A regression test now compiles those actual multi-file README examples against the built package declarations.
| ## Opt-in projection-aware reads | ||
|
|
||
| Call `.withRowSchema()` before selecting or including fields to get an immutable, | ||
| detached PostgreSQL read query. Its `rowSchema` describes the actual decoded | ||
| projection, including SQL nulls, nested dates and exact decimal/bigint strings. | ||
|
|
||
| ```ts | ||
| const read = query(knex, UserSchema).withRowSchema() | ||
| .select(u => ({ id: u.id, createdAt: u.createdAt })); | ||
| const SourceSchema = read.rowSchema; // metadata only; no SQL | ||
| const rows = await read.where(u => u.id, 42); | ||
| ``` | ||
|
|
||
| Supports typed flat joins/aggregates, declared and explicit nested relations, | ||
| named projections, composite cursors and STI/CTI branch schemas. Existing query | ||
| behavior is unchanged. See the [consumer guide](./READ-SCHEMAS.md) for multi-file | ||
| mapping examples, representation rules, supported operations and migration limits. | ||
|
|
There was a problem hiding this comment.
this part doesn't seem to be correctly placed, why this is in the top of the file?
There was a problem hiding this comment.
Addressed in 6f7fc689. The README now introduces installation, quick start, schema definition, CRUD, filtering, relations, scopes, and projections before composable and projection-aware reads. I also moved the corresponding website sections after the basic examples and before the API reference.
| "COMPOSABLE_QUERIES.md", | ||
| "READ-SCHEMAS.md" |
There was a problem hiding this comment.
we should not have this, instead place these files to readme with correct order and good readability for first-time user
There was a problem hiding this comment.
Addressed in 6f7fc689. Both COMPOSABLE_QUERIES.md and READ-SCHEMAS.md have been removed, with their content consolidated into the README in prerequisite order. The package files list contains only dist again; npm includes the README automatically. Links from ORM, mapper, and the website now point to the relevant README anchors.
| declare module '@cleverbrush/schema' { | ||
| interface NumberSchemaBuilder< | ||
| TResult, | ||
| TRequired extends boolean, | ||
| TNullable extends boolean, | ||
| THasDefault extends boolean, | ||
| TExtensions | ||
| > { | ||
| /** Set a SQL storage type; schema-aware reads retain its representation. */ | ||
| columnType<const SQL extends string>( | ||
| type: SQL | ||
| ): SqlNumber< | ||
| TResult, | ||
| TRequired, | ||
| TNullable, | ||
| THasDefault, | ||
| TExtensions, | ||
| SQL | ||
| >; | ||
| /** Set bigint storage; schema-aware reads return exact strings. */ | ||
| bigint(): SqlNumber< | ||
| TResult, | ||
| TRequired, | ||
| TNullable, | ||
| THasDefault, | ||
| TExtensions, | ||
| 'bigint' | ||
| >; | ||
| /** Set smallint storage; schema-aware reads return numbers. */ | ||
| smallint(): SqlNumber< | ||
| TResult, | ||
| TRequired, | ||
| TNullable, | ||
| THasDefault, | ||
| TExtensions, | ||
| 'smallint' | ||
| >; | ||
| /** Set decimal storage; schema-aware reads return exact strings. */ | ||
| decimal( | ||
| precision: number, | ||
| scale: number | ||
| ): SqlNumber< | ||
| TResult, | ||
| TRequired, | ||
| TNullable, | ||
| THasDefault, | ||
| TExtensions, | ||
| 'decimal' | ||
| >; | ||
| } | ||
| } |
There was a problem hiding this comment.
why do we need it? is it something that we can't do with native @cleverbrush/schema?
There was a problem hiding this comment.
Native schema extensions already support storing this metadata at runtime. The missing piece was retaining the exact metadata type through fluent calls, so read types can distinguish bigint/decimal strings from ordinary numbers. The earlier module augmentation was a workaround for that typing gap, not a necessary database mechanism.
Addressed in 6f7fc689 with application-agnostic defineMetadataMethod() and InferExtensionMetadata in @cleverbrush/schema. Extension authors can record an argument, fixed value, or computed value through the native immutable extension mechanism. knex-schema uses that for columnType/bigint/smallint/decimal; all SQL interpretation stays in knex-schema. Schema itself has no knex or SQL dependency. Tests cover all supported factory kinds, native modifiers, ordinary extensions, metadata replacement, and strict cross-file declaration consumers.
| for (const [name, method] of Object.entries(methods)) | ||
| if (typeof prototype[name] !== 'function') | ||
| Object.defineProperty(prototype, name, { | ||
| value: method, | ||
| configurable: true, | ||
| writable: true | ||
| }); |
There was a problem hiding this comment.
this looks like a hack, why do we need it?
There was a problem hiding this comment.
The patch was compensating for the type-inference gap described in the other thread; it was not needed for runtime storage. I agree it was the wrong boundary. Removed read-storage.ts, its numeric module augmentation, and its prototype patch in 6f7fc689. The four storage methods are now normal factory-local native extensions. Runtime and compile-time regressions verify that importing knex-schema does not add bigint/decimal/columnType/smallint to plain schema numbers. The separate pre-existing primary-key implementation is unchanged in scope.
| ## Synchronous mapping | ||
|
|
||
| Use the same `configure()` API and retrieve a synchronous mapper when every final | ||
| step and nested mapping is synchronous: | ||
|
|
||
| ```ts | ||
| const registry = mapper().configure(Source, Target, m => m | ||
| .for(t => t.label).compute(s => s.name.toUpperCase())); | ||
| const toTarget = registry.getSyncMapper(Source, Target); | ||
| const results = rows.map(toTarget); // ordinary values, not promises | ||
| ``` | ||
|
|
||
| Completeness checking, `.from()`, ignores, nested objects/arrays and compatible | ||
| auto-mapping still apply. Async computations or nested async mappings reject | ||
| `getSyncMapper()` at compile time; runtime guards cover JavaScript and unsafe | ||
| casts too. Callbacks are never probed during configuration. A function falsely | ||
| typed as synchronous that returns a promise/thenable throws when invoked. | ||
| `getMapper()` still always returns an async function. There is no separate | ||
| `configureSync()` API. | ||
|
|
||
| Configure once and reuse the same source/target schema instances. Queries can | ||
| provide their projection schema through `.withRowSchema().rowSchema`; see the | ||
| [projection-aware read guide](../knex-schema/READ-SCHEMAS.md), including separate | ||
| definition/mapping/service files and explicit polymorphic dispatch. The mapper | ||
| performs no database calls or application enrichment. | ||
|
|
There was a problem hiding this comment.
again, why it's at the top of the file? how new user is supposed to read README.md then? if he even doesn't know what this library for.
There was a problem hiding this comment.
Addressed in 6f7fc689. Synchronous mapping now follows the introduction, installation, quick start, compile-time safety, auto-mapping, and mapping strategies. Added getSyncMapper to the API reference and updated the introductory workflow to mention both async and synchronous retrieval. The documentation websites follow the same basics-before-advanced order.
| ## Detached reads with result schemas | ||
|
|
||
| `db.users.withRowSchema()` enters an immutable read-only API whose `rowSchema` | ||
| matches its decoded selection and includes. Results remain **detached even when | ||
| the context uses `{ tracking: true }`**. This avoids attaching partial projections | ||
| as incomplete tracked entities. Ordinary entity queries keep their old behavior. | ||
|
|
||
| ```ts | ||
| const read = db.users.withRowSchema() | ||
| .select(u => ({ id: u.id, name: u.name })); | ||
| const Source = read.rowSchema; | ||
| const toDto = mapper().configure(Source, UserDto, m => m) | ||
| .getSyncMapper(Source, UserDto); | ||
| const users = (await read).map(toDto); | ||
| ``` | ||
|
|
||
| Typed relation customizers return their configured query; nested graphs are | ||
| decoded in one SQL statement. STI/CTI readers expose `variantRowSchemas` for | ||
| explicit application mapping. See the [read-schema consumer guide](../knex-schema/READ-SCHEMAS.md) | ||
| for numeric/null/date rules, examples and compatibility boundaries. | ||
|
|
There was a problem hiding this comment.
again, why at the top of the file?
There was a problem hiding this comment.
Addressed in 6f7fc689. Detached result-schema reads now follow setup, DbSet basics, relations, change tracking, and row versioning, so the reader has the context to understand “detached.” Added withRowSchema to the DbSet API table and updated links to the consolidated knex-schema README.
| ## Projection-aware reads and synchronous mapping | ||
|
|
||
| `query(knex, Schema).withRowSchema()` and `db.entities.withRowSchema()` opt into | ||
| immutable detached PostgreSQL reads. `rowSchema` matches the selected decoded | ||
| shape, including nested relations, SQL nulls, Date values and exact decimal/bigint | ||
| strings. Metadata access never executes SQL. STI/CTI readers expose | ||
| `variantRowSchemas` for explicit application discriminator dispatch. Legacy reads | ||
| are unchanged. See [/knex-schema#row-schemas](/knex-schema#row-schemas) and | ||
| [/orm#detached-read-schemas](/orm#detached-read-schemas). | ||
|
|
||
| `mapper().configure(Source, Target, configure).getSyncMapper(Source, Target)` | ||
| returns a synchronous complete mapping when all final steps and nested mappings | ||
| are synchronous. Async or disguised thenables are rejected; callbacks are not | ||
| probed at configuration time. `getMapper()` remains asynchronous. Reuse schemas | ||
| and prepared mappings; do not fetch data inside pure mapping. See | ||
| [/mapper#synchronous-mapping](/mapper#synchronous-mapping). | ||
|
|
There was a problem hiding this comment.
why at the top? look at this file and make it actual and to have logically correct sequence.
There was a problem hiding this comment.
Addressed in 6f7fc689. Reorganized llms.txt as an overview/package map, a complete multi-file schema → contract → handlers → registration → client workflow, then schema authoring, HTTP services, database/mapping, forms/jobs, and reference links. Updated stale APIs and removed incomplete examples; projection-aware reads and synchronous mapping now live in their relevant data section. The actual four-file workflow and native metadata README example are compiled in a strict built-package consumer regression test, including declaration emission.
1 participant
Original request
Implement point 1, “Projection-aware and synchronous mapping,” from the consumer-experience proposal in Framework, as a separate PR against
development, with JSDoc, documentation, README updates, tests, and a minor changeset. Xpenser adoption follows publication and is not part of this PR.What changed
withRowSchema()readers for ordinary queries, aliased joins, and ORM entity sets. Immutable, detached query plans expose the selected runtimerowSchemawithout executing SQL.variantRowSchemasfor application-owned mapper dispatch.getSyncMapper()with eligibility inferred through the existingconfigure()chain, including nested arrays, final overrides, completeness checks, and runtime thenable guards. No second configuration API;getMapper()remains async.defineMetadataMethod()andInferExtensionMetadatato native schema extensions. Literal, constant, and computed metadata retain their types through immutable chains; numeric SQL classification stays in knex-schema. Removed the new global numeric-storage augmentation/prototype patch entirely (the pre-existing primary-key implementation is unchanged).@cleverbrush/mapper,@cleverbrush/knex-schema,@cleverbrush/orm,@cleverbrush/schema, and@cleverbrush/di. No package publishing, Xpenser changes, or proposal-file edits.Reasoning and boundaries
The opt-in surface keeps legacy query mutability, driver representations, and entity tracking unchanged. One shared field description drives SQL projection, decoding, and metadata; source schemas no longer need to be duplicated in application mappers. Pure mappings can be prepared once and reused without per-row promises, while fetching, authorization, enrichment, DTO conversions, and union dispatch remain explicit application concerns.
Native extensions already stored runtime metadata; the missing capability was retaining its literal type through fluent modifiers. Generic metadata methods close that gap without adding database concepts to schema or patching global numeric builders. Documentation teaches the package basics before advanced reading/mapping and describes the current API rather than PR history.
The read API targets PostgreSQL. Enter it before legacy shape changes, ordering/pagination, or raw callbacks. Unsupported opaque shapes fail explicitly. Timezone-less SQL dates/timestamps use UTC in this mode; JavaScript dates retain millisecond precision. Optional schemas with input defaults are rejected because their inferred requirement can differ from persisted nullability; use separate input and storage schemas. Exact numeric hints within already-stored JSON are likewise rejected: represent those JSON values as strings.
Blog post
Skipped: this is Framework library infrastructure, not an Xpenser product feature. Consumer-facing package and website documentation is included instead.
Screenshots / preview evidence
Manually verified locally with agent-browser: docs
/mapper,/knex-schema,/orm,/llms.txt, and schema-site/mapper; the relocated guide sections follow introductory material, code examples display, and no browser errors were reported. Screenshots below show the current advanced sections. This repository does not configure a PR application deployment, so there is no hosted preview URL.Validation
npm run lintnpm run build— 21 tasksnpm run test— 4,276 unit and compile-time tests across 201 passing test files; no type errorsnpx tsc --noEmit --incremental false -p libs/knex-schema/tsconfig.typecheck.json— clean strict consumer check against built package declarationsnpm run test:queries:integration— 37 PostgreSQL tests, including the real read-schema-to-mapper workflow and transaction safety guardsnpm run typecheck:schema-siteandnpm run typecheck:docs-sitenpm run docs— generated API documentation, no errors; 288 non-fatal reference warnings remainnpm run bench -- --project benchmarks mapper.bench.ts— initial implementation benchmark: prepared synchronous mapping was 2.01× / 2.53× / 2.70× faster for flat / computed / nested fixtures. These are mapping-only results, not endpoint-latency claims. Mapper runtime is unchanged by this review update.6f7fc689— Lint, Build & Test (Node 24) and PostgreSQL Query Integration.