Skip to content

refactor: organize the parser into subsystem modules (#380 part 2) - #389

Merged
oglassdev merged 10 commits into
mainfrom
cleanup/parser-layout
Sep 24, 2026
Merged

oglassdev merged 10 commits into
mainfrom
cleanup/parser-layout

Conversation

@oglassdev

@oglassdev oglassdev commented Sep 24, 2026 •

Copy link
Copy Markdown
Owner

Part 2 of #380. This restructures the jet3 parser crate without changing its behavior.

Layout

crates/jet3/src/
  lib.rs  database.rs  testkit.rs (cfg(test))
  format/        binary, offset, page, page kind, header (PAGE_BYTES), source, raw stream, page image, data page directory, limits, resource, error, commit state
  catalog/       catalog cursor, records, name keys, overflow
  definition/    table/column/index definitions (reader and writer side by side), definition header layout, long-value map, map location
  row/           directory, reader/writer layout, insert/update/delete page ops
  long_value/    reader, writer, mutation
  alloc/         allocation, usage maps, traversal, mutation maps and patches
  index/{key,tree}/  scalar/text/locale/binary keys, tree pages and rows, mutation
  properties/    property blob and column/table properties
  relationship/  catalog, keys, flags, mutation, cascade
  write/         atomic publish, page edits, insert/update/delete/field update, auto number
  schema/        existing-schema edit operations
  create/        database creation (composer/)
  validate/      read-only validation

Changes

  • Real directories. All 153 #[path] attributes are gone. Tests sit next to their module as *_tests.rs.
  • Private modules. Every module is now private. The public API is the root re-exports, guarded by #![warn(unnameable_types)], and about 29 writer internals are now pub(crate).
  • Removed:
    • the pub use creation as create alias
    • re-export chains
    • four stale #![allow(dead_code)] attributes and the dead code under them
    • three variants that were never constructed
  • Folded tiny forwarding modules into their homes:
    • long_value_mutation_map
    • row_offsets
    • long_index_key
    • update_index_key
    • index_counter
    • index_allocation
    • row_reuse_page, row_insert_eof, row_slot
    • schema_publish
    • auto_number_*
    • mutation_map_write
  • Shared constants.
    • PAGE_BYTES has one definition; it was repeated in 17 places.
    • The definition-header layout now has named offsets instead of raw header[21..23] and 43 + n*8 arithmetic.
    • The data-page offsets are shared.
  • Behavior-based names.
    • Numeric* → Scalar*, since those types also handle Text, Binary and GUID.
    • InitialLongIndex → InitialScalarIndex.
    • CandidateCheckError → ImageCheckError.
    • Test files are now named for what they test.
    • Error message text is unchanged.
  • Wrappers. The General-only wrappers were collapsed; callers now pass the sort order explicitly.
  • Doc comments. The 40–50 line docs on public write/create functions are down to a description, an # Errors section and a pointer to V1_SCOPE.md.
  • Test kit. An in-crate testkit provides budget() (35 copies before), TempDir (14 guards before) and database_image.
  • Paths updated. The evidence paths in the support matrix and the protocol branch registry point at the new files. The registry schema now allows subdirectories.

Left for the consolidation PR, on purpose:

  • the two reserve helpers and the push<T> helpers, which charge work units and return errors differently from each other
  • a single insert/update/delete driver
  • a shared data-page editor
  • shared catalog system-row writing
  • error consolidation

Removed from the public API (crate is 0.0.0)

  • The jet3::<module>:: paths. Everything is now reached through the root re-exports.
  • Writer internals that nothing outside the crate used:
    • atomic_update, atomic_create, atomic_create_with_hook
    • encode_row, RowColumnLayout
    • DataPageBuilder, PageImage, page_tag
    • encode_table_definition, TableDefinitionSpec
    • encode_catalog_record, CatalogRecordSpec, catalog_record_len
    • SHARED_COMMIT_SLOT_COUNT, DATABASE_HEADER_PAGE_NUMBER
  • CandidateCheckError, which was renamed to the crate-internal ImageCheckError.

Verification

  • Byte-identical outputs. The new scripts/golden-check.sh 9638d38 runs:

    • 58 example-generator runs
    • 30 jet3-cli create/schema/mutate requests
    • validate on every output

    All 1,923 files match the base byte for byte (SHA-256), including each command's stdout, stderr and exit code.

  • Test counts. cargo test -p jet3 gives 823 passed, 6 ignored, 17 integration and 2 doc tests, the same as main.

  • Windows. cargo check --target x86_64-pc-windows-gnu is clean.

  • just ready passes: 899 tests, 0 failures.

🤖 Generated with Claude Code

Replace the flat module list and #[path] attributes with real directories
(format, catalog, definition, row, long_value, alloc, index/{key,tree},
properties, relationship, write, schema, create, validate). Tests sit beside
their module. Add scripts/golden-check.sh to compare database outputs of two
revisions byte for byte.
All parser modules are now crate-private. Root re-exports used by the CLI,
test kit, examples, fuzz targets, benches or named by the public API stay
public (guarded by #![warn(unnameable_types)]); writer internals become
pub(crate). Remove the unused `create` alias and the re-export chains in
column_writer, definition::index and create. Test-only encoders and accessors
move under #[cfg(test)] or into their test files; never-constructed candidate
check variants are removed.
Define PAGE_BYTES once in format::header (replacing 17 local copies and
JET3_PAGE_BYTES), collect the table-definition header offsets and record
sizes in definition::header, share the data-page directory offsets from
format::data_page_directory, and move page_tag beside PageKind. Drop stale
dead_code allowances; test-only helpers move under #[cfg(test)] and unused
constants are deleted.
Merge long_value mutation_map into mutation, row offsets into reader_layout,
the Long key encoder into the scalar key module, update_key and counter into
index::mutation, index_allocation into index::mutation_load, reuse_page and
insert_eof into row::insert_page, slot into row::directory, schema publish
into schema::edit, the two AutoNumber modules into write::auto_number, and
mutation_map_write into alloc::mutation_map.
Rename the Numeric* index types to Scalar* (they also cover text, binary and
GUID keys), InitialLongIndex to InitialScalarIndex, the candidate checks to
image checks, and history-named test files by what they test. Collapse the
General-only wrappers (catalog name validation/equality, NameKey,
encode_table_definition, CreationProperties and the plan_table_schema chain)
so callers pass the sort order or encoding context explicitly.
State what each operation does and how it fails, with one pointer to
docs/plans/V1_SCOPE.md for scope, instead of listing experiment ids and
construction details that live beside the code encoding them.
Add #[cfg(test)] testkit with the default budget, a self-removing TempDir,
and the Jet 3 page-zero image builder, replacing 35 budget helpers, 14
hand-written temp-directory guards and 11 copies of the page-zero bytes.
@oglassdev
oglassdev merged commit 430a6cb into main Sep 24, 2026
10 checks passed
@oglassdev
oglassdev deleted the cleanup/parser-layout branch September 24, 2026 12:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant