Skip to content

Decode embedded language JSON into final owned storage - #1128

Merged
helly25 merged 7 commits into
mainfrom
perf/direct-language-json-storage
Nov 7, 2026
Merged

helly25 merged 7 commits into
mainfrom
perf/direct-language-json-storage

Conversation

@helly25

@helly25 helly25 commented Nov 7, 2026 •

Copy link
Copy Markdown
Collaborator

Decode the lazily loaded language database into its final string and move that owned result into static storage. This removes an intermediate buffer and copies while preserving decoding errors and view lifetime. Current command controls do not establish a material startup speedup.

AG;DR

Implementation

  • Replace the zero-initialized byte vector and string conversion with std::string::resize_and_overwrite, feeding final storage directly to the existing Brotli decoder. The narrowly annotated C pointer adapter remains within the decoder callback; publish only successfully written bytes, and publish no unwritten destination bytes after failure.
  • Preserve invalid-data and exact-size DataLoss errors. Add committed valid-empty and undersized-destination coverage alongside existing invalid-data and length tests. Use the mutable StatusOr<string> only to move its owned value into the existing lazy static string; returned views remain backed by that static owner.
  • No public API, CLI, environment, worker, VFS or initialization-trigger changes. Ordinary listing does not load this database. No permanent benchmark rule, new benchmark module dependency or profiler runtime is added.

Measured evidence and limits

  • docs/benchmarks/language-json-storage-20261107.md and its 154,700-byte .json.gz preserve 108 source/raw/log records; SHA-256 59d7645aea9397b4bba4443a903e5e3dfd295ca590ac2333a4f816288ab431d8.
  • Fresh current production-function images, byte-identical fixtures and source/config/image receipts complete three sessions with nine repetitions retaining seven and an identical-image repeat. Decode-plus-copy improves 6.05% Mac/2.87% Linux; decode-plus-move improves 5.09%/2.54%. Comparing the old decode/retained-copy path to the new decode/retained-move path improves 9.87%/3.90%; that includes both storage and ownership changes.
  • Complete whole-command output oracles pass. Type-list changes -0.24% Mac/+0.58% Linux, versus same-image repeats +1.26%/+0.36%. Unchanged listing controls cost +1.28%/+0.59%. These establish no material full-command improvement, and no RSS/peak-memory value was measured. All raw samples and slower controls remain available.
  • The payload is 109,261 bytes. Unstripped experimental full images are unchanged at 8,275,000 bytes on Mac and 320 bytes smaller on Linux; these are not release-download sizes. Mac is unpinned; Linux is pinned. No builds or sanitizer checks overlap timing on the measured host.
  • Preserve earlier modeled kernels and the separate zero-initialized-string variant without relabeling their timing as current uninitialized-buffer evidence. The historical 500-launch Mac summary is inconclusive and lacks individual samples/full source provenance; it remains supporting history rather than final qualification. Fresh production-function and command populations close that provenance gap.

Integration and publication

  • Based on Reuse validated rg output transactions in matcher results #1127's exact published 92469874d258200a1e07fcbabcc9c8b150602294, including its demonstrated one-blank-line Trunk correction. Merge Reuse validated rg output transactions in matcher results #1127 first, then synchronize/revalidate this child in its actual main context. Runtime module paths are independent of O22, but shared tracking/release publication overlap is preserved through this ancestry.
  • Add concise CHANGELOG/performance analysis, mark the narrow O25 storage cleanup integrated, and retain the broader immutable-catalog/first-parse investigation. Register report HTML and supported raw files download mapping. The gzip is below the large-file threshold, so no additional exception is introduced. No existing evidence/history is removed.

Validation

The private current-main implementation passed all 323 documented core/registered-extension release targets on each host, three focused canonical ASan targets on both, and the same three under Linux MSan. Linux real FUSE is required. Actual-source/image receipts and raw results are retained. A sandbox cache-write setup failure before Bazel analysis is preserved separately.

The successor integration on #1127's initial c4d head passed the three optimized language targets on Mac with verified identical module inputs. The subsequent ancestor correction removes only an unrelated blank line; all three module files and evidence bytes remain exact. This is development validation, not a substitute for this PR's own current-head GCC, coverage, sanitizers, CI clang-tidy, benchmark aggregate and required gates. No local clang-tidy is run, as requested by the user. Exact changed-file Trunk passed (eight supported text/source files; gzip content identity was checked independently), along with canonical Markdown table formatting, release-site configuration and diff checks. Normal signed commit 081d5be6627d004f2983837d1221399559e36910 and normal push succeeded with all configured commit/push hooks enabled and payload hashes unchanged. No local clang-tidy executable ran: the existing hook returned cleanly at its mandatory missing-compile-database guard before any Bazel/toolchain lookup. CI clang-tidy remains required. The user-requested skip was also supplied to the normal Git environment; no hook or protection was disabled.

CI clang-tidy identified the required char-storage to Brotli uint8_t byte-pointer cast. A narrow NOLINTNEXTLINE(cppcoreguidelines-pro-type-reinterpret-cast) now documents that single C-ABI conversion; decoding still writes directly into the final string, with no additional copy or allocation. Scoped formatting, changed-file hooks explicitly skipping local clang-tidy, and normal signed commit/push hooks passed. The parent’s three real CI lint corrections are incorporated through a signed synchronization that was pushed immediately before the ABI annotation commit. Actual CI clang-tidy and all current-head required checks must pass in this PR context; no archived or parent checks substitute for them. Original measurement evidence remains immutable.

Current main synchronization

#1127 merged unchanged as 1786340680499f50b08dee5b1ceffbc20fd7b533. After automatic retargeting, synchronized this PR to that actual main with signed merge 334cc891bbdfb2b297cc3d874c368466a31c6942, pushed immediately after normal hooks. The full resulting Git tree is identical to the prior qualified d89fa1d75b38a5303c498e2d66045e05c3492620 tree: three adjacent documentation conflicts preserved the child content after proving the squash tree exactly matched its already-included parent. No runtime or evidence bytes changed. GitHub initially rejected verified signatures transiently; after its API confirmed both commits verified, the unchanged normal push succeeded, without a rewrite or protection bypass. Fresh own main-context checks on this pushed head are required; the earlier stacked run's success is not substituted.

Base automatically changed from perf/o22-transactional-record-publication to main November 7, 2026 07:02
@helly25
helly25 merged commit 1278c89 into main Nov 7, 2026
32 checks passed
@helly25
helly25 deleted the perf/direct-language-json-storage branch November 7, 2026 08:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant