Skip to content

perf(metadata): bound locator compaction pressure - #608

Closed
forhappy wants to merge 1 commit into
codex/bounded-catalog-openingfrom
codex/locator-layer-pressure
Closed

forhappy wants to merge 1 commit into
codex/bounded-catalog-openingfrom
codex/locator-layer-pressure

Conversation

@forhappy

@forhappy forhappy commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Frequent locator checkpoint flushes can let sorted runs accumulate while L0 work occupies the single compaction slot. The 1,000-publication regression failed with 47 active layers against its existing bound of 40. Bound L0 and compaction fan-in together at 16, start maintenance at the corresponding pressure threshold, and align coordinator polling with the existing 500 ms worker/result-commit intervals.

This changes the shared metadata writer settings used by native publication and shared generation maintenance. Coverage-only updates and reader repair keep their existing compaction-free writer entry points. No public API, stored encoding, dependency or environment option changes.

Validation: 80 focused locator tests pass (the dedicated stress test remains ignored by the normal slice); strict all-target metadata Clippy, payload-only and remote-index checks pass. The frontier-only candidate passed the unchanged 1,000-publication regression with 20 active layers but took 661.65 seconds. The clean-source polling-aligned stress run passed in 147.72 seconds; the separate guarded release library build and formatting check also passed. The explicit stress run is separate from the source-guarded 80-test slice. No stress assertion or baseline is relaxed. This component qualification does not establish full repository opening or 10 GiB performance.

Evidence map: CLI push and shared generation maintenance call GitObjectLocatorWriter::open_for_publication; it reads persisted L0 pressure, then opens SlateDB with canonical locator settings. The locked SlateDB 0.15 scheduler prefers eligible L0 work and uses max_compaction_sources for sorted-run backpressure; its coordinator defaults to five seconds. Explicit writer close, lock renewal/release, checkpoint visibility and coverage proofs remain at their existing owners. The parent PR #603 retains a 32-source frontier and the default coordinator interval. This settings change is isolated from #603's visibility-read coalescing.

Is this the best fix? The pressure policy belongs in the shared locator owner, so both publication paths receive the same bound. It addresses the reproduced accumulation without raising resource concurrency, changing lookup authority or adding a second maintenance path. The extra coordinator polling can increase object-store maintenance traffic; hosted S3 latency/request qualification is still required.

Draft, stacked on #603. The advisor plan and both large-file audit reports are excluded. No merge requested.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant