perf: reduce repeated knowledge bundle validation with reproducible retrieval evidence #190

Problem and accepted implementation scope

The requester explicitly authorized a deeper knowledge-base audit, local performance/quality measurements, an optimization PR and an architecture diagram in the active Codex conversation on 2026-10-27. This records implementation and publication delegation; merge and goal closure remain with the requester/maintainers.

Baseline: 4817f92121df73fd9cd5fb4a319e3c56a68b4a5b (current main at start). Read-only profiling found 108 catalog records and a 342-file, 2,294,958-byte shared skill library. A relevance collect builds 60 complete manifests and hashes the same library for dependency validation and resource inventories. These are diagnostic observations, not an accepted speedup.

Bounded hypothesis and alternatives

Remove redundant work while preserving complete validation and byte-equivalent retrieval outputs. Start with per-manifest resource digest reuse and already-validated path reuse. Any additional optimization must preserve mutation detection and exact revisions; do not add a global cache over mutable Bundles. Changing ranking, recall semantics, content, schemas, default enablement, or introducing a vector database/service is outside this slice.

Related #128 / merged #129 already delivered optional selection/context and a bounded real-agent comparison. #128 retained 34 task PASS and 2 INCONCLUSIVE; it did not accept cost-saving/preserved-quality efficacy. This work does not reopen that campaign or overwrite its results.

Plan and prospective acceptance

1. Freeze a constructed, independently authored retrieval set (development and withheld cases), workload/protocol, validators and hashes before candidate measurement. State that queries are not private production logs. Report Success@5/MRR@5 for labelled answerable cases, no-answer behavior separately, and per-query results; do not claim complete Recall or agent efficacy.
2. Measure baseline and candidate with the same corpus, interpreter, arguments and output budgets. Keep knowledge search, default collect, relevance collect and context delivery distinct. Measure fresh-reader and reused-reader calls plus full CLI invocations. Repeat paired AB/BA groups; retain raw timings, outputs/digests, sample counts and environment. Distinguish process-cold from OS filesystem-cache-cold (the latter is not claimed).
3. Quality gate: all frozen cases preserve serialized results, result ordering, omissions, revisions, resource hashes and budgets exactly; no decrease in measured retrieval metrics. Preserve stale-index, content mutation, dependency mismatch, unsafe-path, checkout-update and revision rejection tests.
4. Performance gate for this CPU-only retrieval campaign: at least 20% lower paired median hot relevance-collect latency, with a 95% bootstrap interval supporting a reduction; report cold/API/CLI costs and any regression separately. No engine/model/service performance claim. Numeric gates stay in this issue/evidence rather than reusable defaults.
5. Deliver a focused PR, English source-grounded audit, reproducible measurement packet, and editable English/Chinese knowledge architecture diagrams matching the supplied visual style. Independent code/evidence/diagram review and relevant local/package tests precede readiness.

Ownership and deliverables

Accountable orchestrator: OpenAI Codex parent. Material repository authors will use explicitly configured approved model/effort and be disclosed in the PR; independent reviewers are separate from implementers. The user-requested scope permits implementation without another design approval. Preserve existing evidence and all negative findings.

Evidence: a new campaign packet with plan/results/patch/validator, plus a readable report. Existing #128 evidence is cited as historical owner-reported evidence, not rerun or newly independently verified here. No new paid model/GPU calls are required for retrieval correctness and CPU performance measurements.

Completion

• Frozen plan, inputs and validators
• Baseline and candidate measurements, including withheld queries
• Minimal runtime change and mutation/integrity regression tests
• Independent review and package validation
• Architecture diagrams and audit report
• Linked PR with actual checks and limitations

Use non-closing Refs in the PR. Issue acceptance and closure remain with the requester/repository maintainers.

Frozen pre-measurement identities (2026-10-27)

• Baseline: 4817f92121df73fd9cd5fb4a319e3c56a68b4a5b.
• Fixed runtime candidate: 407bb244651bec1c68f4b3e94b0c697b233c4a09; Python-source inventory SHA-256 d6edbaf975cf3bc00902d6aa7cf2bb955dc58b3e37c74f59bb226c5f0d2fa261.
• Evaluation freeze v2 content SHA-256: a465367319f3058398eb1f8579ee5d4174b8cb90457793e90788fb5796151057.
• Independently authored 24 synthetic cases: 16 development (14 answerable, 2 negative), 8 withheld (6 answerable, 2 negative). Candidate source was fixed before withheld release. Partial source-grounded labels support Success@5/MRR@5, not complete Recall or production-distribution claims.
• Formal workload: 20 balanced AB/BA paired groups, each side containing 16 API and 4 CLI measurements; quality compares every one of 24 × 4 mode outputs per revision. No trimming or result-driven query selection.
• Independent review identified missing-coverage and optimized-Python assertion gaps before any candidate retrieval measurements. The validator was amended with explicit coverage checks and exceptions; the v1 manifest and amendment rationale are retained. No query/label/runtime candidate change. Formal performance/quality results are still pending; no speedup or quality improvement is claimed here.

Delivery and measured results

PR: #193

The bounded manifest change, source-backed audit, reproducible CPU retrieval
packet and editable English/Chinese architecture diagrams are prepared in
the accompanying optimization PR. The runtime reuses digest/path work within each manifest call and
preserves mutation/revision checks; the tested runtime source identity is fixed
in the packet.

The prospectively frozen 24-case comparison preserved all 96 outputs per side
exactly. Success@5/MRR@5 stayed 0.20/0.20 for search/default and 1.00/0.9625 for
the existing #129 relevance/context-item path. Synthetic source labels are
partial AI judgements, not human gold. Negative probes and the post-hoc 19/20
correct-owner context delivery diagnostic remain explicit; one complete
mandatory context exceeds the fixed budget.

Prospective paired CPU result: 40.01% hot relevance paired-median latency
reduction, 95% bootstrap interval [38.38%, 41.06%], from 20 balanced paired
groups, 640 API calls and 160 CLI processes. All 400 output pairs and 80 raw
CLI stdout pairs matched. Per-mode tables retain search/decode regressions,
including CLI relevance decode p95 1.810→3.046 ms. No engine/service,
agent-task or production-distribution benefit is claimed. Whole local pytest
was not green; all initial failures/errors reproduced on baseline. Manifest
checks and isolated package construction/content checks passed, with qualified
test and environment records retained. Read-only inventory checks reported
0 errors/9 bundles/120 hashes; 23 inventory tests and Ruff passed. Independent
read-only review reproduced quality/body-delivery counts and exact benchmark
statistics and checked diagrams. Reviewer model/effort is unknown. Remote CI was pending at PR creation and has since passed: Python 3.10/3.11 full-check and installed-package jobs all succeeded in run 40500855160. The knowledge-model-policy status also passed on head 2cc1d00cb8e7c2a99e277f4dc88e8f1441315ab7.

This is a progress/evidence submission for maintainer review. Goal acceptance,
merge and Issue completion/closure remain with the requester/repository
maintainers. Proposed for maintainer review; human/maintainer acceptance has
not been recorded.

Current delivery and cleanup scope

PR #193 merged as ff93302e26acc193fc7ec6a7513daf69e0721837 with its original implementation, tests and artifacts. The requester subsequently narrowed the retained repository content to working code and tests. PR #195 therefore removes the evidence packet, audit/plan documents and their inventory support from the post-merge tree, while preserving the merged runtime optimization and all manifest regressions. Raw evidence, reports and diagrams remain local. This supersedes earlier packaging statements; already-merged history is not rewritten.

Cleanup validation: 37 manifest/inventory tests passed; Ruff passed; existing evidence inventory reports 0 errors and 112 hashes. Cleanup ready/review CI 40503902024 passed. Original performance measurements remain specific to their recorded revisions and corpus; the cleanup introduces no new performance claim. The visual performance summary is posted in #193's first author explanation comment and renders natively in GitHub. This Issue remains open.

PR #195 was merged on the requester's explicit instruction as 5c6a8f6af7ecd8fa71cedd31889820eb1546aa50. The current main tree no longer contains the removed docs/evidence packet; the runtime optimization and manifest regressions remain byte-identical to merged #193. Issue closure remains a separate decision.