Share R1 semantic preparation through cooperative scope capabilities

docs/plans/bounded_full_audit_20260929.md

@@ -0,0 +1,108 @@
+# Bounded complete FULL audit
+
+Status: repair in progress. No production repair or complete audit is claimed.
+Tracking: [issue 11](https://github.com/OpenHCSDev/NominalRefactorAdvisor/issues/11).
+Base: `0844525ecaba93e090a064a4ae4914466b2dae60` (current remote main,
+29 September 2026). Worktree: `fix/bounded-full-audit-20260929`.
+
+## Required behavior
+
+Retain the complete declared production and dependency source corpus, every
+registered detector, authenticated source identities and existing native proof
+admission. `--json --raw-findings --json-payload full` must retain source index,
+observation graph and fibers, semantic-descent graph and refactor gate,
+finding-recipe plan, and raw-record evidence. A compact `agent` or `summary`
+payload is not a substitute. Increasing timeouts is not the repair.
+
+The motivating workload is OpenHCS plus its eight bundled libraries' production
+source roots. Tests follow the existing discovery policy. Scientific images,
+held-out annotations and notebook reference answers are not inputs to this task.
+
+## Source evidence at the base
+
+- `cli.py:754`, `JsonPayloadSections.compact_analysis_compatible`, excludes
+  source-index, observations/fibers and finding-recipe demand.
+- `cli.py:3911`, the compact analysis branch, therefore does not serve FULL.
+- `cli.py:3960`, the FULL branch, eagerly obtains all parsed modules before
+  `analyze_modules_with_cache`.
+- `ast_tools.py:2738`, `parse_python_module_roots`, retains every parsed result
+  in a list. `PythonModuleRootParser.parsed_source_paths` also returns a list.
+- `cli.py:1197`, `JsonPayloadBuilder`, consumes parsed modules for observations,
+  semantic descent and `CodemodSourceSnapshot.from_modules` for source-index and
+  recipe exports. These obligations cannot simply be removed from eligibility.
+- `analysis.py:273`, `DetectorAnalysisWorkerPlan`, uses no process pool with one
+  requested analysis worker. The baseline requests one parse and analysis worker.
+
+The shared checkout at `52fe8b4` is preserved but is not the repair base: it
+predates merged R1 detectors and subsequent native-proof changes. Historical
+79-detector receipts do not establish current main's detector coverage.
+The current source's `default_detector_types_for_analysis()` returns 85
+registered detectors. Import location was verified against the isolated worktree.
+
+## Current baseline and resource contract
+
+One FULL scan ran against the isolated current source and the same nine
+analysis/context roots. Its separate receipt is
+`scan-recovery-full-2048-cli-restart-current-main.*`; it does not overwrite the
+earlier held-prelaunch receipt. Native scan budget is 160 seconds, guarded shell
+shard 165 seconds, sampled scanner RSS ceiling 2 GiB. Start requires 11 GiB
+available and memory PSI full at most 1 percent for ten seconds. Stop on less
+than 8 GiB available, sampled RSS over the ceiling, or sustained pressure.
+This is a diagnostic envelope, not a new production default or a proof of
+constant-space behavior.
+
+Actual result: the guard stopped the scanner at 94 seconds when sampled RSS
+reached 2,105,892 KiB, above the 2,097,152 KiB ceiling. Exit 143 reflects that
+guard termination, not a native deadline. The FULL JSON is empty and the owned
+scanner is absent after cleanup. Thus no complete coverage or full/raw R1 result
+is available. This is a reproduced resource failure on current main with the
+original source context, not just the earlier held-prelaunch condition.
+
+An attempted 20-second `py-spy` attach to this owned scanner was denied by the
+operating system. It exited 1 and produced no profile. No privilege or kernel
+policy was changed. The scan result is not presented as controlled timing.
+
+## Ownership and validation work
+
+Main owns integration, the baseline and public tracking. Darwin reviews existing
+source/fact/snapshot lifetime owners; Nash independently reviews full-export,
+cache-invalidation and proof tests. Both initial reviews completed read-only
+against the same production base. Neither produced a production patch. Nash's
+test worktree is clean; no test patch or test run was made. Both workers are now
+closed to reduce the fleet after the desktop resource check warned about disk
+headroom and swap. The two owned worktrees were moved intact under `/home/ts/wt`.
+
+The user explicitly directed that NRA must not block OpenHCS work. Use the
+lightweight census/overlay scripts and bounded canonical Python queries for
+local ownership investigation, followed by actual native regression and MCP
+checks. Keep the FULL repair independent, without relabeling local evidence as
+global audit completion. The next OpenHCS defect is the already reproduced
+PolyStore POINT archive codec issue, tracked in PolyStore issue 12 / draft PR13.
+
+The census and overlay completed against current NRA production sources with
+no parse warning. A bounded Python query through `parse_python_modules` and
+`ModuleSyntaxIndex` inspected all 24 original classes in `roi.py` and 29 in
+`roi_converters.py`, exposing the existing ImageJ converter family. It exited 0
+in 0.94 seconds at 64,300 KiB maximum RSS. This is original-syntax inspection,
+not resolved native binding/MRO, invocation, codec or biological proof.
+
+The architecture review identified the decisive FULL lifetime problem: cached
+`CodemodSourceContext` still carries AST-bearing `ClassFamilyIndex`, and recipe
+compilation/simulation retains complete parsed snapshots. Extending bounded
+projection/source-index/observation owners is insufficient unless native recipe
+and proof materialization also gains a bounded original-identity lifetime.
+Do not merely enable compact eligibility or defer the same eager snapshot.
+The independent test review identified missing combined FULL cold/warm export,
+complete-status, dependency-invalidation and native-proof rejection regressions.
+
+Prefer one source owner with bounded native materialization and derived exports
+over another source roster, parser, proof cache or metadata mirror. Exact target
+ownership remains under investigation. Preserve fail-closed absent, ambiguous,
+changed and unsupported source/proof results. Tooling meets the same ownership
+standard as product code (refactor-audit AGENT-8); a partially migrated second
+route is not a completed repair (AGENT-2).
+
+Acceptance requires real all-detector/global completion, native FULL export
+behavior, raw-record evidence, authenticated context invalidation, and bounded
+wall time/RSS on this workload. Focused tests, cache hits or parse-only success
+must be reported separately, not promoted to full audit or biological proof.

docs/plans/r1_preparation_owner_357_20261001.md

@@ -0,0 +1,182 @@
+# R1 preparation owner checkpoint
+
+Source owner: Schrodinger/Codex. Integration and original complete acceptance:
+parent. References: [NRA issue11](https://github.com/OpenHCSDev/NominalRefactorAdvisor/issues/11),
+[pinned PR12](https://github.com/OpenHCSDev/NominalRefactorAdvisor/pull/12),
+[OpenHCS issue357](https://github.com/OpenHCSDev/openhcs/issues/357).
+
+Worktree: `/home/ts/wt/nra-r1-preparation-owner-357-20261001`.
+Branch: `fix/r1-preparation-owner-357-20261001`.
+Pinned baseline: `673c062fc656e9c74f1eddcab30f036c9befbc1f`.
+The baseline worktree and live main are read-only. No newer separate NRA
+implementer was present in PR12/issue11 comments or open PRs at admission.
+PR359's original consumer repair is already independently qualified; this
+source assignment does not duplicate that consumer/revision staging owner.
+
+## Required relation and current evidence
+
+Retain original revisions `791650087e01c9d25766f12802a37296851f37cf` and
+`3d58912d6e74e4b3dd4fa9e07a6b4a9d7dd9c7e4`, all three production roots and
+eight exact recursive dependencies. Original repaired consumer SHA256:
+`4c6282f52e7c188068dd9d63399f5522a7cb1d222504d0c0b809be10303884ca`.
+No detector copy, context pruning, source-cache spoofing, proof weakening,
+resource increase or global completion claim.
+
+Original complete attempt: 165.05s, 178.82MiB, exit-9. Original phase diagnostic:
+60.03s, 179.55MiB, exit124. Its 15/30/45s stacks point into preparation before
+the first scan_counts result, not a dominant-cost proof. Preserve both failures.
+The immutable diagnostic and transcript are the parent's
+`paired350-installed-20261001/diagnose-r1-phase-20261001.sh` and
+`producer-installed-20261001/paired350-r1-phase-diagnostic-20261001.{log,command.json}`
+under `/home/ts/wt/openhcs-issue-batch-20260929`.
+Its archived scratch address is terminal and must not be reused.
+
+## Admission and next source boundary
+
+Serial source diagnostics/tests only: one CPU, kernel512MiB/no swap, outer60s.
+Exact frozen installed Python3.12, read-only dependencies. No installs,
+downloads, native/UI/MCP/science, shared locks, subagents or paid provider calls.
+Startup helper reported 18.0GiB available RAM, home7.5GiB, swap13.6GiB and
+advisory critical disk/swap status. No additional worker fleet is started.
+The new owned disposable root is
+`/home/ts/.cache/agent-scratch/nra-r1-preparation-owner-357-20261001`.
+Retain recipes/profiles/resource receipts before cleaning exact owned scratch.
+
+No repository/ancestor AGENTS.md exists in the pinned source checkout or
+checked `/home/ts/code/projects` ancestry. Supplied standing AGENTS instructions
+and current NRA/refactor-audit skills remain binding.
+
+First obtain one bounded useful original-source profile, not another identical
+full scan. Trace actual determining declarations and their consumers before
+choosing a production change. Existing CollectedFamily/cache/source identities
+and proof admission remain the authority. Shared algorithms must live on an
+existing behavior-owning ancestor; leaves provide only varying hooks. Independent
+capabilities compose through genuine cooperative MRO only where justified.
+Relevant review: IMPL-12/13 repeated preparation, BOUND-1/2 admission boundaries,
+MEMB-1/2/5 declaration-derived inventories and TIME-1/3/9 replacement closure.
+Do not invent container dispatch families, a parallel schema or a compatibility
+facade. A new family/capability must need only its declaration, not consumer edits.
+
+Status: source implementation checkpoint. Original complete two-revision R1,
+installed acceptance and global85-detector/FULL audit remain unqualified.
+
+## Admitted source-owner move (2026-10-01)
+
+The bounded original-consumer profile contains eight parses and is not a full
+audit. `CompactSemanticModuleProjectionFamily._collect` consumed 3.231s;
+its presentation visitor and `DeclaredTypeCheckModule.collect` independently
+walked the same live AST (the check visitor's function handlers alone consumed
+0.702s). AST admission checks consumed 0.829s overall: the sparse predecessor
+stack does not establish them as dominant. No whole-context dominance is claimed.
+
+Required answers are unchanged presentation/class supplements and unchanged
+declared-attribute checks, including ignored lambda checks, shadowed builtins,
+nested scope ownership and exact original source locations. These independently
+meaningful capabilities already inherit `ClassFunctionStackNodeVisitor`; that
+ancestor owns stack lifecycle and the single traversal endpoints. A source-bound
+ancestor owns the shared parsed-module context, not a second identity store.
+The compact semantic collector composes both existing capabilities via C3 and
+cooperative `super()`, rather than invoking a second module traversal. The
+type-check record owns construction from its collector. No consumer type switch,
+new registry, detector copy or cache policy change is required.
+
+Catalog review: IMPL-12/13 (duplicate traversal mechanism), IMPL-4 (capabilities
+must really participate in cooperative traversal), MEMB-1/2 (no parallel roster),
+IDEN-5 (one source context), TIME-1/3 (delete second traversal; no fallback facade).
+The supported refactor DSL does not certify this cooperative visitor fusion;
+it is a manually authored source refactor with separate behavioral/proof gates.
+No native equivalence, installed acceptance or global audit claim follows.
+
+## First behavioral checkpoint
+
+Composition/publication suite: 20 passed, 5.79s wall, 86.79MiB maxRSS.
+The first invocation rejected pytest's inherited parallel flags (exit4); the
+next retained one failed new-case test (6 passed) exposing presentation early
+returns hiding another capability's events. Fixed by capability-local subtree
+suppression and continued shared traversal, not weakened assertions. Subsequent
+check cases cover projected assignment/annotated-assignment/return descendants.
+
+Original four-file source-family comparison: identical raw SHA256 and content
+signatures, 458 presentations, 36 declared checks, 292 class supplements.
+Collection 0.708180s baseline -> 0.619938s candidate (single sample, about12.5%).
+Process wall was 2.77s -> 3.61s; no cold-process/full-context improvement claim.
+Baseline/candidate maxRSS 80.92/83.10MiB. No source pruning of the original
+two-revision acceptance inputs is licensed by this bounded component comparison.
+
+Broader five-file test shard: INCOMPLETE exit124, 60.01s, 88.42MiB. Partial dots
+are not an accepted test count. Remaining checks split under the same bounds.
+All failures and profiles remain retained for the final source handoff.
+
+## Frozen source handoff
+
+Production checkpoint `2ae2309967904e0195f0d8e75b9b9988c736a52a`.
+[Validation index](../validation/r1_preparation_owner_357_20261001/INDEX.md)
+and [freeze manifest](../validation/r1_preparation_owner_357_20261001/freeze-manifest.json)
+retain exact commands, profiles, source/consumer hashes, all failed attempts,
+ownership/new-case review and cleanup. Draft PR16 is the visible source vehicle.
+
+83distinct source cases pass; one native proof test remains failing identically
+on candidate and unchanged baseline with `unproved_execution_effects`. The60s
+broader shard timeout, first pytest rejection and repaired new-case failure are
+retained. No proof weakening or blanket success claim. Peak observed source
+shard443.25MiB remained below512MiB.
+
+Canonical four-file component comparison via original module/path owner gives
+identical content signatures at0.747386 ->0.612257s collection in one sample;
+cold-process wall2.99 ->3.73s, so no overall startup claim. Original unchanged
+consumer profiles both stop intentionally at20CPU (exit75); semantic family
+cost3.231 ->2.853s, second standalone check traversal absent, not a completed
+two-revision comparison. Original3roots/eightexactdeps remain intact.
+
+Parent retains original R0/R1 160/165complete qualification and installed
+acceptance. Global85detector/FULL remains separately unqualified. No native/UI,
+science replay, install, lock, provider, download or other-owner source change.
+Owned19MiB scratch removed after handle/process proof and evidence preservation;
+the parent's original terminal cache address was neither reused nor removed.
+
+## Authorized same-node follow-through
+
+At predecessor c6ec13f the new independent same-node experiment demonstrated
+that projection handlers bypassed downstream Assign/AnnAssign/Return hooks,
+although earlier Call-only extension evidence passed. Source19f21b6 repairs
+this in the existing shared traversal owner, preserves local suppression and
+single traversal, and removes the bypassing helper. No confirmed installed
+regression is asserted. The [follow-through receipt](../validation/r1_preparation_owner_357_20261001/same-node-followthrough/README.md)
+and its separate freeze manifest retain both failing experiments, 47 distinct
+passing current source cases (7 new), unchanged original component signatures,
+resource bounds and cleanup. Original failure evidence remains unchanged;
+parent's installed run and full qualification remain untouched.
+
+## Original full-context failure and bounded publication profile
+
+Parent's exact original18910a5 qualification is terminal ScanDeadlineExceeded160s,
+exit1/164.42s/187.41MiB, not a pass. The latest [publication/preparation receipt](../validation/r1_preparation_owner_357_20261001/publication-profile/README.md)
+retains that witness and one bounded45CPU profile inside the unchanged60s source
+envelope. AST retention consumed2.752/45.582s sampled work, not demonstrated
+overall dominance. Source remains19f21b6. A concrete unresolved IMPL-12/13
+duplication is recorded: the two composed function capabilities independently
+rescan the same lexical body bindings (4393 calls each). It requires source-scope
+owner reuse with lifecycle/proof behavior, not another cache authority or guard
+waiver. This profile/review checkpoint adds no production fix or new test pass.
+
+## Shared determining function scope
+
+Sourcefbee396 extends the existing source-bound ancestor to own one original
+lexical binding determination per function event, supplied through cooperative
+context hooks to the independent presentation/check capabilities. Repeated leaf
+calls/event overrides and copied unsafe postlude name-stack lifecycle are deleted;
+the existing owning stack ancestor now protects both traversal and postlude.
+No cache, store, roster, copied algorithm or proof/AST-admission waiver is added.
+BindingCensus supplies only a new declaration/hook, participates before/after C3
+and demonstrates real nested/async ordering, freshness and exception restoration.
+
+The [source receipt](../validation/r1_preparation_owner_357_20261001/function-scope-reuse/README.md)
+and [freeze](../validation/r1_preparation_owner_357_20261001/function-scope-reuse/freeze-manifest.json)
+retain the red two-determination experiment,79 distinct current source passes
+(12 new), identical complete component signatures and exact bounds/cleanup.
+Single-sample component timing is essentially flat; no complete R1 speedup is
+claimed. Original parent failure and baseline-equal native proof failure remain
+retained. Class syntax facet binding requests remain independently owned.
+Parent retains original160/165/512MiB complete qualification and installed gate;
+global85-detector/FULL remains separately unqualified. Source development remains
+active, not terminated by a fixed submission count or renamed a global pass.