Derive native function-catalog readiness from current projections

docs/validation/analysis-claim-scope-guide-20261003.rst

@@ -0,0 +1,92 @@
+Claim-scoped development guidance: source-only checkpoint
+========================================================
+
+Scope and original owners
+-------------------------
+
+This is a generic how-to correction, not a biological acceptance, a new
+evaluation rule, an assay recipe, or a hot update to an active scientist.
+Previously frozen outcomes and the qualified catalog511-package01 target remain
+unchanged. No benchmark answers, object counts, tuned parameters or trial
+transcripts are included in the guide changes.
+
+Current main inspected: f7de9efad393bce83a3756a271f0cd4383783ea9. PR472 current
+head 6aad9f1818f464ac6b8eea24bbd0a1569f559131 owns historical pinned394/404 wheel
+receiving, not these guides. Singer confirmed no analysis/custom guide claim;
+Planck confirmed no current guide claim and no borrower of this checkout.
+Their current522/494 source families do not edit the three changed references.
+Current open PR heads and the nearest related owners' claims were checked;
+no independent implementation or guide owner was inferred from an old PR title.
+
+The original ``mcp_knowledge_base_manifest.json`` projects these existing sources:
+
+* ``openhcs_autonomous_analysis_strategy``: packaged ``analysis-strategy.md``;
+* ``openhcs_viewer_qa``: packaged ``viewer-qa.md``;
+* ``openhcs_custom_function_workflow``: packaged ``custom-function-authoring.md``.
+
+``ImageAnalysisWorkflowAuthoringContext`` and
+``CustomFunctionAuthoringContext`` already route to those document identities.
+The plugin's original ``AgentSkillBundle`` includes the same reference files;
+no manifest entry, generated duplicate, skill mirror or new guide is required.
+
+The canonical reStructuredText entry/reference owners are
+``docs/source/guide_for_biologists/domain_expert_onboarding.rst``,
+``docs/source/user_guide/custom_functions.rst``, and
+``docs/source/development/callable_artifact_authoring.rst``. They retain their
+existing scope and are not copied or rewritten here. Live QA gate declarations
+remain owned by ``ImageAnalysisQaPolicy``; no runtime policy/codec/catalogue or
+enum definition changes.
+
+Gap and correction
+------------------
+
+Existing guidance already requires earliest-stage diagnosis, foreground before
+unclumping, ambiguity triage, one semantic change, regression witnesses and
+continued development within the authorised resource budget. Those procedures
+are not duplicated. The missing instruction was how to scope conclusions when
+only some objects or relationships remain unresolved.
+
+The strategy now preserves independently supported claims, records unresolved
+cases and exclusions, permits justified bounds/sensitivity analysis, and limits
+whole-population claims when the unresolved cases affect them. It neither
+certifies completeness from sampled positives nor treats partial support as a
+reason to stop repairing a clear failure. Viewer QA links to that single owner
+and ties candidate rejection to the requested claim's acceptance criteria.
+
+Custom authoring now requires evidence for actual imports, decorators, helper
+types and enum members in the installed version before source submission.
+Existing public architecture inspection accepts curated symbol IDs, not arbitrary
+import paths. ``InternalApiSymbol`` returns signature/doc/source-location facts,
+not complete enum members or source text. The guide does not promise that this
+route alone proves a member, invent a reflection endpoint, or copy an enum list.
+An unavailable declaration is an exact authoring-contract gap, not evidence that
+unrelated scientific outputs are invalid.
+
+Acceptance boundary and preservation
+-----------------------------------
+
+Complete use-openhcs and Diataxis skills were read. The strategy, viewer QA,
+custom authoring and analysis-learning references and relevant original context,
+knowledge manifest, skill bundle and architecture projection relationships were
+read. Diataxis kept the change an action-oriented how-to, with facts linked to
+their existing reference owners rather than duplicated API catalogues.
+
+Normal current-main integration preceded the three reference edits. It preserves
+merged520 resource guidance and merged503 source; all foreign submodule worktrees
+and gitlinks are untouched. The three catalog511 production files remain
+byte-identical to qualified cb263946e92929c2476f1125517964713ac931ae.
+
+Source review and ``git diff --check`` pass. Original knowledge routes still name
+the same canonical paths and relative links resolve to existing references.
+No imports, tests, builds, installs, native/MCP/client/provider launches or
+scientific contact were used. This is source availability/reasoning evidence,
+not installed retrieval acceptance or an autonomous biological improvement.
+Future package/retrieval and fresh blind performance evaluation remain separate;
+the previously qualified target still contains the previous guide bytes.
+
+Read-only lookup negatives were retained in the tool history: an initial
+``gh`` request inferred the wrong repository and returned no PR472; explicit
+``-R OpenHCSDev/openhcs`` resolved the actual PR and current open heads. Guessed
+``skill_manifest.py``/``architecture_service.py`` names were absent; actual
+``skill_bundle.py``/``architecture_projection_service.py`` owners were read.
+These were source lookup failures, not scientific attempts or runtime operations.

docs/validation/catalog-current-readiness-20261003.rst

@@ -0,0 +1,227 @@
+Current native function-catalog readiness
+========================================
+
+Owner and reproducer
+--------------------
+
+FunctionCatalogService owns the existing metadata-derived public projections;
+FunctionCatalogPreparation owns their worker, future, cancellation and progress.
+The native READ/SEARCH/DETAIL/REFERENCE template already calls this owner.
+
+The retained H001_REPEAT02 journey demonstrated successful custom registration,
+an old READY preparation receipt, and subsequent catalog READ timeout at 4975ms.
+An intervening describe succeeded; the preceding search is UNKNOWN, not replayed.
+No inner per-callable cost was measured. This is a stale readiness correction,
+not a claim that one particular reflection operation has been timed or optimized.
+
+The old successful future was reused forever, while metadata invalidation could
+make the next public request rebuild a complete catalog projection in REP. Only
+the compact view was prepared before serving; default full-view search could
+also cause that work synchronously.
+
+Change
+------
+
+* Derive current readiness from the existing projection keys and metadata,
+  RegistryService.cached_metadata_snapshot, and the original persisted-source
+  revision owners. Do not maintain another generation, catalog or revision cache.
+* Prepare both existing public views before advertising READY. Refresh only
+  these projections after invalidation, using the original preparation worker;
+  do not repeat native kernel startup for an ordinary source change.
+* Coalesce pending work; preserve failed/cancelled futures without automatic
+  retry. A completed historical future is not current READY after invalidation.
+* Apply the existing OperationCancellation to per-callable projection work.
+  Check cancellation after the last entry and before successful publication.
+* Keep FunctionCatalogPreparation on the original FunctionCatalogServiceABC
+  boundary. That ABC owns the shared two-view refresh recipe; local and endpoint
+  implementations own their respective current-readiness decisions. The endpoint
+  implementation asks the original bound native preparation-status owner.
+* Move the endpoint implementation's existing state lock to the ABC owner;
+  reuse it for short cache adoption/publication sections in both implementations.
+  Reflection and source-revision polling remain outside it. An old view cannot
+  enter a cache that another caller has already switched to new metadata.
+* Leave registration admission, receipt and original invalidation untouched.
+  The next public catalog request uses the same existing preparation gate.
+
+Production scope is agent/services/function_catalog_service.py,
+agent/services/endpoint_function_catalog_service.py and
+runtime/function_catalog_preparation.py. RegistryService, CallableProjection,
+custom-source decoding, compiler, native/MCP protocol and deadlines are excluded.
+Root394 was notified before production edits in comments 5965485052/5965541595;
+the additional ABC/endpoint hunk was declared in comment 5965707429. Its checked
+8e06a382 head does not change these files. No competing catalog authority,
+generation store, protocol, timeout or fallback was introduced.
+
+Source and custody
+------------------
+
+Parent explicitly released the former programme source claim on the existing
+/home/ts/wt/openhcs-input-preparation-20260929 checkout, originally e0dd14ed.
+The exact full path matches Dalton's direct ownership response. Parent reports
+privileged whole-borrower count 0 and access omissions 0. The complete Dalton
+receipt was personally read at neurite-development-skill383-20261001/output/
+resource-owner-20261002/COPERNICUS-CATALOG-SOURCE-REUSE-INPUT-PREPARATION.rst
+under the existing issue-batch root. No source reuse of the
+borrowed Singer505 or live436 checkouts was made. The branch was switched
+nonrecursively from current main87d9a99; foreign submodule directories and .git
+files were not updated/reset/cleaned. Their changed status against current-main
+gitlinks is retained and excluded from this change. No environment was created.
+
+Full current NRA/refactor-audit skills, required pattern catalog and applicable
+antipatterns were read. The existing selected-family AST receipt covers 28
+production modules and 8 installed dependency modules, 5235 source sites, with
+zero parse errors. Dynamic behavior is not proven by that receipt. The standing
+OPENHCS-HISTORY.md path is absent at main87d9a99 and the canonical checkout; no
+replacement reminder/history store was invented.
+
+Acceptance and remaining boundary
+---------------------------------
+
+After the coherent source change, bounded controls must cover:
+
+* cold preparation and ordinary startup; warm repeated compact and full views;
+* one-worker coalescing after add, same-id update and delete, without kernel rewarm;
+* body-only persisted-source changes despite unchanged catalog membership ids;
+* historical READY rejection and honest pending progress before refresh completes;
+* projection failure, cancellation, retained terminal futures and incarnation mismatch;
+* original read/search/detail/reference strategy consumers and admission behavior.
+
+Source controls do not establish installed/native acceptance. A later explicitly
+released engineering slot must exercise the ordinary installed public registration,
+search, describe and reference journey with unchanged budgets, plus responsiveness
+during preparation. No currently live scientist, native, UI or UNKNOWN attempt
+will be contacted/replayed. Inner reflection cost remains unmeasured. A registry
+inventory lock held by a separate concurrent registry worker is an excluded Root
+seam if encountered; it is not silently fixed with another lock or timeout here.
+
+Source checkpoint and actual controls
+------------------------------------
+
+Draft PR511 was published at 3e1e336089ccf2d3d3e20a849a885193daff1bba before
+longer verification. The postchange selected-family AST receipt has 28 production
+modules, 5262 sites, 8 dependency modules and zero parse errors. Runtime claims
+remain separate from this evidence. ``git diff --check`` passed.
+
+All four bounded collection attempts are retained in the original engineering
+receipt root as catalog511-source01..04.log/time; each used CPU1/512MiB/Swap0/60s.
+They executed no scientific functions, native servers, sockets or viewers:
+
+1. systemd working directory omitted: pytest rejected relative confcutdir.
+2. corrected directory: dependency import preceded source bootstrap; corrected
+   the test import ordering through the original source owner.
+3. collection then reached missing compiled openhcs.core._tabular_native.
+4. reused the accepted505 real extension directly, without copying/building it;
+   its C++ source matches this checkout, SHA256 of the ABI3 extension is
+   7de8f671e79263518e56219b30085b2c39c9518db63739298c4ffc9265196aae.
+   Collection then failed because the protected old external/zmqruntime checkout
+   lacks ViewerReuseAdmissionABC required by current-main viewer_protocol.
+
+No assertions ran in those four attempts: this was a precise dependency-collection blocker, not a passing
+regression suite. Attempt04 peaked at 179752KiB RSS (unit peak123.3MiB), zero swap,
+3.015s. The last run reused an explicit actual compiled dependency for source
+controls only; it is not ordinary installed/public/native acceptance. No substitute
+registry/factory/backend, native extension stub, path fallback or protocol was
+introduced into production. All foreign external trees remain unchanged. They
+were not updated or patched. Parent then explicitly authorized binding existing
+qualified dependency origins read-only for source controls; no installed file or
+environment was changed.
+
+The authored controls cover warm full/compact public projection, current-source
+revision, add/update/delete/body-only invalidation, refresh coalescing, retained
+failure/cancellation, incarnation rejection and original main-thread startup.
+Receiving controls and nominal family correction
+-----------------------------------------------
+
+The existing shared interpreter is
+/home/ts/wt/openhcs-paired-raw-installed-parent-20261001/.venv/bin/python.
+The explicitly asserted dependency origins are that environment's
+lib/python3.12/site-packages for zmqruntime, PolyStore, arraybridge,
+metaclass-registry and pyqt-reactive, and
+/home/ts/wt/basicpy-live-candidate-20260930/.venv/lib/python3.12/site-packages
+for python-introspect (the shared interpreter's original declared dependency).
+The real accepted505 ABI3 extension remains at the existing engineering receipt
+root's scratch/installed505/openhcs/core/_tabular_native.abi3.so. It was not copied,
+built or substituted. Source module paths and dependency origins/hashes are
+asserted and recorded by catalog511-shared-dependency-controls.py on each run.
+Determining zmqruntime, PolyStore and python-introspect files were compared against
+the current recorded Git objects through the canonical repository; this is not
+a claim of whole-package identity or installed/native acceptance.
+
+All receiving attempts remain in the same receipt root:
+
+* source05: 18 passed, one test-fixture binding failure.
+* source06: 71 passed, six fixture failures (router-only fixtures did not declare
+  current catalog readiness; one cleanup cancelled work before expecting success).
+* source07: 77 passed after coherent fixture corrections. Real invalidation,
+  cancellation and failed-future negatives were retained.
+* source08: 78 passed, including the overlapping-reader obsolete-cache-commit
+  control, midpoint metadata/source invalidation, all nominal ABC implementations
+  and the final-entry cancellation checks. Test time 9.57s, wall12.94s,
+  cgroup peak214.2MiB, Swap0, CPU1/512MiB/60s, exit0. Two pytest configuration
+  warnings result from deliberately disabled plugin autoload, not failing tests.
+
+The production ABC owns the only shared refresh recipe and family state-lock
+declaration. Both implementations retain their original catalog projections;
+all preparation/strategy consumers take the existing nominal boundary. No
+reflection or source-poll performance claim follows from these controls.
+No build, installation, scientific execution, server or viewer launch was
+performed. Saved diagnosis, UNKNOWN requests and scientific records remain
+immutable. Actual ordinary installed public/native acceptance still awaits an
+explicitly released engineering slot; :95 is not yet released.
+
+Current-main receiving boundary
+-------------------------------
+
+Working checkpoint 593eecfcb was normally merged with current main236644e43 at
+3cccbb0f1, including guide512/d11032292 and disjoint registered-source warmup514.
+No dependency working tree was changed by this nonrecursive merge. The main
+change records ObjectState9c31e4b07b5537c0ed2307c5a63034c708b86bbb and
+python-introspect6f5ac0b79d6cef163df6a8389e72a889a91ed548.
+
+Merged-source attempt09 is retained: 77 passed, one failed in 8.87s,
+unit12.161s/peak203.3MiB/Swap0/exit1. The original real-registry startup control
+reaches main514's new LazyDataclassFactory.registered_type_pairs call, absent in
+the previously qualified dependency. This is an exact receiving-dependency
+boundary, not passing current-main acceptance; the negative was not weakened.
+The two current dependency Git objects are not present in the canonical local
+dependency stores, whose working heads differ and include foreign dirty edits.
+They were neither imported nor fetched/updated. Parent was asked for existing
+qualified origins for those exact main514 dependencies. No fallback, stub,
+monkeypatch or installation substitutes for this missing behavior.
+
+The complete selected family includes production implementations and test
+subclasses/fakes/consumers. The existing startup PreparedCatalog fake now inherits
+the nominal service and shared ABC recipe rather than copying a one-view recipe;
+startup expectations observe both views. The first expanded AST attempt03
+also matched a historical pytest .txt transcript: its non-Python parse error is
+retained, not erased. Subsequent source enumeration explicitly selects .py files;
+the transcript is evidence, not a source module.
+
+Expanded AST attempt04 also remains retained: a Git pathspec union accidentally
+selected an unpackaged benchmark script, causing a missing installed-comparison
+file error. Corrected attempt05 restricts the additional family to Python files
+under tests: 28 production modules/5291 sites, 27 test modules/3561 sites and
+8 original receiving-dependency modules, zero parse errors. Source identity is
+5dc21d5720baca32f8170f89b2359c8b8408b4ba; receipt SHA256
+f63738f8a40989b50f133b4db8a220f6da54f088c5c9d72f6434b7f642fe5645.
+The selected production family has exactly one self._state_lock declaration and
+one prepare_projections recipe, both on FunctionCatalogServiceABC. This is
+focused static source evidence, not global/dynamic or installed proof.
+
+Two targeted startup-fixture checks were already submitted before the parent's
+instruction to resolve main514 origins before more checks; attempt10 completed
+2 passed/1.80s, unit5.103s/peak139.8MiB/Swap0/exit0. It verifies the final nominal
+fixture consumes the shared two-view recipe; it does not supersede attempt09's
+failed real-registry startup control. No more tests/builds were submitted after
+that instruction. Main514/issue513 and the determining upstream PR bodies were
+read completely: their named owner is Tristan/trissim, with recorded21-pass/
+5-skip integration controls and R0/R1 acceptance, but no published exact qualified
+dependency-origin receipt path. Parent was asked for the original tested artifact
+or source snapshot through its ordinary dependency mechanism. Current foreign
+dependency worktrees and their missing-object boundary were left unchanged.
+
+Owned receiving-control scratch is recorded under the original receipt root's
+scratch/catalog511-controls06, 07, 08, 09 and 10 (pytest/data/cache), with the earlier
+attempt scratch retained alongside it. No uncertain request or science evidence
+is disposable. All bounded source-control processes have known terminal exits;
+there is no retained engineering native/server/viewer handle.