Expose dev-client decode diagnostics and reconcile derived response fields (#578)

docs/validation/mcp-shell-derived-diagnostic-20261004.rst

@@ -0,0 +1,235 @@
+Closed-journal MCP shell exit1 investigation
+==========================================
+
+Singer owns OpenHCS response/JSON diagnostic receiving; python-introspect's
+declaration decoder is a shared dependency seam being coordinated with Dewey.
+Root394/541 source/units and merged567 native semantics remain independent.
+
+Pinned currentmain2df0918353b47fa07d19fe4b6cc8b52b5e4c1695. Existing checkout
+was reused on a new branch after567 closed; six foreign dirty gitlinks and all
+untracked originals remain untouched. Open PRs394/404/472/494/518/522/541/468
+have no active dev-client shell/result implementation claim. Current394 file
+census has no dev_client, dto/functions or serialization/json crossing.
+
+Original engineering567/public94-attempt01/mcp.stdout records eight public
+operation receipts with empty visible errors; original shell ends terminal1.
+No new client/native process or request was used to determine the cause.
+
+Actual original qualified target8513 decoder applied offline to all eight closed
+receipts: seven decode without error; registration-status becomes
+McpDevPayloadFailure with mcp_payload_invalid/ValueError:
+CustomFunctionRegistrationObservation received undeclared field(s): outcome.
+
+Original python-introspect.dataclass_from_mapping only admits init=True fields.
+Observation.outcome is declared init=False and derived from publication/saved
+proofs in __post_init__; original to_jsonable emits all dataclass fields.
+The same nominal declaration therefore serializes a field its decoder rejects.
+McpDevToolResult.has_errors correctly sees the rejection; persistent shell
+aggregates that returncode. The custom McpDevPayloadFailure JSON projector emits
+only receipt, deleting rejection diagnostics from visible JSON. This explains
+the exit1 without alleging native registration/status/close failure.
+
+Original read-only diagnostic script/logs remain under engineering567/
+public94-attempt01/DIAGNOSE-CLOSED-RESULTS01.py/.stdout/.stderr: terminal0,
+4.12s,236672KiB peakRSS,Swap0; no sockets or source/image execution.
+Actual original dev_client_core.py SHA256
+2aaaffec501e901352192ed35016caa6ce00122b429e41b3f7c8683eb3fd1aed;
+target01/python_introspect/dataclass_projection.py SHA256
+82662ff8eaa4f2aeb3b33edd61abae57020a7f139c3aa396d82dcecf8b801160.
+
+Required relation and owner destination
+---------------------------------------
+
+One declaration must drive encode/decode of derived fields. Constructor inputs
+remain constructor-owned; supplied derived wire values must be checked against
+the actual constructed result, not dropped, assigned or allowed to override it.
+Unknown fields and mismatched derived proofs must fail closed. Reuse the original
+python-introspect decoder, not a consumer-specific outcome pop/type switch or
+alternative codec. BOUND-1/2/8 apply at this declaration boundary.
+
+The existing McpDevToolResult/McpDevPayloadFailure owns failed-decode diagnostics
+and original receipt. JSON, formatted output, returncode and persistent aggregate
+must project that same owned failure, not silently serialize success while
+retaining hidden exit1. No error suppression/--allow-error-payloads workaround.
+
+Acceptance after coherent implementation: original closed status receipt decodes
+to the declared observation with exact matching proofs, shell returns0 for those
+unchanged successful receipts; genuinely malformed/mismatched/new declaration
+cases retain original receipt and visible typed diagnostics with nonzero exit.
+Use complete relevant AST/dependency/consumer evidence before production edits;
+focused original user entrypoint verification last, no mutation replay/new client.
+
+Working diagnostic checkpoint
+-----------------------------
+
+Deleted the five-line McpDevPayloadFailure serializer that returned only the
+native receipt. The original canonical dataclass serializer now exposes the
+same declared receipt/errors record used by returncode and formatted output.
+The existing failure declaration rejects an empty diagnostic cause. The original
+McpDevToolResult decoder restores this common rejection record through the same
+dataclass_from_mapping codec, so a JSON batch roundtrip keeps the original cause
+once rather than hiding it or wrapping a second rejection. No raw-shape parser,
+business-leaf switch, codec, status flag or registry was added.
+
+Source qualification used the existing viewer-retirement-519/run_source_controls.py
+bootstrap: OpenHCS source activation first, then read-only paired startup and Qt
+backing before their consumers. SHELL-DIAGNOSTIC-CONTROLS02 passed20, terminal0,
+6.91s wall,270032KiB peakRSS,Swap0. It exercised strict producer decode, independent
+capability/MRO behavior, visible JSON cause plus unchanged receipt, batch roundtrip,
+and original transport failure ownership. No server/native operation was launched.
+The previous CONTROLS01 collection failure from unmatched foreign Qt backing is
+preserved, not rewritten. The two plugin-free pytest configuration warnings are
+retained in the original output.
+
+Complete source evidence reused the original audit.Package/Repository tooling:
+704 OpenHCS modules parsed with zero omissions;40 relevant full module ASTs,
+plus12 original qualified python-introspect module ASTs with zero omissions.
+SHELL-FAMILY01.jsonl/.stderr retain declarations, imports, writes and consumers.
+Actual census terminal0,13.06s,538200KiB peakRSS,Swap0; this is source evidence,
+not a complete NRA R1 ownership proof and not a <=512MiB claim. The transport
+derived-field census finds Observation.outcome as the sole init=False field in
+agent/MCP DTOs; internal cache/lock/thread init=False fields are not alternate
+registration status authorities.
+
+Remaining: the shared python-introspect declaration decoder still rejects the
+original successful derived outcome. Dewey was asked for the existing dependency
+owner implementation/release; no foreign dependency checkout or gitlink was
+edited. This diagnostic checkpoint makes that original cause visible, but does
+not yet claim successful observation CLI roundtrip or corrected shell terminal0.
+The original native positive and original shell terminal1 remain separate facts.
+
+Consumer closure and shared-code disposition
+-------------------------------------------
+
+The related sampling, pipeline/artifact-plan and UI-action JSON assertions now
+retain the entire original rejected receipt AND assert its typed diagnostic
+projection, instead of asserting the removed diagnostic-erasing shape. No
+success assertion or nonzero exit assertion was weakened. The real
+McpDevClient.execute pipeline command check retains its original controlled
+wire peer and production command/framing/rendering/returncode path.
+
+CONSUMERS03 retained its original terminal1:62 passed/11 failed. Five failures
+identified additional old failure-shape assertions, now migrated. Four were the
+existing sibling test-module import under importlib mode; the original test
+directory is explicitly on PYTHONPATH for CONSUMERS04. Two were an existing
+UI-test global serializer monkeypatch captured by subsequently imported knowledge
+renderer aliases; the independent UI and pipeline/sampling qualifications are
+kept distinct rather than changing production serialization to tolerate it.
+CONSUMERS03's12 UI controls passed. CONSUMERS04's pipeline/sampling61 controls
+passed, terminal0, including the affected real execute boundary. Original raw
+failure and successful output/timing remain byte-exact in the archive.
+
+Original unmodified debt_census.py against main2df0918 and production8fbba870f,
+exact sole changed production file openhcs/mcp/dev_client_core.py: all measured
+debt deltas zero, code lines+3, zero parse omissions, terminal0,0.56s wall,
+25016KiB peakRSS,Swap0. R0 is a syntax ratchet, not live/whole NRA proof.
+
+Dewey explicitly confirmed no released dependency checkout: the foreign
+python-introspect source has active transitive borrowers even though its child
+tree is clean. He forwarded the exact original-decoder requirement to Parent
+integration. Singer has not edited that checkout or its gitlink. The remaining
+shared hunk is dataclass_from_mapping's declared-field algorithm: admit all
+declared names, construct only init fields, decode supplied derived values with
+the same annotation mechanism, compare with actual post-construction owner
+values, and reject contradictions. project_dataclass remains a constructor-field
+projection that intentionally recomputes target-derived facts; it is not a wire
+decoder and must not be turned into a second status authority.
+
+No installed CLI terminal0 claim yet: the original native reply needs that codec
+integration before its successful derived outcome can roundtrip. The diagnostic
+source checkpoint works independently and does not claim recovery/replay of any
+UNKNOWN operation. Planck's disjoint Shapes geometry ownership was confirmed;
+522 selection/lifecycle and541 Root units ownership remain untouched.
+
+Reviewed dependency and actual CLI receiving
+-------------------------------------------
+
+The preceding dependency-release blocker is historical. Parent owns the distinct
+python-introspect checkout and PR10. Source reviewed at ee0b65f and final
+ac7b2388ed800649a8d0157f00056471a466dd20: dataclasses.fields owns membership;
+get_type_hints(include_extras=True) and the existing recursive annotation decoder
+own input conversion; only init fields reach the constructor. Supplied non-init
+values are compared with validated actual constructed values, never assigned.
+Omitted derived fields are legal; contradictory values and unknown fields fail
+closed. No outcome special case, mirrored field roster or second codec exists.
+Equality retains the field owner's existing Python equality contract; this is
+not an array-specific comparison extension. No concrete defect found in review.
+
+Read-only codec source selected after original OpenHCS source activation, before
+any consumers, through the existing source bootstrap. Actual imported codec:
+/home/ts/wt/python-introspect-derived-decode-20261004/src/python_introspect/
+dataclass_projection.py, SHA256
+d2be277051b070d73503025a3addcde8016787e18a369c45b60be0c0edb65fea.
+The receiving process imported version0.1.15 at fad3cbb. Exact source diff to
+ac7b238's0.1.16 changes only __version__ in src/python_introspect/__init__.py;
+the tested decoder and all other source algorithms are byte-identical. No
+foreign gitlink, installed dependency or existing scientific target was changed.
+
+SHELL-COMBINED-CLI05 retained terminal1:17 source controls passed, including the
+new independent derived declaration using inherited cooperative describe hooks,
+omitted derived fields and contradiction rejection without generic consumers
+being edited. Four receiving cases failed before wire dispatch because the new
+fixture mistakenly supplied --no-resident-server instead of the original
+--no-resident option. This was a known harness error, not native UNKNOWN.
+Original log retained:7.24s wall,261568KiB peakRSS,Swap0. The corrected fixture
+also consumes the real pretty-printed JSON documents through stdlib JSONDecoder.
+
+Only the four affected receiving cases ran in SHELL-COMBINED-CLI06. All passed:
+normal dev_client.main -> persistent shell -> original McpDevClient.start/execute
+and command/framing/DTO decoding -> canonical JSON -> shell aggregate returncode.
+Only the transport context/peer is controlled, reusing the existing pipeline
+ControlledWireSession. Process launches are forbidden by the fixture. Exact
+original command arguments/tool order and complete saved replies are retained.
+No old process was contacted, no source was evaluated, no native operation or
+registration was replayed, and no new MCP/native/client subprocess was started.
+
+The eight original successful replies yield shell terminal0. The original
+registration observation is the declared DTO with registered outcome, zero
+errors, one published source and its exact persisted proof; canonical projection
+equals the entire original status receipt. The three copied malformed replies
+(contradictory outcome, boolean outcome, undeclared field) each yield terminal1,
+visible mcp_payload_invalid/ValueError cause and the full original receipt with
+the single explicit test mutation. Native receipt errors remain empty; local
+decode errors remain a separate fact. Re-decoding canonical batches retains the
+same cause once and nonzero status.
+
+Actual controls06 terminal0,4PASS,5.73s wall,254896KiB peakRSS,Swap0. Full canonical
+CLI outputs and per-case statuses are in the raw stdout, not just test assertions.
+No successful17-control repeat or7GiB installed cold import occurred. Before this
+source batch, original resource check read9.3GiB available, critical warnings
+for existing swap/disk, full-memory PSI avg10=.24 then .17. This receipt reports
+actual measured resource cost, not a new hard cap or scientific admission rule.
+
+The real closed native TCP journey remains the accepted merged567 evidence;
+this acceptance establishes its corrected source CLI decoding/rendering/exit
+path with a controlled closed-reply peer. It does not claim a fresh installed
+0.1.16 client/native journey or recovery of an uncertain mutation. Public
+dependency0.1.16 publication and future bundle projection remain Dewey's release
+ownership. Existing OpenHCS dependency floor is still >=0.1.14; original metadata
+owner must carry the corrected minimum for ordinary future installation.
+
+Ordinary installation requirement
+--------------------------------
+
+Parent explicitly released the narrow pyproject.toml metadata hunk into579
+after python-introspect0.1.16 publication. The canonical production requirement
+is now python-introspect>=0.1.16,<0.2, excluding earlier decoders that reject
+derived fields and preserving the breaking-series bound. This supersedes the
+historical floor dependency above. Dewey/518 received the exact requirement
+coordination so later metadata integration must retain0.1.16 rather than0.1.15.
+No gitlink, ObjectState requirement, vendoring or installed package changed.
+
+Dewey's existing PUBLISHED-PYPI-PROOF.rst records workflow37174368240SUCCESS at
+merge430fbbeeb6c7e0a9274d3bf9b0998a10cae5a505 and exact public wheel/sdist source
+identity. Publication is not installed receiving. The four-case CLI acceptance
+and production R0 evidence above are unchanged; no full tests/build/import or
+scientific target mutation was repeated for this one-line metadata correction.
+
+SHELL-METADATA-FLOOR07 uses the original read_project/read_release_candidate and
+CandidateRequirementCompatibility owner, not another requirements parser:
+0.1.14/0.1.15 excluded,0.1.16/0.1.17 admitted,0.2.0 excluded; actual candidate
+floor/compatibility proofs true. Terminal0,0.19s wall,26444KiB peakRSS,Swap0.
+Original raw command/output remains in engineering567/public94-attempt01.
+Root's sole518 metadata ownership was rechecked via the current PR/comments;
+the precise one-line hunk and future430 source-pin boundary were addressed
+directly on518. His broader branch and release family remain unedited.

openhcs/mcp/dev_client_core.py

@@ -407,10 +407,9 @@ class McpDevPayloadFailure:
     receipt: JsonValue
     errors: tuple[AgentError, ...]
 
-
-@to_jsonable.register(McpDevPayloadFailure)
-def _jsonable_payload_failure(value: McpDevPayloadFailure) -> JsonValue:
-    return value.receipt
+    def __post_init__(self) -> None:
+        if not self.errors:
+            raise ValueError("A rejected MCP payload requires its diagnostic cause.")
 
 
 @dataclass(frozen=True, slots=True)
@@ -451,7 +450,13 @@ def _decode_payload(payload, contracts):
                 rejections.append(
                     AgentError.from_exception("mcp_payload_invalid", error)
                 )
-        return McpDevPayloadFailure(payload, (*_agent_errors(payload), *rejections))
+        # The local rejection is a declared transport record too. Its canonical
+        # dataclass projection retains both cause and original native receipt.
+        # Recover that record through the same codec, not another error parser.
+        try:
+            return dataclass_from_mapping(McpDevPayloadFailure, payload)
+        except (TypeError, ValueError):
+            return McpDevPayloadFailure(payload, (*_agent_errors(payload), *rejections))
 
     def first_decoded_payload(self):
         """A missing or rejected payload is not a successful empty record."""

pyproject.toml

@@ -102,7 +102,7 @@ dependencies = [
     "zmqruntime>=0.3.0,<0.4",
     "pycodify>=0.1.3,<0.2",
     "objectstate>=1.1.8,<2",
-    "python-introspect>=0.1.14,<0.2",
+    "python-introspect>=0.1.16,<0.2",
     "metaclass-registry>=0.2.2,<0.3",
     "arraybridge>=0.3.6,<0.4",
     "polystore>=0.3.0,<0.4",