Skip to content

Reset child signal ownership to preserve parent ZMQ endpoints - #160

Merged
trissim merged 7 commits into
mainfrom
plan/issue-140-zmq-failure-20260929
Oct 6, 2026
Merged

trissim merged 7 commits into
mainfrom
plan/issue-140-zmq-failure-20260929

Conversation

@trissim

@trissim trissim commented Sep 29, 2026 •

Copy link
Copy Markdown
Collaborator

Public cancellation of real fork workers can remove the live parent server's IPC endpoints. Fork children inherit serve_forever's SIGTERM/SIGINT handler, which captures the server and calls server.stop(). Terminating those children runs copied server cleanup: the parent process remains alive, but fresh clients cannot reach it.

Existing WorkerExecutorResources now owns child signal initialization. Both the actual fork entry and existing pooled-process initializer reset SIGTERM/SIGINT to SIG_DFL. Parent/inline/thread execution does not call it. Fork catalog, kernels, logging, environment, progress and resources stay inherited; no spawn preparation, cancellation-specific branch, timeout change or extra status authority is introduced.

Closes #140.

Validation completed: 38 existing lane/resource controls pass, including a real fork/SIGTERM regression that checks child defaults, actual signal exit, no parent cleanup invocation and unchanged parent handlers. Genuine ordinary diagnostics on clean 56c3da776 with installed ZMQRuntime0.4.1 establish original fork failure propagation, finite ten-second healthy work, real server-process loss and recovery from transient transport/malformed status observations. They are correctness diagnostics, not pipeline timing measurements.

The causal cancellation packet actual-cancel-ipc-witness-v4 retains exact original server/worker incarnations, inherited handler names and both IPC inodes before cancellation; both paths disappear afterward while the original server is alive. Original waiter and terminal progress agree on CANCELLED. Earlier observer/protocol-scope mistakes and negative packets remain retained. Actual candidate cancellation and same-server recovery are qualified below.

Historical investigation and current exact evidence are in docs/plans/issue140_execution_failure_receipt_20260929.md; private artifacts are under /home/ts/.local/state/openhcs-maintenance/20261006/issue140-fork-acceptance-preparation-v1. The original failure's lifecycle/finalization work is already merged in ZMQRuntime21/OpenHCS697, and malformed status accounting in ZMQRuntime6. This change repairs the separately reproduced inherited-signal defect; it does not add another FAILED path or claim to reconstruct the original reported hang's exact cause.

Actual candidate qualification: normal native extension construction passed in the owned worktree, without copied extensions or installed-environment changes. On clean source b0cf6e9cb, public cancellation kills two genuine fork workers; the original waiter, terminal progress and a fresh attached status client agree on CANCELLED. Both IPC addresses retain their original inodes while the original server remains alive. A new ordinary compile request then completes on the same server, and the exact newly returned artifact ID feeds a new execution which completes finite ten-second work in two new fork children. All four children have SIGTERM/SIGINT defaults; all five owned process incarnations exit after normal disconnect and both runtime leases are free.

The actual v3 helper remains terminal 1 because its private observer passed a frozen nested progress mapping to a wire-dictionary parser after saving the real COMPLETE outcomes. Separate offline acceptance passes using existing ExecutionProgressObservation.terminal_status_response(), checking exact IDs/statuses/PIDs, both IPC inodes, child defaults, ten-second healthy work, the original failed cancellation CSV, and cleanup. Every original packet SHA remains unchanged; no terminal status was fabricated and no additional runtime was needed. Earlier negative fixture attempts (consumed artifact reuse and compile-only plus artifact ID) are preserved explicitly. The concrete endpoint defect is closed; the original reported hang's exact historical cause remains unclaimed. This is correctness acceptance, not performance evidence.

@trissim trissim added documentation Improvements or additions to documentation bug Something isn't working investigation labels Sep 29, 2026
@trissim

trissim commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator Author

Dependency review: a second reproducible unbounded-wait path belongs in the investigation

Reviewed plan head ef3c05ca8563234f9f9102c2b0a1d0188d622330 and pinned ZMQRuntime 1d3b32f4fcead23d5d2079c975eb2ae7646a1478.

The plan correctly avoids inventing a missing FAILED branch. There is another concrete failure mode in the dependency beyond the already-noted alive-process exemption: ExecutionWaiter.wait, lines 53–59 resets consecutive_errors = 0 before ExecutionStatusSnapshot.from_dict(status_response) validates the response. If the socket returns but every status payload fails schema decoding, each iteration is counted as error 1. max_consecutive_errors never takes effect, even when there is no known-server-process callback and no advancing progress.

A provider-free reproduction using the actual ExecutionWaiter class body, a poller returning malformed responses, a decoding stub raising ValueError, no liveness callback, and no sleeps exceeded seven polls with max_consecutive_errors=3; every logged attempt was 1/3. No service or live execution was started. This isolates control-flow accounting; it does not establish that malformed payloads caused the originally reported #140 job hang.

Proper ownership: let the declared status snapshot boundary determine whether a usable status observation succeeded, and reset the waiter error state only after that decode/admission. Keep terminal execution state on the existing lifecycle/record owner; a successful socket receive or a living PID is not successful observation of a job. Avoid a benchmark-local timeout/status cache as the repair (BOUND-1/2, identity of transport/process versus job observation).

Add this malformed-status case to the dependency-focused failure probes before the end-to-end injected worker failure. Preserve the separate healthy-long-job, cancellation and lost-process cases. The eventual implementation belongs with ZMQRuntime's waiter policy, followed by the OpenHCS integration test and validated gitlink, not another polling algorithm in OpenHCS.

@trissim

trissim commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator Author

Assigned waiter finding implemented in paired dependency draft

Named owner: H003/Codex waiter worker. Paired ZMQRuntime PR6, head 945e80248f375e834f1b993ed133268445e5efce, base pinned main 1d3b32f4fcead23d5d2079c975eb2ae7646a1478. Worktree: /home/ts/wt/zmqruntime-issue140-waiter-20260929.

Production change is confined to ExecutionWaiter.wait: decode with the existing ExecutionStatusSnapshot.from_dict boundary before resetting consecutive errors and progress baseline. No second OpenHCS polling cache/timeout or new schema carrier. Terminal lifecycle/record ownership is unchanged.

Real-decoder bounded provider-free no-sleep tests reproduce the base failure: 4 failed/10 passed, malformed polls repeatedly logged attempt1/3. Repair:14 passed. With five existing waiter/client controls:19 passed,52 deselected (latest0.18s). Healthy long job, progressing retries, known-live timeout tolerance, cancellation, failure, server error and lost-process diagnostics remain covered. Receipt: docs/validation/issue140_waiter_admission_20260929.md in the dependency PR.

This fixes the additional reviewed accounting path, not a proven cause of the originally reported issue140 hang; do not close issue140 solely on these tests. Draft/source-validated only, not merged/installed/live verified. CI was queued at exact pushed head; no CI result is claimed.

Coordination: parent owns #159 ViewerStateManager reuse/admission/acquisition and shared gitlink acquisition, plus separate #218 published metadata write policy; these are disjoint from this wait_policy change. Existing PR160 remains the OpenHCS integration umbrella; no competing OpenHCS implementation was created. No frozen installed submodule, environment or scientific artifact was modified. Actual ordinary-ZMQ/OpenHCS injected-failure integration is deferred under Confucius's runtime lock until parent schedules the slot.

@trissim trissim changed the title Draft: trace ZMQ execution failure terminality (#140) Reset child signal ownership to preserve parent ZMQ endpoints Oct 6, 2026
@trissim
trissim marked this pull request as ready for review October 6, 2026 14:22
Copilot AI balanced review requested due to automatic review settings October 6, 2026 14:22

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@trissim
trissim merged commit 21ceb8e into main Oct 6, 2026
2 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working documentation Improvements or additions to documentation investigation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Propagate server execution failures to ZMQ completion waiters

2 participants