fix(deploy): keep daemon alive and surface diagnostics when ESP32-C6 esptool flash aborts

[zackees]

Summary

On Windows, an ESP32-C6 full-flash deploy reached the external esptool subprocess and then caused the fbuild daemon to disappear without returning actionable diagnostics to the client. The board and COM port remained healthy, and the next fbuild daemon status started a new daemon PID.

This blocks FastLED hardware-in-the-loop validation because a failed flash is indistinguishable at the caller from a dropped daemon connection, and the device must be retried without knowing whether the child exited, timed out, or triggered a daemon crash.

Reproduction

Environment:

• Windows, fbuild 2.5.18
• ESP32-C6 on COM9 (USB VID:PID 303A:1001)
• FastLED AutoResearch deploy command:

bash autoresearch esp32c6 --parlio --lanes 1 --strip-sizes 100 --tight-timing --upload-port COM9 --timeout 120s --skip-lint

Observed daemon log sequence:

build completed in 328.7s
Flash: 2,469,437 / 4,194,304 bytes
RAM: 73,936 / 327,680 bytes
verified device differs
flashing .../firmware.bin to COM9 via esptool (esp32c6)

The log ends immediately after the final line. The client eventually exited 1 with no stdout explaining the flash result. COM9 remained present and healthy. A subsequent fbuild daemon status started a new daemon (PID 54588), confirming that the daemon handling the deploy was no longer alive.

Expected behavior

An external flash-tool exit, timeout, cancellation, or I/O failure must remain contained to the deploy operation:

• the daemon stays alive;
• /api/deploy returns a structured non-success response;
• the response and daemon log preserve the child exit status and bounded stdout/stderr, or clearly identify a timeout/cancellation;
• the serial port is released so a retry can proceed.

Current code path

On current main (fdd824131fd1756c2d960ee3397b251ad1f12707):

• crates/fbuild-deploy/src/esp32/deployer.rs logs flashing ... via esptool and calls run_command(..., Some(Duration::from_secs(120))) for a full flash.
• crates/fbuild-core/src/subprocess.rs starts a contained Tokio child and wait_and_capture should convert completion or timeout into a ToolOutput/FbuildError.
• crates/fbuild-cli/src/daemon_client.rs gives deploy a 30-minute HTTP budget.
• crates/fbuild-daemon/src/handlers/operations/deploy.rs owns the daemon-side deploy operation.

The observed daemon disappearance violates that containment contract. The cutoff does not establish whether the initiating fault is subprocess/job containment, a panic, process termination, or daemon supervision; preserving the terminal child/handler diagnostic is part of the required fix.

Related issues (not duplicates)

• Orphaned mkdir install-locks deadlock after daemon crash; stale daemon endpoint survives 'daemon stop'; esptool provisioning failure kills daemon #1213 fixed orphaned install locks, stale endpoints, and an esptool provisioning failure in fbuild 2.5.2. This run completed provisioning/build/verify and failed only after entering full flash.
• audit: blocking operations with no timeout in fbuild-deploy (sub-issue of #802) #804/fix(timeout): close #802 audit family — workspace-wide timeouts on blocking ops #843 added wall-clock containment to native espflash paths. The failing C6 run used the already-bounded external esptool path.

Acceptance criteria

1. Add a focused RED integration test that drives the external ESP flash child through abnormal exit/termination (and, if needed, handler cancellation) and reproduces the dropped response or daemon death.
2. Make the test GREEN: the daemon remains alive and accepts a follow-up status/request.
3. /api/deploy returns a structured actionable failure containing the child exit status plus bounded stdout/stderr, or an explicit 120-second timeout/cancellation reason.
4. The child/process-tree and COM port are released after every failure path.
5. Add or extend an ignored ESP32-C6 HIL full-flash test. On COM9, record the daemon PID before and after the exact deploy; it must remain stable, and success/failure must be explicit at the client.
6. Verify the original FastLED AutoResearch command can retry over the same USB port without replugging the board.

Decisions

• Repository: FastLED/fbuild, because the daemon lifecycle and deploy response fail below FastLED's AutoResearch wrapper.
• Priority: priority: p1; this blocks reliable HIL work but has a reproducible retry path and no evidence of data loss.
• Scope: external ESP esptool full-flash containment and daemon diagnostics. AutoResearch stack sizing is separate.
• Deploy path: retain the sanctioned fbuild deploy pipeline; do not work around it with raw esptool invocations.
• Proof standard: require both a deterministic integration regression and a real C6 full-flash run, because the current unit/HIL coverage verifies C6 images but does not exercise daemon survival across full flash.

Open diagnostic question

What terminates the daemon after the child starts: Windows contained-process/job behavior, a panic or abort in the deploy task, daemon supervision/restart, or another process-level failure? The fix should capture enough terminal state to answer this on the first failing run.

[zackees]

Deferred after the initial two-pass investigation because this is not a bounded source-only fix: the daemon disappears below an already-bounded subprocess path, the initiating process-level failure is not yet captured, and acceptance requires a Windows integration harness plus repeatable ESP32-C6 full-flash HIL evidence.

Restart when all of the following are available in one focused run:

1. Enable daemon crash/panic/process-exit capture (Windows Event Viewer or minidump plus the terminal daemon log) around the C6 full-flash call.
2. Reproduce with a deterministic fake external flash child inside the daemon integration suite, separating child exit, timeout, cancellation, and forced termination cases.
3. Identify the process boundary that kills the daemon and land the daemon-survival/structured-response regression before touching hardware behavior.
4. Validate on COM9 with the same daemon PID before and after both a forced failure and a successful full flash; then rerun the exact FastLED AutoResearch command without a USB replug.

This label is a burn-down disposition, not a claim that the defect is fixed.