Skip to content

Stop a broken stdin pipe from terminating the server - #212

Merged
powerfooI merged 1 commit into
powerfooI:mainfrom
crandrosoff:fix-stdin-epipe-crash
Sep 19, 2026
Merged

powerfooI merged 1 commit into
powerfooI:mainfrom
crandrosoff:fix-stdin-epipe-crash

Conversation

@crandrosoff

@crandrosoff crandrosoff commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

The bug

An unhandled EPIPE terminates the whole server process. Every connected browser
session drops at once. Under systemd the service restarts about two seconds later,
so it reads as a flaky connection rather than a crash, and it is invisible unless
you read the journal.

Measured on one 0.7.4 host over five days: 52 such exits, up to 23 in a single
day
.

EPIPE: broken pipe, write
 syscall: "write",
   errno: -32,
    code: "EPIPE"
      at jo (/$bunfs/root/roamgate-linux-x64:260:55327)
      at _g (/$bunfs/root/roamgate-linux-x64:498:5507)
      at f  (/$bunfs/root/roamgate-linux-x64:498:7824)
      at async qB (/$bunfs/root/roamgate-linux-x64:4152:11970)
Bun v1.4.1 (Linux x64)

It is not historical. It recurred on v0.7.5 at 2026-09-18 14:38:09 MDT on the same
host, while this pull request was open:

EPIPE: broken pipe, write
 syscall: "write",
   errno: -32,
    code: "EPIPE"
      at zo (/$bunfs/root/roamgate-linux-x64:260:66649)
      at pf (/$bunfs/root/roamgate-linux-x64:498:5507)
      at f  (/$bunfs/root/roamgate-linux-x64:498:7824)
      at async wC (/$bunfs/root/roamgate-linux-x64:4184:11972)
Bun v1.4.1 (Linux x64)

Two caller frames are byte-identical to the v0.7.4 trace above, 498:5507 and
498:7824. Mapping zo against the shipped v0.7.5 binary lands on the same
Bun.spawn(..., stdin: "pipe") followed by r.stdin.write(t). Same defect, same path.

Mapping those offsets against the published roamgate-linux-x64 v0.7.4 binary gives
the exact chain:

frame function
jo runProcessWithInputTimeout (server/src/workspace/process.ts)
_g collectIgnoredNames (server/src/workspace/git-ignore.ts)
f listWorkspaceFiles (server/src/workspace/files.ts)
qB the bridge RPC dispatcher handling file.list

Root cause

runProcessWithInputTimeout writes the payload to the child's stdin and never
awaits the result:

proc.stdin.write(input);
proc.stdin.end();

Bun's FileSink.write() and .end() both return number | Promise<number>, and
return a promise whenever the payload cannot be flushed synchronously.

git check-ignore --stdin outside a work tree exits 128 immediately and never
reads stdin. collectIgnoredNames has already handled that exit code, but the
payload is still in flight. The pending flush rejects with EPIPE, nothing holds
that promise, and Bun terminates the process.

Why it fires in practice

collectIgnoredNames runs on list.entries, which is already capped at
LIST_LIMIT = 1000. Payload size therefore depends on entry-name length, and the
failure is a race between the flush and the child's exit. Measured, 40 runs each:

payload names EPIPE
181 KB 1000 x 180 chars 1 / 40
245 KB 1000 x 245 chars 40 / 40

That matches the production frequency: roughly one crash an hour on a host that
browses many non-repository directories, not one per listing.

End-to-end verification

Not just a unit test. Both binaries were compiled with bun run build on Bun 1.4.1,
started as real servers against a real Herdr 0.9.1, and driven through the real
websocket bridge with a genuine file.list RPC on a non-repository directory of
1100 files.

build RPC result server after
main (unfixed) replied, 1000 entries, no error died, exit 1
this branch replied, 1000 entries, no error alive

The unfixed server printed exactly the production error before exiting:

EPIPE: broken pipe, write
 syscall: "write",
   errno: -32,
    code: "EPIPE"
Bun v1.4.1 (Linux x64)

Note that the RPC itself succeeds in both cases. The request returns a correct
listing and then the process dies, which is why this looks like a flaky connection
rather than a failing request.

Why the two obvious fixes do not work

Both were measured, not assumed.

  1. The caller's existing try/catch cannot catch it. collectIgnoredNames
    already wraps its await in try { ... } catch { return new Set(); }. That wraps
    the awaited result, while the rejection comes from a promise the caller never
    receives. In the reproduction the caller sees a clean code: 128 and the
    process still dies.
  2. Wrapping the write in a local try/catch does not work either, for the same
    reason. Measured: still escaped=EPIPE.

Only attaching a handler to the returned value suppresses it.

The fix

Absorb the broken pipe where it originates. The child's exit code and stderr still
describe the failure, so every caller keeps behaving exactly as before, including the
existing code === 128 branch in collectIgnoredNames.

Validation

Run with Bun 1.4.1, matching CI.

check result
bun run format:check exit 0
bun run lint exit 0
bun run typecheck exit 0
bun run build:site exit 0
new test on main fails, with the production error at process.ts:42
new test on this branch passes
bun test zero new failures against a baseline recorded on main

The suite is not fully green in my environment either before or after: a handful of
SSH, port-binding and process-control tests fail for environmental reasons. I compared
failing test names before and after, and the set after this change is a strict
subset of the baseline. No test fails that did not already fail.

A separate finding, deliberately not fixed here

On the same host, v0.7.5 logs a large volume of terminal.link.resolve errors that
v0.7.4 did not: 1708 in 76 minutes, against 0 in the preceding 24 hours. 1050 of
them are this endpoint is still processing another command. Over the same window the
closing slow websocket backpressure disconnects ran at a higher rate than before the
upgrade. The sample is short and I am not claiming a regression, only reporting it.
Happy to open a separate issue with the full numbers if that is useful.

Update: the backpressure half of that observation is now #213, against the same
c0014ce Release 0.7.5 base. It replaces the 8 MB disconnect with per-terminal frame
coalescing. Neither branch contains the other and they merge cleanly. The
terminal.link.resolve error volume is untouched by both and remains unexplained.

🤖 Generated with Claude Code

runProcessWithInputTimeout writes the caller's payload to the child's stdin
and never awaits the result. Bun's FileSink returns number | Promise<number>,
and returns a promise whenever the payload cannot be flushed synchronously.

A child can exit before it reads stdin. git check-ignore --stdin outside a
repository exits 128 immediately, which is the path taken by file.list on any
non-repository directory. The pending flush then rejects with EPIPE. Nothing
holds that promise, so it surfaces as an unhandled rejection and terminates
the whole server process, dropping every connected browser session at once.

The existing try/catch in the caller cannot help: it wraps the awaited result,
while the rejection comes from a promise the caller never receives. Wrapping
the write in a local try/catch does not help either, for the same reason.

Absorb the broken pipe where it originates instead. The child's exit code and
stderr still describe the failure, so the callers keep behaving as before.

Observed in production on 0.7.4: 52 such exits over five days, up to 23 in a
single day, each one restarting the service and disconnecting every client.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot added the enhancement New feature or request label Sep 18, 2026
@crandrosoff
crandrosoff marked this pull request as draft September 18, 2026 19:51
@crandrosoff
crandrosoff marked this pull request as ready for review September 18, 2026 20:02
@crandrosoff
crandrosoff marked this pull request as draft September 18, 2026 22:46
@powerfooI
powerfooI marked this pull request as ready for review September 19, 2026 00:49
@powerfooI

Copy link
Copy Markdown
Owner

Thanks for tracking this down and providing the fix and reproduction details! I’ll take it from here and handle the remaining checks and merge.

@powerfooI
powerfooI merged commit b04030d into powerfooI:main Sep 19, 2026
7 of 8 checks passed
@crandrosoff

Copy link
Copy Markdown
Contributor Author

No problem! Keep up the good work. Love the tool!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants