Skip to content

fix(accel): an accelerator answers through a GPU lease, and a shared tool falls back to the next device - #852

Merged
dmmdea merged 2 commits into
mainfrom
fix/accel-independent-and-fallback
Nov 11, 2026
Merged

dmmdea merged 2 commits into
mainfrom
fix/accel-independent-and-fallback

Conversation

@dmmdea

@dmmdea dmmdea commented Nov 11, 2026

Copy link
Copy Markdown
Owner

Intent

An accelerator is an independent device: it must answer while the node's GPUs are reserved, and a tool that several devices serve must not defer while one of them is reachable.

Scope

Two rules change, both for the accel lane only.

  • Left out: every other task type keeps its lease reading (an agent contract by its seats, the rest by the whole node).
  • Left out: no "owner is down" state, no health pre-routing, no per-call device argument.
  • Left out: the sidecars, accelremote and pipeline.runAccelTask are untouched, so a forwarded job still cannot forward again.

What changed

  • Lease. fleetnode.textLeaseAgainst never refuses the accel task type. An NPU or TPU is not a card a lease names.
  • Fallback. The owner of a shared accelerator tool is tried first. When it cannot take the call (a defer, or a sidecar refusal about the device), the same tool runs on the next device that serves the name, through that device's own tool row and with only the arguments its schema declares. The answer carries fallback{from,reason,served_by}; when none answers, the owner's defer carries fallback_tried. A refusal of the caller's input is the answer and is not retried.
  • Embeddings. offload_image_embed and offload_text_embed move to another device only with any_space:true, because that vector is in another space. Without it the defer carries fallback_withheld.
  • Both surfaces, one function. agent.AccelFallback, called from mcpserver.handleAccelChain and agent.accelLaneTools.
  • What a caller sees. A FALLBACK: sentence in the description of a tool that has a fallback device (it names no device, and the compact tools profile keeps it), and fallback_for in the device's status row. A box with one device, or no shared name, advertises exactly what it did.
  • Docs. ADR 0102, which supersedes ADR 0068's "nothing falls back at call time"; docs/systems/accelerators.md, docs/systems/gpu-lease.md, the operator guide's lease paragraph.

The real run and its output

Both defects were seen live on 2026-11-11: 22 dispatch refusals reading node leased (gpu lease class=text ...) against an idle NPU, and four tools deferring for hours while two other devices that serve them were reachable.

Local gate on this commit:

go build ./...                      ok
go vet ./...                        ok
ok  internal/mcpserver   98.641s
ok  internal/agent       71.951s
ok  internal/fleetnode   42.808s
ok  internal/accelremote  1.300s

Two tests, one per incident:

  • TestATextLeaseOfTheGPUsNeverRefusesAnAccelJob: an accel dispatch under a text lease is admitted 202, and an image job on the same node is still refused 503.
  • TestASharedToolRunsOnTheNextDeviceWhenItsOwnerCannot: through the MCP surface, a tool whose owner has no node to reach runs on the next device with the owner named and only that device's arguments; an embedding stays put without any_space; any_space never reaches a sidecar.

The compact-profile gate caught one real break while this was written (the fallback note glued to the forwarding note, so compact dropped both). The note is now its own fixed sentence and is on the keep list; the gate surfaces list the fourth device.

The live run (an NPU call under a held lease, and a call with the owner's node unreachable) follows the deploy and is recorded on the release.

Blast radius

  • The fleet node's dispatch gate, for one task type.
  • Tool registration and the call path of every shared accelerator tool on the MCP server and in the agent loop. A tool with no second device takes the old handler unchanged.
  • A fallback sends the caller's image to another node of delegate_remotes than the owner's, over the same tailnet-only transport.

Risk

Medium. Tier verdict: review (blast 2.07/3, consequential 0.23), so one security pass ran on the diff; its result is in the first comment.

🤖 Generated with Claude Code

dmmdea and others added 2 commits November 11, 2026 14:24
…tool falls back to the next device

A text lease on a node's GPUs refused every forwarded accelerator call, although an accel
job runs on an NPU or TPU and never on a leased card. And a shared accelerator tool had one
owner and no second choice, so with the owner's node off or refusing it deferred while other
devices that serve the same name were reachable.

- fleetnode.textLeaseAgainst never refuses the accel task type.
- The owner of a shared tool is tried first; when it cannot take the call the next device
  that serves the name answers through its own tool row, and the result carries
  fallback{from,reason,served_by}. One function for both surfaces (agent.AccelFallback).
- An embedding moves to another device only with any_space:true.
- ADR 0102 (supersedes one consequence of ADR 0068); status gains fallback_for.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… sidecar guards its own writes

The security pass on the fallback found that it could hand out_path to a device whose sidecar
checks nothing: the Coral's semantic_segment overwrote any path it was given, the caller's own
image included. The fallback now applies the lane's rule before any device runs, the Coral
sidecar applies the same rule at the sink and writes through a temporary file, and a fallback
device that answers JSON null is a defer instead of a nil-map write.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@dmmdea

dmmdea commented Nov 11, 2026

Copy link
Copy Markdown
Owner Author

Security pass (one, on the first commit) and what was done

MEDIUM, fixed in the second commit. A fallback could hand out_path to a device whose sidecar checks nothing. The Coral's semantic_segment wrote to any path it was given, so with the owner unable to serve, out_path equal to the input image (or any existing file PIL can write) would have been overwritten. The owner's path was guarded before (the NPU card's sidecar, or the caller-side landing of a fleet device), so the fallback opened it.

  • The fallback applies the lane's rule before any device runs: out_path is never the input image, its directory must exist, an existing target is replaced only when it is a regular PNG file. Otherwise the defer carries fallback_withheld.
  • The Coral sidecar applies the same rule at the sink and writes through a temporary file. That also closes the same hole on a box where the Coral is the owner, which predates this change.

LOW, fixed in the second commit. A fallback device answering JSON null reached a write into a nil map. It is now a defer.

Checked and clean. Destinations (only delegate_remotes nodes that list the device, over the tailnet-only transport), argument leakage (any_space and the owner's extra arguments never reach another device), loops (a node runs local lanes only), the lease change (an accel job can reach no GPU lane), secrets (reasons reuse the scrubbed errors).

Not checked. The Hailo-8L sidecar is not in this repository. As a fallback device it is covered by the rule above; as an owner its own out_path handling was not read.

The test TestASharedToolRunsOnTheNextDeviceWhenItsOwnerCannot gained the out_path case: the call is withheld, the next device is not called, the input image is unchanged.

@dmmdea
dmmdea merged commit ddf78e1 into main Nov 11, 2026
6 checks passed
@dmmdea
dmmdea deleted the fix/accel-independent-and-fallback branch November 11, 2026 19:42
@dmmdea dmmdea mentioned this pull request Nov 11, 2026
dmmdea added a commit that referenced this pull request Nov 11, 2026
One change, merged to main through its own pull request (#852) after one
security pass and green CI:

- A text lease on a node's GPUs no longer refuses an accel job: an NPU
  or TPU is not a card a lease names.
- The owner of a shared accelerator tool is tried first, and when it
  cannot take the call the next device that serves the name answers
  through its own tool row; the result carries
  fallback{from,reason,served_by}. An embedding moves to another device
  only with any_space:true.
- out_path is checked before a fallback runs, and the Coral sidecar
  guards its own writes (the two findings of the security pass).
- ADR 0102, which supersedes one consequence of ADR 0068.

Deletes: changes/unreleased/fix-accel-independent-and-fallback.md

Co-authored-by: dmmdea <dmmdea@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant