Repository navigation
fix(accel): an accelerator answers through a GPU lease, and a shared tool falls back to the next device - #852
Conversation
…tool falls back to the next device
A text lease on a node's GPUs refused every forwarded accelerator call, although an accel
job runs on an NPU or TPU and never on a leased card. And a shared accelerator tool had one
owner and no second choice, so with the owner's node off or refusing it deferred while other
devices that serve the same name were reachable.
- fleetnode.textLeaseAgainst never refuses the accel task type.
- The owner of a shared tool is tried first; when it cannot take the call the next device
that serves the name answers through its own tool row, and the result carries
fallback{from,reason,served_by}. One function for both surfaces (agent.AccelFallback).
- An embedding moves to another device only with any_space:true.
- ADR 0102 (supersedes one consequence of ADR 0068); status gains fallback_for.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… sidecar guards its own writes The security pass on the fallback found that it could hand out_path to a device whose sidecar checks nothing: the Coral's semantic_segment overwrote any path it was given, the caller's own image included. The fallback now applies the lane's rule before any device runs, the Coral sidecar applies the same rule at the sink and writes through a temporary file, and a fallback device that answers JSON null is a defer instead of a nil-map write. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Security pass (one, on the first commit) and what was doneMEDIUM, fixed in the second commit. A fallback could hand
LOW, fixed in the second commit. A fallback device answering JSON Checked and clean. Destinations (only Not checked. The Hailo-8L sidecar is not in this repository. As a fallback device it is covered by the rule above; as an owner its own The test |
One change, merged to main through its own pull request (#852) after one security pass and green CI: - A text lease on a node's GPUs no longer refuses an accel job: an NPU or TPU is not a card a lease names. - The owner of a shared accelerator tool is tried first, and when it cannot take the call the next device that serves the name answers through its own tool row; the result carries fallback{from,reason,served_by}. An embedding moves to another device only with any_space:true. - out_path is checked before a fallback runs, and the Coral sidecar guards its own writes (the two findings of the security pass). - ADR 0102, which supersedes one consequence of ADR 0068. Deletes: changes/unreleased/fix-accel-independent-and-fallback.md Co-authored-by: dmmdea <dmmdea@users.noreply.github.com>
Intent
An accelerator is an independent device: it must answer while the node's GPUs are reserved, and a tool that several devices serve must not defer while one of them is reachable.
Scope
Two rules change, both for the
accellane only.accelremoteandpipeline.runAccelTaskare untouched, so a forwarded job still cannot forward again.What changed
fleetnode.textLeaseAgainstnever refuses theacceltask type. An NPU or TPU is not a card a lease names.fallback{from,reason,served_by}; when none answers, the owner's defer carriesfallback_tried. A refusal of the caller's input is the answer and is not retried.offload_image_embedandoffload_text_embedmove to another device only withany_space:true, because that vector is in another space. Without it the defer carriesfallback_withheld.agent.AccelFallback, called frommcpserver.handleAccelChainandagent.accelLaneTools.FALLBACK:sentence in the description of a tool that has a fallback device (it names no device, and the compact tools profile keeps it), andfallback_forin the device's status row. A box with one device, or no shared name, advertises exactly what it did.docs/systems/accelerators.md,docs/systems/gpu-lease.md, the operator guide's lease paragraph.The real run and its output
Both defects were seen live on 2026-11-11: 22 dispatch refusals reading
node leased (gpu lease class=text ...)against an idle NPU, and four tools deferring for hours while two other devices that serve them were reachable.Local gate on this commit:
Two tests, one per incident:
TestATextLeaseOfTheGPUsNeverRefusesAnAccelJob: anacceldispatch under a text lease is admitted 202, and an image job on the same node is still refused 503.TestASharedToolRunsOnTheNextDeviceWhenItsOwnerCannot: through the MCP surface, a tool whose owner has no node to reach runs on the next device with the owner named and only that device's arguments; an embedding stays put withoutany_space;any_spacenever reaches a sidecar.The compact-profile gate caught one real break while this was written (the fallback note glued to the forwarding note, so compact dropped both). The note is now its own fixed sentence and is on the keep list; the gate surfaces list the fourth device.
The live run (an NPU call under a held lease, and a call with the owner's node unreachable) follows the deploy and is recorded on the release.
Blast radius
delegate_remotesthan the owner's, over the same tailnet-only transport.Risk
Medium. Tier verdict:
review(blast 2.07/3, consequential 0.23), so one security pass ran on the diff; its result is in the first comment.🤖 Generated with Claude Code