Skip to content

Built-in hardware reservations and agent job log - #88

Merged
data-angel merged 2 commits into
mainfrom
fleet-reservations
Oct 17, 2026
Merged

data-angel merged 2 commits into
mainfrom
fleet-reservations

Conversation

@data-angel

Copy link
Copy Markdown
Contributor

Replaces the sideband pair lock (pair.sh / pair-mac.sh) with a lease that LLooM itself manages, and adds a job log, so people stop asking agents "how's it going".

Reservations. These are expiring leases on resources listed in config.reservations.resources.

  • Take runs the resource's actions: route aliases away, then drain and suspend models. Each action runs on the local gateway or on another node through its admin API.
  • Release undoes them in reverse. It only reports success once the resumed models pass their health check.
  • A failed restore leaves the lease stuck, and so does a restart mid-action. The resource stays held until a release succeeds.
  • Each step records what it found, so release restores exactly that.
  • Queues, renewal, warnings, auto-restore on expiry, heartbeat-extended leases and indefinite leases are all supported.
  • Expiry and promotion pause during deployment fences.

Jobs. Agents post notes, progress, metrics and receipts, and every update is a heartbeat. Jobs that go quiet are marked stalled, and after 24 hours abandoned. Boards follow an event feed at /gateway/schedule?since=.

CLI. Agents use lloom schedule | reserve | renew | release | job. Set LLOOM_HEAD_URL and LLOOM_HEAD_API_KEY to target the head; the local key is never sent there. See docs/reservations.md.

The Tessera side is Enntity/tessera#2: a Fleet band, with jobs as tiles.

Verification

  • New tests: test/schedule.test.mjs (15) and test/schedule-integration.test.mjs (5). They cover HTTP, a remote node owner gateway, the CLI end to end, --wait, and redaction of an echoed key.
  • npm run check:js, test:cluster, test:unit, lint and format:check all pass.
  • An independent review pass found route restore when nothing was active, resuming models that were already suspended, --wait exit codes, boot on a bad state file, update atomicity, bounds, redaction, and fence interaction. All are fixed and have tests.
  • Not deployed. Rolling this out on the head (03) needs a gateway restart, a pair resource config, and new pair-mac.sh shims. That waits for your go-ahead.

🤖 Generated with Claude Code

data-angel and others added 2 commits October 17, 2026 14:59
Agents shared the Spark pair through a sideband mkdir lock (pair.sh) that
suspended LLooM and rerouted aliases in separate steps, and the only way to
learn how a job was going was to ask the agent. The schedule-owning gateway
(normally the head) now does both.

- Reservations are expiring leases on configured resources. Taking one
  runs the resource's actions (route aliases away, then drain and suspend
  models), locally or on another cluster node through its admin API.
  Release undoes them in reverse and reports success only after the resumed
  models pass their health check, so traffic never goes back to a suspended
  or unhealthy model. A failed restore, or a restart mid-action, leaves the
  lease `stuck` and the resource held until a release succeeds.
- Each step records what it found: a route already on its target and a model
  already suspended are left as they were, and a route with no active profile
  is refused. Resources that control the same model or route are config
  errors.
- Queues with renewal, TTLs with warnings and auto-restore on expiry, leases
  extended by their holder's job heartbeats, and indefinite leases where a
  resource allows them. Expiry and promotion pause during a deployment fence.
- Jobs: notes, progress, metrics, receipts and heartbeats, with
  stalled/abandoned detection, an event feed (`/gateway/schedule?since=`) for
  boards such as Tessera, and a rotated NDJSON history.
- CLI: `lloom schedule`, `reserve`, `renew`, `release`, `job`. They target a
  remote head with LLOOM_HEAD_URL/LLOOM_HEAD_API_KEY and never send the local
  key there. `reserve --wait` exits 0 only once active, rides out head
  blips, and gives its place back on Ctrl-C.
- Schedule refusals reach the CLI as redacted, bounded messages, including
  redaction of the key just sent. An unreadable schedule.json disables only
  the schedule routes, never inference.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Ships in the npm package (skills/) beside docs/reservations.md.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@data-angel
data-angel merged commit c6fbe2b into main Oct 17, 2026
2 checks passed
@data-angel
data-angel deleted the fleet-reservations branch October 17, 2026 22:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant