Repository navigation
ci: vet the end-to-end and live test files - #86
Merged
Merged
Conversation
`test/e2e` and the adapters' live tests sit behind the `e2e` and `live` build tags. No CI job runs them, and none compiled them either, so a change could break them without anything failing until someone next ran them by hand. `make vet` now also vets with `-tags "e2e live"`, the experimental pass adds the same tags, and the `go` CI job calls `make vet`. This compiles and vets those tests; it does not run them. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Signed-off-by: Jalil <reachjalil@gmail.com>
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
4 tasks done
reachjalil
added a commit
that referenced
this pull request
Nov 11, 2026
* feat: add the control plane, policy engine, scheduler and stores
differently-server exposes the v1 API with scoped API keys, organizations
and projects, atomic quota admission and activation, fair queueing and
policy-driven placement that keeps external providers fail-closed. The
scheduler provisions idempotently, retries safely and reconciles orphans
and lost instances. State lives in Postgres (memory store for dev), with
a tamper-evident audit chain, a trust report, usage estimates, Prometheus
metrics and a built-in dashboard. Adapters cover Firecracker pools,
Daytona, Freestyle, remote plugins and a dev-only host adapter.
Signed-off-by: Jalil <reachjalil@gmail.com>
* feat: add the Firecracker node agent and in-guest agent
differently-agent runs Firecracker microVMs on KVM nodes, isolates each
VM on its own tap device and firewall chain, reaches the guest over
vsock, and serves the adapter protocol to the control plane. It supports
stop/start from the retained disk, pause/resume, crash-consistent disk
snapshots and recovery after restarts.
Signed-off-by: Jalil <reachjalil@gmail.com>
* feat: add the differently CLI and Go client
Signed-off-by: Jalil <reachjalil@gmail.com>
* feat: add TypeScript and Python SDKs
Signed-off-by: Jalil <reachjalil@gmail.com>
* feat: add Helm chart, Terraform, systemd units and image builds
Signed-off-by: Jalil <reachjalil@gmail.com>
* docs: add architecture, decisions, guides and feature parity
Signed-off-by: Jalil <reachjalil@gmail.com>
* ci: add CI, release and dependency automation
Signed-off-by: Jalil <reachjalil@gmail.com>
* ci: add code scanning and back-merge main after releases
main stays the default branch and is protected by organization rules
(linear history, signed commits, code scanning, code quality). Releases
are squash-merged from pre-release, so a workflow merges main back into
pre-release afterwards to keep the next release PR conflict-free. CodeQL
provides the code scanning results main requires. Docs now tell
contributors and agents to target pre-release explicitly.
Signed-off-by: Jalil <reachjalil@gmail.com>
* feat(site): add Astro website and Starlight docs for GitHub Pages (#2)
* feat(site): add Astro website and Starlight docs for GitHub Pages
Landing page plus docs single-sourced from docs/, SDK READMEs,
api/openapi.yaml and the CLI help output via scripts/sync-docs.mjs.
Builds validate internal links. Deploys with actions/deploy-pages on
pushes to pre-release.
Signed-off-by: Jalil <reachjalil@gmail.com>
* feat(site): restyle in black and white with a new mark
Monochrome palette, Inter and JetBrains Mono, and a layered-square
mark for the abstraction over many backends. Applies to the landing
page and the Starlight docs theme.
Signed-off-by: Jalil <reachjalil@gmail.com>
* fix(site): escape backslashes in generated API table cells
Signed-off-by: Jalil <reachjalil@gmail.com>
---------
Signed-off-by: Jalil <reachjalil@gmail.com>
* fix(security): pin agent identity, verify VM PIDs, add fail-closed audit and private metrics (#1)
Each node agent now creates its own secret, pinned when its ID first
registers. Heartbeats for a known ID with another secret or pool are
refused and audited, the control plane calls every agent with that
agent's own secret, and DELETE /v1/admin/agents/{id} retires a rebuilt
node's identity. A node holding the shared registration token can no
longer impersonate or command another node.
The agent only signals a Firecracker PID after confirming from
/proc/<pid>/cmdline that it is the VM's own process, so a reused PID is
never killed.
audit.failClosed records each state-changing request before acting and
answers 503 when that fails; audit failures are counted in
differently_audit_append_failures_total in both modes.
metrics.listen serves /metrics on its own address. The Helm chart uses it
by default (port 9091) so metrics never sit behind the ingress, and the
trust report's limitations now reflect these settings.
Signed-off-by: Jalil <reachjalil@gmail.com>
* feat(site): lead the landing page with orchestration and resource management (#4)
Signed-off-by: Jalil <reachjalil@gmail.com>
* test(e2e): run the real stack on Firecracker microVMs and fix what it found (#3)
Add an end-to-end suite (make e2e) that runs Postgres, the server and a
node agent booting real Firecracker microVMs in Docker on any host with
/dev/kvm, including Colima with nested virtualization on Apple Silicon.
It covers boot resources, exec, files, network isolation, stop/start,
pause/resume, fork, quota queueing, agent restart recovery, cleanup,
audit, the trust report and the metrics listener.
Fixes found by running it:
- start refused every request after the first (handler state reused).
- A restarting microVM was hidden from the agent's list, so the
reconciler failed it mid-start; the reconciler now also waits 30s after
a change before correcting a sandbox.
- The reconciler now follows instance state: a microVM that stopped
behind the control plane's back makes its sandbox stopped.
- Stop on aarch64 syncs the guest before the VMM is killed.
- sandbox list -o json always prints an array.
- build-rootfs.sh works on macOS hosts (shared temp dir, no unpacking
into the host mount).
Signed-off-by: Jalil <reachjalil@gmail.com>
* feat(adapters): add an OpenSandbox backend, checked against a live server (#5)
OpenSandbox servers can now back a pool. The opensandbox adapter covers
create (idempotent via metadata), exec and streamed exec, files,
pause/resume, snapshots, restore and direct port endpoints through the
lifecycle API and execd, with the server API key as a credential
reference. Because OpenSandbox can be self-hosted, pools must declare
trust explicitly; placement, quota, audit and fail-closed external rules
are unchanged.
A spike recorded what OpenSandbox offers and what was observed. Running
the real 1.2.0 server (Docker runtime) on Colima caught four places where
the deployed execd differs from the published spec: argv is rejected,
upload metadata must be a file part, events are newline-delimited JSON,
and line terminators are stripped. The adapter handles each, and the fake
server in the tests reproduces them.
ADR 0019 adopts OpenSandbox as a backend now and sets the criteria for
making it the default runtime in place of the Firecracker node agent
(ADR 0002 stays in force until then). Feature parity, roadmap, landscape,
adapter spec, Helm schema and a pool guide are updated.
Signed-off-by: Jalil <reachjalil@gmail.com>
* refactor: run sandboxes on OpenSandbox instead of Differently's own node agent (#6)
Differently's value is orchestration and resource management: one API,
tenancy and scoped keys, quotas and fair queueing, policy-gated placement,
adapters, the audit chain and the trust report. Its own Firecracker node
agent duplicated what OpenSandbox already maintains, so OpenSandbox becomes
the default private runtime (ADR 0020).
- Remove differently-agent, differently-guest, the firecracker adapter,
agent registration, heartbeats and per-agent secrets, guest image scripts
and systemd units. Migration 0002 drops the agents table; the config
parser and Helm schema reject the removed settings.
- Helm chart 0.2.0 deploys the control plane only and registers an existing
OpenSandbox server through a new opensandbox values block.
- The trust report shows each pool's declared isolation and says isolation
and egress come from the runtime.
- The e2e suite now runs Postgres, the server and a real OpenSandbox server.
- Fix: two pools fronting the same backend no longer delete each other's
live sandboxes as orphans.
- Add round-trip tests for the HTTP plugin protocol, whose only coverage
was the removed agent's tests.
- Make the AWS Terraform KVM node group optional (Fast Sandbox only).
- Rewrite the docs, feature parity matrix and site for the new runtime.
Signed-off-by: Jalil <reachjalil@gmail.com>
* fix(cli): print every error instead of exiting silently (#7)
The CLI silences cobra's own error output and printed only API errors, so
an unknown flag, a missing argument or an unreachable server exited 1 with
no message. Print any other error as is.
Signed-off-by: Jalil <reachjalil@gmail.com>
* feat(site): add a features section comparing OpenSandbox, Differently and a single provider (#9)
Landing page gets a Features section: what OpenSandbox provides, what
Differently adds on top, and a comparison with using one provider. A new
docs Features page holds the full version, and the sidebar links the
OpenSandbox pools guide and spike.
Signed-off-by: Jalil <reachjalil@gmail.com>
* feat: serve OpenWork's sandbox contract, routing one image name to every provider (#10)
OpenWork's @openwork/sandbox contract is the most complete statement of
what a sandbox client needs. Close the gaps that kept Differently from
serving it, so OpenWork can hold one Differently key while Differently holds
the OpenSandbox, Daytona and Freestyle credentials and routes by policy.
- Templates map one name to each pool's own image (templates[].images), and
placement only considers the pools a template lists.
- An Idempotency-Key names a live sandbox: once it has ended, the key
creates a new one (migration 0003).
- opensandbox pools: lineEndings: kept gives byte-exact exec output with
execd release-1.2.0, and a TTL change moves OpenSandbox's own expiry.
- Add the OpenWork guide and backlog epic E22 for the remaining gaps.
Signed-off-by: Jalil <reachjalil@gmail.com>
* feat: manage provider keys and API keys in Differently, and switch providers (#11)
Operators can now add a Daytona, Freestyle, OpenSandbox or plugin pool with
its API key stored in Differently, switch which pool new sandboxes go to,
and rotate keys without downtime.
- Stored provider credentials: PUT /v1/admin/secrets/{name} seals a value
with AES-256-GCM under a key-encryption key that never enters the
database (secrets.keyEnv or keyFile); pools reference it as secret:name.
Values are write-only. Replacing one rotates it live (ADR 0021).
- Every replica re-reads the pool registry and each pool's credential on
every refresh, so pool changes and rotated credentials (stored, file
mounts such as Key Vault) reach all replicas without a restart.
- POST /v1/admin/pools/{name}/use sends new sandboxes to a pool and drains
the others; external pools need explicit approval.
- POST /v1/api-keys/{id}/rotate issues a replacement and keeps the old key
for a grace period; GET /v1/api-keys/{id}.
- CLI, dashboard, Go/TS/Python SDKs, Helm (generated secrets key,
runtimeProviders egress), docs and an e2e test on a real OpenSandbox.
Signed-off-by: Jalil <reachjalil@gmail.com>
* docs(openwork): note how OpenSandbox's Kubernetes runtime affects OpenWork computers (#12)
* docs(openwork): note how OpenSandbox's Kubernetes runtime affects OpenWork computers
Pause and snapshots need a registry on that runtime, port URLs need its
ingress gateway, and a first image pull can exceed its pod-ready timeout.
Observed on a k3s install on Azure.
Signed-off-by: Jalil <reachjalil@gmail.com>
* fix(opensandbox): adopt a sandbox OpenSandbox keeps creating after a 504
OpenSandbox's Kubernetes runtime answers a create with 504 when its 60 s
pod-ready wait runs out, typically during a first image pull, but keeps
creating the sandbox. Failing there excluded the pool for that sandbox,
which on a single-pool install left it nowhere to go. Find the sandbox by
its metadata and keep waiting instead.
Signed-off-by: Jalil <reachjalil@gmail.com>
* Revert "fix(opensandbox): adopt a sandbox OpenSandbox keeps creating after a 504"
This reverts commit 8c80ada1823eb778231d6cb1d820bfc25a11c4cd.
Signed-off-by: Jalil <reachjalil@gmail.com>
* docs(openwork): OpenSandbox deletes a sandbox after its pod-ready timeout
Observed on the Azure k3s install: the server cleans the sandbox up after
the 504, so a retry creates a new one.
Signed-off-by: Jalil <reachjalil@gmail.com>
---------
Signed-off-by: Jalil <reachjalil@gmail.com>
* chore(release): prepare v0.1.0-alpha.1 (#15)
Signed-off-by: Jalil <reachjalil@gmail.com>
* fix(opensandbox): handle what real runtimes return, replayed in go test (#13)
Running Differently against OpenSandbox's Kubernetes runtime (k3s on Azure)
and Docker runtime exposed behaviour the fake server did not:
- An image that cannot be pulled marked the shared pool unhealthy for every
tenant on each retry. The Docker runtime's IMAGE_PULL_FAILED now fails the
sandbox at once (invalid_request); the Kubernetes runtime's 504
POD_READY_TIMEOUT is retried as a new adapter error, sandbox_failed,
which never marks the pool unhealthy.
- Pause without a snapshot registry is accepted and never starts; it now
returns unsupported_operation after 30 s instead of provider_unavailable
after 3 minutes. Snapshots without a registry say so.
- A sandbox being deleted (Stopping/DELETING) is treated as gone.
- ports.expose is opt-in: direct endpoints are runtime-internal addresses.
- The chart's default apiURL and NetworkPolicy match OpenSandbox's own
chart (opensandbox-system, port 80).
Both runtimes' traffic is recorded under testdata/recordings and replayed
by TestReplayRecordings, so every fix is checked against real responses
without a server. The live test now asserts per-runtime outcomes instead of
logging, and the e2e suite adapts to the pool's capabilities and checks
that an unpullable image keeps the pool healthy. The suite passes 11/11 on
both runtimes.
Signed-off-by: Jalil <reachjalil@gmail.com>
* fix(security): reject unsafe devhost refs and serve sandbox files as downloads (#16)
devhost passed refs through filepath.Base, which keeps "." and "..", so
Delete("..") removed the adapter root. Refs now must match the generated
ID pattern. File reads from the API and the adapter protocol now send
Content-Disposition: attachment, nosniff and a sandbox CSP so browsers
never render sandbox file contents on the API origin.
Signed-off-by: Jalil <reachjalil@gmail.com>
* fix(sdk): trim trailing slashes without a backtracking regex (#17)
Signed-off-by: Jalil <reachjalil@gmail.com>
* ci: approve pull requests once CI passes (#18)
main's ruleset requires one approval, so release pull requests waited on a
person even when every check was green. A final CI job approves a pull
request from this repository once all CI jobs pass, using the workflow
token (the repository allows Actions to approve). Forks are never approved,
and a new push dismisses the approval until the next green run.
Signed-off-by: Jalil <reachjalil@gmail.com>
* docs: guide agents through maintaining and improving Differently (#19)
* docs: guide agents through maintaining and improving Differently
Add docs/maintaining.md: where to change what, a verification ladder from
package tests to a real cluster, how to keep the recorded OpenSandbox
runtimes current, the adapter contract's error semantics, where to look
first when optimising for self-hosting, and traps already hit. AGENTS.md
points to it and notes the CI approver and failure classification.
Signed-off-by: Jalil <reachjalil@gmail.com>
* docs(openwork): describe the runner's sandbox gateway settings
OpenWork's #6649 configures Differently as a sandbox gateway with its own
SANDBOX_GATEWAY_* settings and a per-org sandboxGateway flag, falling back
to the runner's own provider only when Differently is unhealthy. Replace
the guide's old HEADLESS_COMPUTER=differently and DIFFERENTLY_* names.
Signed-off-by: Jalil <reachjalil@gmail.com>
---------
Signed-off-by: Jalil <reachjalil@gmail.com>
* docs: make Differently easy to evaluate and deploy anywhere (#21)
* docs: make Differently easy to evaluate and deploy anywhere
Rewrite the README around a splash diagram (light and dark), release, CI
and license badges, links to the website, three getting-started paths and
a deploy-anywhere table with honest status. Add a verified guide for one
Linux VM on any cloud (k3s, OpenSandbox's charts and Differently's release
chart), a choose-where-to-run table and current support matrix, quickest
path notes for AWS, Google Cloud and Azure, SUPPORT.md and a docs link in
the issue chooser. Refresh the website's landing page from what has run:
a deploy-anywhere section, the verified, not verified and in-progress
lists, and getting started. Fix examples that ran python3 on images
without it.
Signed-off-by: Jalil <reachjalil@gmail.com>
* docs(site): publish the launch video and note the parity work in PR #20
Add a 720p web encode of the re-rendered launch video (36 s, silent) and
its poster to the website's video slot, with a caption that summarises it,
and link the poster from the README. Mention port URLs alongside the
snapshot-registry parity work.
Signed-off-by: Jalil <reachjalil@gmail.com>
---------
Signed-off-by: Jalil <reachjalil@gmail.com>
* feat(opensandbox): bring OpenSandbox pools close to Freestyle (#20)
* feat(opensandbox): bring OpenSandbox pools close to Freestyle
On OpenSandbox's Kubernetes runtime, with the controller's snapshot
registry, pools now offer stop/start with the disk kept, disk snapshots
and fork (runtime: kubernetes). OpenSandbox's pause commits the root
filesystem and frees the pod, so it is offered as stop/start and a memory
pause is refused.
Port URLs come from OpenSandbox's ingress gateway in route mode uri with
Secure Access (ports: signed): signed, time-limited, no header or key.
Egress is a pool setting (egress: none|allowlist), enforced by
OpenSandbox's egress sidecar. Exec stdin is staged in a file.
Checked on k3s on Azure and Docker: the e2e suite passes 13/13 on both,
with stop/start, snapshots, fork, port URLs and stdin exercised, and the
runs are recorded and replayed in go test.
Signed-off-by: Jalil <reachjalil@gmail.com>
* docs(opensandbox): record signed port URLs over public TLS and WebSockets
Signed-off-by: Jalil <reachjalil@gmail.com>
---------
Signed-off-by: Jalil <reachjalil@gmail.com>
* feat: stop idle sandboxes and wake them on the next call (#22)
Sandboxes can set idleTimeoutSeconds (or inherit defaults.idleTimeoutSeconds).
The scheduler leader pauses pools that keep memory, or stops pools that keep
the disk, once no exec, file, port or keepalive call has come in that long,
and marks the sandbox idled. The next such call wakes it transparently; a
sandbox a caller stopped or paused stays put. New keepalive endpoint for use
the control plane does not see, such as a desktop viewer on a port URL.
Works for any backend that can stop or pause. Checked end to end on
OpenSandbox's Kubernetes runtime (stopped after 60 s, woken by exec in about
1.7 s) and Docker runtime (paused, woken in under 0.1 s): e2e 14/14 on both.
Signed-off-by: Jalil <reachjalil@gmail.com>
* docs: add a provider catalog with a tested status for each backend (#23)
Each backend Differently can place sandboxes on now has a page with its
status (verified live, tested with a fake server, documented, or not
supported), capabilities, configuration and exactly what was tested. The
catalog explains how to choose between a built-in adapter and a remote
plugin and lists every place a new adapter is registered.
A new adapter kind can no longer become private by omission: only remote,
devhost and fake default to private, any other kind must be listed as
external or declare its trust, and a test fails until each kind is in
exactly one trust list.
Docs, README and the site now record the Kubernetes runtime's stop/start,
snapshots, fork, signed port URLs and egress policy as verified. The
dashboard can set an OpenSandbox pool's runtime, port URLs and egress, and
the Helm values no longer mention a Daytona key the adapter ignores.
Signed-off-by: Jalil <reachjalil@gmail.com>
* feat(adapters): add Vercel Sandbox and Fly.io Machines providers (#24)
* feat(adapters): add Vercel Sandbox and Fly.io Machines providers
Add in-process adapters for Vercel Sandbox (vercel) and Fly.io Machines
(fly), written from each provider's published REST API and tested against
fake servers. Both are registered in wiring, config (always external) and
the Helm schema.
Document Modal, Cloudflare's Sandbox SDK and Containers, and Fly.io Sprites
under docs/providers with the remote plugin shape each would need: Modal and
Cloudflare have no hosted HTTP API to create sandboxes, and Sprites
specifies exec only over WebSocket.
httpx gains ErrorShape, which normalises provider error bodies into the
shape the client reads, and lets callers set their own Accept header.
Signed-off-by: Jalil <reachjalil@gmail.com>
* feat(adapters): offer vercel and fly in pool add and the dashboard
Signed-off-by: Jalil <reachjalil@gmail.com>
---------
Signed-off-by: Jalil <reachjalil@gmail.com>
* feat(adapters): add Blaxel sandboxes and document Runloop, Beam and Koyeb (#26)
Add a blaxel adapter built from Blaxel's published control-plane and
sandbox API specs: idempotent create by name, paged list filtered by
installation, exec and streamed exec over the sandbox's NDJSON stream,
staged stdin, files, token-protected port previews and a TTL backstop.
Blaxel is always trust: external. Failures are classified by what Blaxel
says: quota and 429 requeue, a bad image is invalid_request, a missed
ready deadline or a FAILED sandbox is sandbox_failed.
httpx now honours a caller's Accept header and can send fixed headers
with every request, which the Blaxel stream and workspace selector need.
Provider pages: Blaxel (tested with a fake server), Runloop (not
supported: the service is winding down), Beam and Koyeb (documented: no
documented HTTP route to create or reach a sandbox, so a remote plugin
on their SDKs).
Signed-off-by: Jalil <reachjalil@gmail.com>
* feat(adapters): add Boat, E2B and Morph Cloud adapters, document CodeSandbox (#25)
* feat(adapters): add Boat, E2B and Morph Cloud adapters, document CodeSandbox
Boat, E2B and Morph Cloud get in-process adapters written from each
provider's published API and tested against fake servers. CodeSandbox is
documented as needing a remote plugin: its in-VM operations use an
undocumented WebSocket protocol.
All three are registered as external providers, so they receive no work
until policy approves them.
Signed-off-by: Jalil <reachjalil@gmail.com>
* fix(e2b): make E2B pools declare their trust
E2B's infrastructure is open source and can be self-hosted, so e2b joins
declaredTrustKinds: a pool must say whether its operator runs it. The
adapter reports the pool's declared trust. Boat and Morph Cloud stay
always external; a config test covers all three.
Signed-off-by: Jalil <reachjalil@gmail.com>
* fix(e2b): refuse command requests over 32 MiB before framing them
envelope now checks the message size before allocating, so the length
cannot overflow or truncate; Exec returns invalid_request for an oversized
command. readEnvelope uses the same bound.
Signed-off-by: Jalil <reachjalil@gmail.com>
---------
Signed-off-by: Jalil <reachjalil@gmail.com>
* feat(adapters): run sandboxes on AWS, Azure, Google Cloud and Northflank sandbox services (#27)
* feat(adapters): add Boat, E2B and Morph Cloud adapters, document CodeSandbox
Boat, E2B and Morph Cloud get in-process adapters written from each
provider's published API and tested against fake servers. CodeSandbox is
documented as needing a remote plugin: its in-VM operations use an
undocumented WebSocket protocol.
All three are registered as external providers, so they receive no work
until policy approves them.
Signed-off-by: Jalil <reachjalil@gmail.com>
* fix(e2b): make E2B pools declare their trust
E2B's infrastructure is open source and can be self-hosted, so e2b joins
declaredTrustKinds: a pool must say whether its operator runs it. The
adapter reports the pool's declared trust. Boat and Morph Cloud stay
always external; a config test covers all three.
Signed-off-by: Jalil <reachjalil@gmail.com>
* fix(e2b): refuse command requests over 32 MiB before framing them
envelope now checks the message size before allocating, so the length
cannot overflow or truncate; Exec returns invalid_request for an oversized
command. readEnvelope uses the same bound.
Signed-off-by: Jalil <reachjalil@gmail.com>
* feat: run sandboxes on AWS, Azure, Google Cloud and Northflank sandbox services
Add adapters for the cloud providers' own sandbox services:
aws-agentcore (Amazon Bedrock AgentCore Code Interpreter),
azure-sandboxes (Azure Container Apps Sandboxes), azure-sessions
(Azure Container Apps dynamic sessions, Python code-interpreter pools),
gcp-sandboxes (Google Cloud Agent Platform shell sandboxes) and
northflank. Each is tested against a fake server built from the
provider's published API; none has run against a live account.
Cloud IAM credentials need no SDK: Entra ID client credentials and
workload identity federation, Google service-account JWT grants, and
AWS Signature Version 4 live in internal/adapters/cloudhttp, with the
AWS event-stream decoder. Backends that only run commands share
shellcmd (cwd, env, stdin, timeout and files over base64) and pyshell
(shell commands through Python's subprocess).
The new kinds are in declaredTrustKinds: every pool states its trust,
external, or private when the operator's policy treats their own cloud
account so. They are registered in the CLI help and the dashboard too.
Provider pages in docs/providers cover them, plus AgentCore Runtime and
GKE Agent Sandbox, which are documented without an adapter.
Signed-off-by: Jalil <reachjalil@gmail.com>
---------
Signed-off-by: Jalil <reachjalil@gmail.com>
* feat(adapters): add agent-sandbox and CubeSandbox pools (#28)
* feat(adapters): add Boat, E2B and Morph Cloud adapters, document CodeSandbox
Boat, E2B and Morph Cloud get in-process adapters written from each
provider's published API and tested against fake servers. CodeSandbox is
documented as needing a remote plugin: its in-VM operations use an
undocumented WebSocket protocol.
All three are registered as external providers, so they receive no work
until policy approves them.
Signed-off-by: Jalil <reachjalil@gmail.com>
* fix(e2b): make E2B pools declare their trust
E2B's infrastructure is open source and can be self-hosted, so e2b joins
declaredTrustKinds: a pool must say whether its operator runs it. The
adapter reports the pool's declared trust. Boat and Morph Cloud stay
always external; a config test covers all three.
Signed-off-by: Jalil <reachjalil@gmail.com>
* fix(e2b): refuse command requests over 32 MiB before framing them
envelope now checks the message size before allocating, so the length
cannot overflow or truncate; Exec returns invalid_request for an oversized
command. readEnvelope uses the same bound.
Signed-off-by: Jalil <reachjalil@gmail.com>
* feat: run sandboxes on AWS, Azure, Google Cloud and Northflank sandbox services
Add adapters for the cloud providers' own sandbox services:
aws-agentcore (Amazon Bedrock AgentCore Code Interpreter),
azure-sandboxes (Azure Container Apps Sandboxes), azure-sessions
(Azure Container Apps dynamic sessions, Python code-interpreter pools),
gcp-sandboxes (Google Cloud Agent Platform shell sandboxes) and
northflank. Each is tested against a fake server built from the
provider's published API; none has run against a live account.
Cloud IAM credentials need no SDK: Entra ID client credentials and
workload identity federation, Google service-account JWT grants, and
AWS Signature Version 4 live in internal/adapters/cloudhttp, with the
AWS event-stream decoder. Backends that only run commands share
shellcmd (cwd, env, stdin, timeout and files over base64) and pyshell
(shell commands through Python's subprocess).
The new kinds are in declaredTrustKinds: every pool states its trust,
external, or private when the operator's policy treats their own cloud
account so. They are registered in the CLI help and the dashboard too.
Provider pages in docs/providers cover them, plus AgentCore Runtime and
GKE Agent Sandbox, which are documented without an adapter.
Signed-off-by: Jalil <reachjalil@gmail.com>
* feat(adapters): add agent-sandbox and CubeSandbox pools
Add two self-hosted runtimes behind the adapter contract, both with
operator-declared trust:
- agent-sandbox (kubernetes-sigs/agent-sandbox): Sandboxes, or
SandboxClaims on a warm pool, through the Kubernetes API with the
standard library instead of client-go. Exec and files run over
WebSocket pods/exec (v5 protocol, so stdin works), and stop/start
suspends the Sandbox when the pool gives it a volume. Checked live
on kind with agent-sandbox v1.0.8, directly and through a server.
- cubesandbox (TencentCloud/CubeSandbox): KVM microVMs through
CubeAPI and E2B's envd behind CubeProxy, with streamed exec, files,
pause/resume, memory snapshots and fork. Tested with a fake server.
Document both, and microsandbox as not supported: it has no
self-hosted server API since its rewrite.
Signed-off-by: Jalil <reachjalil@gmail.com>
* feat(cli,dashboard): offer agent-sandbox and cubesandbox when adding a pool
Signed-off-by: Jalil <reachjalil@gmail.com>
* fix(cubesandbox): refuse command requests over 16 MiB before framing them
envelope now checks the marshalled size against maxMessage before
allocating, so the length can neither overflow nor truncate; the request
is refused as invalid_request. Reading already used the same bound.
Signed-off-by: Jalil <reachjalil@gmail.com>
---------
Signed-off-by: Jalil <reachjalil@gmail.com>
* docs: list all 26 providers in the catalog with their tested status (#30)
* feat(adapters): add Boat, E2B and Morph Cloud adapters, document CodeSandbox
Boat, E2B and Morph Cloud get in-process adapters written from each
provider's published API and tested against fake servers. CodeSandbox is
documented as needing a remote plugin: its in-VM operations use an
undocumented WebSocket protocol.
All three are registered as external providers, so they receive no work
until policy approves them.
Signed-off-by: Jalil <reachjalil@gmail.com>
* fix(e2b): make E2B pools declare their trust
E2B's infrastructure is open source and can be self-hosted, so e2b joins
declaredTrustKinds: a pool must say whether its operator runs it. The
adapter reports the pool's declared trust. Boat and Morph Cloud stay
always external; a config test covers all three.
Signed-off-by: Jalil <reachjalil@gmail.com>
* fix(e2b): refuse command requests over 32 MiB before framing them
envelope now checks the message size before allocating, so the length
cannot overflow or truncate; Exec returns invalid_request for an oversized
command. readEnvelope uses the same bound.
Signed-off-by: Jalil <reachjalil@gmail.com>
* feat: run sandboxes on AWS, Azure, Google Cloud and Northflank sandbox services
Add adapters for the cloud providers' own sandbox services:
aws-agentcore (Amazon Bedrock AgentCore Code Interpreter),
azure-sandboxes (Azure Container Apps Sandboxes), azure-sessions
(Azure Container Apps dynamic sessions, Python code-interpreter pools),
gcp-sandboxes (Google Cloud Agent Platform shell sandboxes) and
northflank. Each is tested against a fake server built from the
provider's published API; none has run against a live account.
Cloud IAM credentials need no SDK: Entra ID client credentials and
workload identity federation, Google service-account JWT grants, and
AWS Signature Version 4 live in internal/adapters/cloudhttp, with the
AWS event-stream decoder. Backends that only run commands share
shellcmd (cwd, env, stdin, timeout and files over base64) and pyshell
(shell commands through Python's subprocess).
The new kinds are in declaredTrustKinds: every pool states its trust,
external, or private when the operator's policy treats their own cloud
account so. They are registered in the CLI help and the dashboard too.
Provider pages in docs/providers cover them, plus AgentCore Runtime and
GKE Agent Sandbox, which are documented without an adapter.
Signed-off-by: Jalil <reachjalil@gmail.com>
* feat(adapters): add agent-sandbox and CubeSandbox pools
Add two self-hosted runtimes behind the adapter contract, both with
operator-declared trust:
- agent-sandbox (kubernetes-sigs/agent-sandbox): Sandboxes, or
SandboxClaims on a warm pool, through the Kubernetes API with the
standard library instead of client-go. Exec and files run over
WebSocket pods/exec (v5 protocol, so stdin works), and stop/start
suspends the Sandbox when the pool gives it a volume. Checked live
on kind with agent-sandbox v1.0.8, directly and through a server.
- cubesandbox (TencentCloud/CubeSandbox): KVM microVMs through
CubeAPI and E2B's envd behind CubeProxy, with streamed exec, files,
pause/resume, memory snapshots and fork. Tested with a fake server.
Document both, and microsandbox as not supported: it has no
self-hosted server API since its rewrite.
Signed-off-by: Jalil <reachjalil@gmail.com>
* feat(cli,dashboard): offer agent-sandbox and cubesandbox when adding a pool
Signed-off-by: Jalil <reachjalil@gmail.com>
* fix(cubesandbox): refuse command requests over 16 MiB before framing them
envelope now checks the marshalled size against maxMessage before
allocating, so the length can neither overflow nor truncate; the request
is refused as invalid_request. Reading already used the same bound.
Signed-off-by: Jalil <reachjalil@gmail.com>
* docs: list all 26 providers in the catalog with their tested status
The catalog now covers every provider the adapter work added: 16 with a
built-in adapter (two verified live, the rest tested against fakes of the
provider's published API), 8 reachable through a remote plugin, and 2 that
cannot be supported, each with trust, isolation and what persists.
The README, the website and the CHANGELOG now describe those providers
instead of naming Daytona and Freestyle alone, and record that OpenWork's
sandbox gateway has merged.
Signed-off-by: Jalil <reachjalil@gmail.com>
---------
Signed-off-by: Jalil <reachjalil@gmail.com>
* feat(opensandbox): add the Fast Sandbox runtime with native idle pause and wake (#29)
runtime: fastsandbox creates Firecracker microVMs from a template, offers
pause/resume with memory kept, and passes a sandbox's idle timeout to
OpenSandbox's lifecycle.idlePolicy. Pools whose backend idles natively are
skipped by Differently's idle sweep, and the reconciler follows the
backend's pauses and gateway wakes.
Signed-off-by: Jalil <reachjalil@gmail.com>
* fix(scheduler): requeue a sandbox whose create outlives the provision timeout (#32)
The retry after a timed-out Create ran on the Create's expired context, so
with PostgreSQL its store writes failed silently and the sandbox stayed
provisioning, re-provisioned on every tick. Bound only the backend call by
the timeout.
Signed-off-by: Jalil <reachjalil@gmail.com>
* docs: bring the website, README and docs up to date with the providers (#33)
* docs: bring the website, README and docs up to date with the providers
The provider catalog's eight-column table was unreadable in the docs
site's content column; it is now a status table and a short capability
table. The README, the splash graphics, the social preview and the
website described Daytona and Freestyle as the only managed providers, and
the README counted 26 usable providers where 16 have adapters; they now
describe the 16 adapters and plugins for the rest.
The architecture page's evidence table still reported the end-to-end suite
at 10/10 on Docker only and the Helm chart as never installed; it now
matches what ran. The OpenSandbox page notes the Fast Sandbox runtime
(tested with a fake server only) and links the canonical repository.
Signed-off-by: Jalil <reachjalil@gmail.com>
* docs(site): bring the Features page up to date with the runtimes and providers
It still said OpenSandbox pools had no stop/start or stdin, that only the
Docker runtime had run end to end, and that Daytona and Freestyle were the
only other backends.
Signed-off-by: Jalil <reachjalil@gmail.com>
---------
Signed-off-by: Jalil <reachjalil@gmail.com>
* docs(providers): fit the capability table in the docs column (#35)
It overflowed the docs site's content column by about 56 px and cut off
the Port URLs column. Fork now shares the Snapshots column and the longest
cells are shorter, so the five columns fit without scrolling.
Signed-off-by: Jalil <reachjalil@gmail.com>
* docs: plain project copy on the website, without the decision records (#37)
* docs(site): describe Differently as project documentation
The landing page spoke to the reader and leaned on slogans ("Honest about
what is proven", "Orchestration you own"). It now uses plain section
titles and neutral descriptions. The status section leads with what has
been tested on real infrastructure, each item labelled, and lists what has
not run yet without the runtime limitation that belongs in the docs. The
top headline is unchanged.
Signed-off-by: Jalil <reachjalil@gmail.com>
* docs: remove the architecture decision records
The records were internal working notes. Pages that cited them now say
what was decided in a few words, or drop the citation where the text
already explains it. The site no longer has a Decision records section,
and CONTRIBUTING and GOVERNANCE describe design proposals without them.
Signed-off-by: Jalil <reachjalil@gmail.com>
---------
Signed-off-by: Jalil <reachjalil@gmail.com>
* feat(opensandbox): run Fast Sandbox live, with https port URLs and fork (#39)
* fix(opensandbox): give Fast Sandbox port URLs the gateway's https scheme
portScheme only applied to signed port URLs, which Fast Sandbox templates
refuse. Its gateway in route mode uri returns scheme-less endpoints, so a
gateway behind TLS gave http:// port URLs and execd addresses on a TLS-only
port. portScheme now applies to every gateway endpoint returned without a
scheme, including execd reached past the server proxy.
Signed-off-by: Jalil <reachjalil@gmail.com>
* feat(opensandbox): run Fast Sandbox live, with fork and a recording
Fast Sandbox ran on single-node k3s under nested KVM through the adapter
and with OpenWork's Workbot desktop. A restore from a snapshot now sends
the SandboxPool's size (poolCPU, poolMemory), which OpenSandbox requires
for fork. The recording harness covers template mode and a gateway reached
directly, replay collapses repeated polls, and fastsandbox-1.2.0 replays in
every go test. Docs record measured timings and the gaps found.
Signed-off-by: Jalil <reachjalil@gmail.com>
---------
Signed-off-by: Jalil <reachjalil@gmail.com>
* test(e2e): check Fast Sandbox microVM pools and desktop streaming (#41)
* test(e2e): add memory, idle-wake and desktop streaming checks for microVM pools
Signed-off-by: Jalil <reachjalil@gmail.com>
* test(e2e): document the desktop template on microVM pools
Signed-off-by: Jalil <reachjalil@gmail.com>
* fix(opensandbox): give Fast Sandbox port URLs the gateway's https scheme
portScheme only applied to signed port URLs, which Fast Sandbox templates
refuse. Its gateway in route mode uri returns scheme-less endpoints, so a
gateway behind TLS gave http:// port URLs and execd addresses on a TLS-only
port. portScheme now applies to every gateway endpoint returned without a
scheme, including execd reached past the server proxy.
Signed-off-by: Jalil <reachjalil@gmail.com>
* feat(opensandbox): run Fast Sandbox live, with fork and a recording
Fast Sandbox ran on single-node k3s under nested KVM through the adapter
and with OpenWork's Workbot desktop. A restore from a snapshot now sends
the SandboxPool's size (poolCPU, poolMemory), which OpenSandbox requires
for fork. The recording harness covers template mode and a gateway reached
directly, replay collapses repeated polls, and fastsandbox-1.2.0 replays in
every go test. Docs record measured timings and the gaps found.
Signed-off-by: Jalil <reachjalil@gmail.com>
* test(e2e): run the suite against a Fast Sandbox pool
Templates replace images there: DIFFERENTLY_E2E_TEMPLATE and
DIFFERENTLY_E2E_PYTHON_TEMPLATE pick them. Where the pool is a microVM,
check the guest's size instead of cgroup limits, and the precise refusals
of sandbox env and of a template the pool does not serve.
Signed-off-by: Jalil <reachjalil@gmail.com>
* docs(e2e): record the partial Fast Sandbox run and the new tests
Signed-off-by: Jalil <reachjalil@gmail.com>
---------
Signed-off-by: Jalil <reachjalil@gmail.com>
* chore(release): prepare v0.1.0-alpha.2 (#43)
Moves everything since v0.1.0-alpha.1 into a dated v0.1.0-alpha.2 section,
adds the changes the log was missing (the Fast Sandbox runtime, the
provision-timeout requeue fix, the security fixes, CI approval, the
microVM checks and the documentation work) and rewrites Verified to match
what has run. The README, the single-VM guide and the website now name
v0.1.0-alpha.2.
Signed-off-by: Jalil <reachjalil@gmail.com>
* test(providers): run the Freestyle and Daytona adapters against live accounts (#45)
Add live tests (build tag live, skipped without FREESTYLE_API_KEY or
DAYTONA_API_KEY) that drive every adapter operation against the real APIs
and delete everything they create. Running them, and a local
differently-server with a pool on each provider, found bugs the
spec-derived fakes could not:
Freestyle
- the env file's mode was sent as 0600, which Freestyle reads as decimal,
so commands could not read the sandbox env;
- rewriting an existing file makes it owned by root, so a replayed create
broke the env file: an existing file is now removed first;
- poweroff ran as the default user, who may not power off: it runs as root;
- with automatic restart on, Freestyle boots a powered-off VM again about
5 s later: VMs are created with it off.
Daytona
- stop and start returned while the sandbox was still stopping or
starting: they wait for the state;
- a command timeout (408) was reported as provider_unavailable: it is now
exit code 124;
- a repeated delete, or one during a stop, failed with a 409: it waits and
succeeds;
- snapshot delete sent the name where Daytona needs the id, and a snapshot
was returned before it was active.
The fake servers now behave as the live APIs did, so go test covers each
fix. The provider pages, catalog, feature parity, roadmap and POC 7 record
what ran. Daytona's linux-vm class did not run: the account had no VM
snapshot in its only region.
Signed-off-by: Jalil <reachjalil@gmail.com>
* fix(site): show a full pool's sandbox waiting in the launch video (#46)
The "Full? It fails closed." beat showed 422 no_eligible_pool, and the
audit line a rejected sandbox. When the eligible private pools are full and
external placement is never, policy keeps the sandbox waiting in the queue;
422 is only for a request no pool may take. The beat now shows waiting and
stays queued, in the diagram and the audit line. Nothing else in the video
changed.
Signed-off-by: Jalil <reachjalil@gmail.com>
* docs: record the live Daytona and Freestyle runs on the site and README (#47)
#45 ran both adapters against live accounts and marked them verified in the
provider catalog, but the README, the website, the Features page and the
deployment matrix still said no managed provider had run live. They now list
what ran on each and count the other 12 adapters as tested against fakes.
Signed-off-by: Jalil <reachjalil@gmail.com>
* fix(chart): let config pools use stored provider keys (#49)
* fix(chart): let config pools use stored provider keys
The server accepts credentials: {secret: name} on a pool declared in the
config file, but the chart's values schema allowed only env and file, so a
config pool on a key stored in Differently failed the schema check. An
external pool declared that way also got no egress to its provider, since
only providers.* opened providerCIDRs on 443.
Signed-off-by: Jalil <reachjalil@gmail.com>
* fix(chart): mount extra volumes in the preflight init container
Preflight checks that every pool credential resolves, but only the server
container got server.extraVolumeMounts, so a file: credential on a mounted
volume (a Key Vault CSI mount, say) failed preflight and the pod never
started.
Signed-off-by: Jalil <reachjalil@gmail.com>
---------
Signed-off-by: Jalil <reachjalil@gmail.com>
* fix(server): finish a lifecycle change when the caller stops waiting (#50)
Stop, start, pause and resume ran the backend call on the request's
context. A Fast Sandbox pause writes a full checkpoint and takes about three
minutes for a 4 GiB desktop, so a client with a shorter timeout canceled it
halfway: OpenSandbox went on pausing while the record still said running.
The transition now runs detached from the request, bounded at 15 minutes,
and is recorded when the backend finishes.
Signed-off-by: Jalil <reachjalil@gmail.com>
* feat(server): list only the pools a tenant may use in capabilities (#51)
GET /v1/capabilities listed every pool to every key. OpenWork's runner reads
it as the pools its key may be placed on and pauses idle computers only when
all of them can pause, so a pool the organization's policy denies (a
stop-only container pool, say) turned pausing off for everyone. A tenant
key now sees the pools its global, organization and project policies allow
for the default classification; operator keys still see every pool. The
policy checks move into policy.PolicyReasons, shared with placement.
Signed-off-by: Jalil <reachjalil@gmail.com>
* fix(opensandbox): set the Fast Sandbox clock and wait out long checkpoints (#53)
A Fast Sandbox microVM is restored from a memory snapshot, so it started
with the time its template or checkpoint was written; the adapter now sets
the guest clock through execd right after create and resume. Pause and
snapshot write a full checkpoint, about three minutes for a 4 GiB desktop,
which could outlast the three-minute ready timeout; they now wait up to
checkpointTimeoutSeconds (default 900). The fastsandbox-1.2.0 recording was
made again on a two-node Azure cluster with Workbot's desktop template.
Signed-off-by: Jalil <reachjalil@gmail.com>
* feat(console): add an admin console you can host anywhere (#52)
The console is an Astro app in console/ that talks only to the public API,
so all state and rules stay in the control plane. It has a sidebar with
Home, API keys, Sandboxes, Snapshots, Providers, Secrets, Policies, Usage,
the audit log, the trust report and Settings.
It builds two ways. The default is static files for any host, where the
browser holds the API key the person enters. Server mode runs on Node with
Better Auth (email and password, an email allowlist, optional GitHub) on
Node's built-in SQLite; there only the console server holds the key, and
its /api proxy forwards /version and /v1/* after checking the session.
To let a console on another origin call the API, the server gains
cors.allowedOrigins (exact origins, empty by default), and the TypeScript
SDK no longer imports node:crypto so it runs in browsers. CI builds both
console modes, and the site publishes the console guide.
Signed-off-by: Jalil <reachjalil@gmail.com>
* docs(deploy): record a verified Azure install serving OpenWork (#55)
docs/deploy/azure-vms.md describes the two-VM install that now serves
OpenWork's Workbot computers through Fast Sandbox, Freestyle and Daytona,
with what it took and what was measured. The OpenWork guide, the provider
and deploy status tables, the roadmap and the site record the end-to-end
run. The e2e suite now runs on an install with several pools and replicas:
the env refusal is pinned to the pool under test, the trust report check
looks for that pool among others, and tests get room for Fast Sandbox
checkpoints.
Signed-off-by: Jalil <reachjalil@gmail.com>
* feat(scheduler): paused sandboxes give their slot back, with queued resumes, preemption and lifecycle limits (#56)
Signed-off-by: Jalil <reachjalil@gmail.com>
* docs: bring the docs, maintainer guide and website up to date with what has run (#57)
Signed-off-by: Jalil <reachjalil@gmail.com>
* test(store): give the released-slots conformance test a pool row to lock (#59)
Signed-off-by: Jalil <reachjalil@gmail.com>
* feat(console): show queued resumes and released slots, with the SDK fields behind them (#58)
Signed-off-by: Jalil <reachjalil@gmail.com>
* perf: measure Fast Sandbox desktop pauses and shorten them with sparse templates (#60)
* feat(metrics): time each pause on its backend as differently_pause_seconds
On a pool that checkpoints, a paused sandbox frees its slot only when the
backend finishes the pause, so pause time is capacity. Every pause
Differently asks for (API, idle sweep, preemption, maxRunSeconds) now goes
through scheduler.Pause, which records how long the backend took, per pool.
A backend's own idle pause (Fast Sandbox native idle) is not timed.
Signed-off-by: Jalil <reachjalil@gmail.com>
* feat(capsim): take pause and resume times as flags; model the measured pause
differently-capsim gains -pause and -resume, so the effect of a faster
pause can be checked from the command line. The Workbot model's pause is
182 s, the median measured for an in-use desktop with sparse templates.
Signed-off-by: Jalil <reachjalil@gmail.com>
* test(pause): add the harness that splits Fast Sandbox pauses into phases
Pause and resume a sandbox through Differently repeatedly, take each
phase's time from the controller, fastlet, runtime agent and Garage logs,
sample the state-root disk and CPU every second, and check that the guest
kept its processes, open files and memory, tmpfs and disk contents. Also
a store benchmark that uploads like the agent does, and a stand-in that
punches a cached template's zero runs into holes.
Signed-off-by: Jalil <reachjalil@gmail.com>
* docs: measure Fast Sandbox desktop pauses and shorten them with sparse templates
docs/research/fast-sandbox-pause-time.md records where a Workbot desktop's
pause spends its time on the Azure test host (memory dump, a checksum pass
that reads the whole 14 GB checkpoint back, a single-stream upload, all
limited by the state-root disk), the candidates ranked, and before/after
measurements: leaving the template's zero runs as holes took the median
pause from 216 s to 180 s with resumes unchanged and state kept. It carries
two Fast Sandbox patches (sparse pull, parallel multipart upload) and the
remaining upstream ideas with expected gains.
capacity.md, the pools guide (new "Shortening pauses"), the Azure guide
and the pages quoting the pause time follow; the capacity table is
regenerated with the measured pause.
Signed-off-by: Jalil <reachjalil@gmail.com>
* docs: record why Fast Sandbox resumes can take two minutes, with the fix
A resume that misses the node's image cache takes 117-138 s instead of 2 s.
The fastlets share that cache but rank eviction by their own use counts, so
the GC deleted checkpoints paused sandboxes still needed. Record the cause,
the production results and the patch, and note that a fastlet upgrade stalls
on a single KVM node.
Signed-off-by: Jalil <reachjalil@gmail.com>
---------
Signed-off-by: Jalil <reachjalil@gmail.com>
* fix(lifecycle): show pausing and keep work off a sandbox mid-pause (#61)
* fix(lifecycle): show pausing and keep work off a sandbox mid-pause
A Fast Sandbox pause takes minutes: the VM is frozen for a memory dump,
resumed while the checkpoint uploads, then deleted. Throughout, Differently
reported running. Exec and file calls succeeded, and their work was gone
when the checkpoint was restored. A resume in that window did nothing.
Idle preemption made this easy to hit. Activity was recorded only for
sandboxes with an idle timeout, so the rest looked idle a minute after they
last started, and could be paused in the middle of a command.
A sandbox being paused now reports pausing and refuses exec, file and port
calls. API, idle and preemption pauses share one implementation, and a
failed pause goes back to running. A resume during a pause calls it off
where the backend allows (Fast Sandbox), which keeps the sandbox as it is;
elsewhere it waits in the resume queue. The reconciler settles a pause whose
replica stopped. Activity is recorded for every preemptible sandbox, and a
running command counts until it ends.
Clients that list sandboxes by status need to add pausing.
Signed-off-by: Jalil <reachjalil@gmail.com>
* test(lifecycle): count only finished pauses in differently_pause_seconds
Signed-off-by: Jalil <reachjalil@gmail.com>
---------
Signed-off-by: Jalil <reachjalil@gmail.com>
* docs(lifecycle): record how pauses behave on a real Fast Sandbox (#62)
Calling off a pause has now run on a real Fast Sandbox, so the parity row
moves from fake servers to live. The numbers go in the capacity docs, along
with what a called-off pause still costs: its checkpoint keeps using the disk,
and a pause started right after it took about 8 minutes.
Signed-off-by: Jalil <reachjalil@gmail.com>
* ci: run the Python SDK job on an image that still has Python 3.9 (#81)
`ubuntu-latest` now resolves to Ubuntu 26.04, for which setup-python has no
Python 3.9 build: "The version '3.9' with architecture 'x64' was not found
for Ubuntu 26.04". The `sdk (python 3.9)` job fails at setup on every new
pull request, the 3.13 job is cancelled with it, and the approve job never
runs. The last run on `pre-release` (2 November) predates the image change.
The SDK declares `requires-python >= 3.9`, so the job is pinned to
ubuntu-24.04 instead of dropping the version from the matrix.
Not verified: the workflow only runs on GitHub.
Signed-off-by: Jalil <reachjalil@gmail.com>
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
* fix(scheduler): keep sandboxes queued when a placement policy cannot be read (#72)
A failed read of the global, organization or project policy was treated as
"no policy at that scope". During a database blip, a sandbox whose
organization sets `external: never` (or denies a pool) could be placed with
that scope left out, and run on an external pool the global policy allows.
Policy and snapshot reads now return their error. The scheduler leaves the
sandbox queued and retries; create, placement explanation and capabilities
answer 503 `provider_unavailable` instead of deciding on a partial view.
Test: TestPlacementFailsClosedWhenPolicyUnreadable (fails on the old code
with the sandbox running on the external pool).
Signed-off-by: Jalil <reachjalil@gmail.com>
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
* fix(reconciler): never delete or adopt backend instances Differently does not own (#73)
The reconciler deletes every instance a pool lists that the store does not
know. Daytona and Freestyle trusted the provider's own label/metadata filter
for that list, and Daytona's create reused the first sandbox the filter
returned. A provider that ignores or renames the filter returns the whole
account: each sandbox then looked like an orphan and was deleted, and create
could adopt someone else's sandbox.
- The reconciler leaves alone any instance that carries no sandbox ID, counts
it (`unowned_skipped`) and logs the pool.
- Daytona and Freestyle `List` keep only instances labelled with this
installation and a sandbox ID, as the OpenSandbox adapter already does.
- Daytona `Create` reuses a sandbox only when its label carries the ID.
Not covered: the other provider adapters still rely on their backend's filter
for the installation match; the reconciler guard covers them only for
instances with no sandbox ID.
Tests: TestDaytonaIgnoresSandboxesItDoesNotOwn,
TestFreestyleListIgnoresVMsItDoesNotOwn, and a new case in
TestFailureRecoveryAndReconcile (all three fail on the old code).
Signed-off-by: Jalil <reachjalil@gmail.com>
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
* fix(server): reject a negative or zero ttlSeconds (#74)
A negative `ttlSeconds` on create was stored as given: it passed the
`maxTTLSeconds` quota check (-1 is below any limit) and the sandbox got no
expiry at all. A `PATCH` with `ttlSeconds: 0` set the expiry to "now" and the
sandbox was deleted within a second, although 0 on create means "the operator
default".
Create now rejects a negative TTL (0 still takes the default). Update requires
a positive TTL; ending a sandbox now is what DELETE is for. The OpenAPI
document states both.
Test: TestInvalidTTLIsRejected (fails on the old code: create answered 202).
Signed-off-by: Jalil <reachjalil@gmail.com>
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
* fix(scheduler): keep placing sandboxes while a pool does not answer (#75)
Pool probes and the reconciler ran on the scheduler loop. A backend that
stops answering holds each probe for its 5-second timeout and each reconcile
list for 30 seconds, and nothing was dispatched, provisioned or terminated on
any pool while the loop waited: one unreachable provider slowed placement
everywhere.
Both now run beside the loop, one pass at a time each. The idle sweep still
starts after the reconcile pass it follows, without holding up the next one.
`Scheduler.RefreshInterval` makes the probe interval settable for tests.
Test: TestUnreachablePoolDoesNotStallScheduling (on the old loop the sandbox
is still queued after 2 s).
Signed-off-by: Jalil <reachjalil@gmail.com>
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
* fix(scheduler): let a concurrent delete or TTL change win over a stale decision (#76)
The scheduler lists sandboxes, decides, then writes. Three of those writes
set the status without looking at the record again, so a request that landed
in between was overwritten:
- expiry: a sandbox whose TTL was extended after the list was read was still
marked "ttl expired" and deleted;
- the reconciler's "failed" (instance disappeared or errored, provisioning
exhausted, queue timeout) replaced a delete already in progress;
- crash adoption set a sandbox that had just been deleted back to running.
Each write now re-checks the record under its row lock and leaves it alone
when the state it decided on is gone. Audit events and metrics are recorded
only for changes that happened.
Test: TestSchedulerDoesNotOverwriteConcurrentChanges (all three cases fail on
the old code).
Signed-off-by: Jalil <reachjalil@gmail.com>
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
* fix(adapters): keep the backend's TTL backstop behind Differently's own expiry (#77)
OpenSandbox, Daytona and Freestyle were sent the sandbox TTL as their own
deletion deadline. Theirs counts from the create request; Differently's
counts from the moment the sandbox is ready. The backend therefore deleted
first, by however long provisioning took, and a reconcile pass in that gap
ended the sandbox as "failed: backend instance disappeared" instead of
expired.
The three adapters now send the TTL plus a five-minute grace, the convention
the AgentCore, Azure, Google and Northflank adapters already follow. A TTL
update already moved OpenSandbox's deadline to five minutes after
Differently's.
Tests: TestDaytonaBackstopOutlastsTheTTL, TestFreestyleBackstopOutlastsTheTTL
and updated OpenSandbox create assertions. Not run against live backends.
Signed-off-by: Jalil <reachjalil@gmail.com>
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
* fix(pools): keep a working adapter when its credential cannot be read (#78)
Every pool refresh (about every 5 s) re-reads each pool's credential to spot
a rotation. Any read error was fingerprinted as "unresolved", which differs
from the stored fingerprint, so the pool was rebuilt, the rebuild failed on
the same error, and the pool had no adapter: exec, files and placement on it
answered 503 until a later pass. One failed database read took a healthy
provider pool down.
A read that fails without saying the credential is gone now leaves the
adapter as built and logs a warning. A deleted secret or missing file still
takes the pool down, and a changed value still rebuilds it.
Test: TestSyncKeepsAdapterWhenCredentialUnreadable, the package's first test
(fails on the old code).
Signed-off-by: Jalil <reachjalil@gmail.com>
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
* fix(cli): send the dev operator token to loopback addresses only (#79)
With no API key set, the CLI used the dev server's operator token for any URL
that contained "localhost". A host such as localhost.example.com, or any URL
with "localhost" in its path, received the token; 127.0.0.1 and [::1] did not.
The token is now used only when the URL's host is localhost, a *.localhost
name or a loopback address.
Test: TestDevTokenOnlyForLoopback.
Signed-off-by: Jalil <reachjalil@gmail.com>
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
* fix(client): let a streamed exec run longer than the request timeout (#80)
The Go client's HTTP client has a 10-minute overall timeout, and it covered
streamed exec too. `differently sandbox exec` on a command that ran longer
than ten minutes was cut off mid-stream with a timeout error, although the
command was still running and sending output.
Streamed exec now uses the same HTTP client without the overall timeout; the
caller's context bounds it. Other requests keep the timeout. A synchronous
exec is still limited by it.
Test: TestExecStreamOutlivesTheRequestTimeout, the package's first test
(fails on the old code).
Signed-off-by: Jalil <reachjalil@gmail.com>
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
* docs(changelog): record the placement, scheduler and client fixes (#82)
Adds one Unreleased section for the nine defect fixes and the CI fix merged
in #72 to #81. They were left out of those pull requests so that nine
branches would not edit the same lines.
It names the two behaviour changes a client can notice: a PATCH with
`ttlSeconds: 0` now answers 400, and the CLI's dev token also works for
127.0.0.1.
Signed-off-by: Jalil <reachjalil@gmail.com>
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
* chore(build)!: keep only the verified adapters in the default build (#83)
The default build (published image, release binaries, make build) now holds
opensandbox, daytona, freestyle, the remote plugin protocol, devhost and
fake. The other 13 adapter kinds and the cloudhttp, pyshell and shellcmd
helpers carry //go:build experimental_adapters, so they compile in only with
that tag. No adapter code is removed.
- wiring: the 13 registrations move to wiring_experimental.go.
- config: DefaultAdapters and ExperimentalAdapters; KnownAdapters is what
the binary can build. A pool of a kind left out is refused, in the config
file and through POST /v1/admin/pools, with an error naming the tag.
- pools: a stored pool of such a kind stays registered and reports
unavailable with the same reason.
- Helm: pools of these kinds need server.experimentalAdapters=true.
- Dashboard, console and CLI help list only the default kinds.
- Makefile: GO_TAGS for build and images; `make experimental` builds, vets
and tests with the tag and is part of `make verify`. CI gains a
"go (experimental adapters)" job that the auto-approval waits for.
- Docs, site and changelog say which kinds are in the default build.
BREAKING CHANGE: a server built without -tags experimental_adapters no
longer accepts pools of agent-sandbox, cubesandbox, e2b, boat, morph,
vercel, fly, blaxel, aws-agentcore, azure-sandboxes, azure-sessions,
gcp-sandboxes or northflank.
Signed-off-by: Jalil <reachjalil@gmail.com>
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
* docs(sdk): install the SDKs from this repository, not from a registry (#84)
`pip install differently` installs an unrelated project: that PyPI name
belongs to cariad/differently. `npm install @differently/sdk` returns 404.
Neither SDK has been published, and the READMEs led with both commands.
The READMEs now say the SDKs are not published and give install lines that
work: pip from the repository's `sdk/python` subdirectory, npm from a built
checkout. The Python README warns against the PyPI name.
Also corrected: the method tables listed `admin.agents.list`, which does not
exist, and left out `apiKeys.get/rotate`, `admin.secrets` and `pools.use`;
the README quickstart now says an operator key needs `projectId`.
Signed-off-by: Jalil <reachjalil@gmail.com>
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
* fix(audit): show the newest events in the CLI, dashboard and console (#85)
`GET /v1/audit-events` returns events oldest first. The CLI's
`usage audit`, the dashboard's Audit tab and the console's audit log asked
for the first page (100 or 200 events) and reversed it, so on any
installation with more events than that, nothing recent ever appeared.
The endpoint takes `order=desc` (newest first; the page's nextCursor is the
`beforeSeq` of the next, older page) and `beforeSeq`. Both stores implement
it, the three views use it, and the TypeScript and Python SDKs take `order`
and `beforeSeq`/`before_seq`. The default order is unchanged.
Tests: TestAuditEventsNewestFirst (server), new cases in the store
conformance suite's AuditChain (run against Postgres in CI only), and one
test in each SDK.
Signed-off-by: Jalil <reachjalil@gmail.com>
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
* ci: vet the end-to-end and live test files (#86)
`test/e2e` and the adapters' live tests sit behind the `e2e` and `live`
buil…
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is this about?
CI compiles and vets the end-to-end and live test files.
What problem does it solve?
test/e2eand the adapters' live tests sit behind thee2eandlivebuild tags. No CI job runs them, and none compiled them either, so a change to an API they use could break them without anything failing until someone next ran them by hand.How is it implemented?
make vetalso runsgo vet -tags "e2e live" ./....make experimentalvets with-tags "experimental_adapters e2e live".goCI job callsmake vetinstead ofgo vet ./....This compiles and vets those tests. It does not run them: running the end-to-end suite in CI is separate work.
How was it tested?
make fmt-check vetpasses locally, andgo vet -tags "e2e live experimental_adapters" ./...exits 0. The CI change is exercised by this PR's own run.Checklist
make verifypasses (make fmt-check vetrun here; nothing else changed)git commit -s)api/openapi.yaml, both SDKs anddocs/feature-parity.md(no API change)🤖 Generated with Claude Code