From 0e6501f31ffbf698c53ca97945a54c2832c357c4 Mon Sep 17 00:00:00 2001 From: Kristin Martin Date: Tue, 14 Jul 2026 20:55:16 +0000 Subject: [PATCH 1/7] Fix API details and style nits in warm pools blueprint - Use port 80 with the http handler for the Flycast service; Flycast is HTTP-only and .flycast hostnames can't get public certs - Drop min_machines_running from the example; it's inert with autostop off - Don't claim a specific 422 status for app-name collisions or a 60s max on the wait endpoint (docs and OpenAPI spec say 60s is the default) - Repoint the lease link to an anchor that exists and drop the anchor on the exec link (exec isn't in the API reference yet) - Use the canonical /docs/launch/processes/ URL instead of a redirect - Match blueprint conventions: sentence-case heading, colon separators in Related reading --- blueprints/warm-pool-user-machines.html.md | 25 +++++++++++----------- 1 file changed, 12 insertions(+), 13 deletions(-) diff --git a/blueprints/warm-pool-user-machines.html.md b/blueprints/warm-pool-user-machines.html.md index de3a5503e4..d582f0a972 100644 --- a/blueprints/warm-pool-user-machines.html.md +++ b/blueprints/warm-pool-user-machines.html.md @@ -41,7 +41,7 @@ curl -X POST "https://api.machines.dev/v1/apps/pool-x7f3k2/ip_assignments" \ -d '{"type": "private_v6", "network": ""}' ``` -App creation returns `422` if the name is taken. Your worker will retry after crashes, so treat that as idempotent: `GET /v1/apps/{name}` to confirm the app exists and reuse it. If it doesn't exist, the `422` was a real validation error, so rethrow it. +App creation fails if the name is already taken. Your worker will retry after crashes, so treat creation as idempotent: on a name-collision error, `GET /v1/apps/{name}` to confirm the app exists and reuse it. If it doesn't exist, the failure was a real validation error, so rethrow it. ## Track pool state in a database @@ -87,10 +87,9 @@ curl -X POST "https://api.machines.dev/v1/apps/pool-x7f3k2/machines" \ "mounts": [{"volume": "vol_abc123", "path": "/data"}], "services": [{ "internal_port": 8080, - "ports": [{"port": 443, "handlers": ["tls", "http"]}], + "ports": [{"port": 80, "handlers": ["http"]}], "autostart": true, - "autostop": "off", - "min_machines_running": 1 + "autostop": "off" }], "checks": { "health": {"type": "tcp", "port": 8080, "interval": "15s", @@ -101,7 +100,7 @@ curl -X POST "https://api.machines.dev/v1/apps/pool-x7f3k2/machines" \ } }' -# Wait for the VM to boot (max timeout is 60s) +# Wait for the VM to boot (timeout defaults to 60s) curl "https://api.machines.dev/v1/apps/pool-x7f3k2/machines/${MACHINE_ID}/wait?state=started&timeout=60" \ -H "Authorization: Bearer ${FLY_API_TOKEN}" ``` @@ -114,7 +113,7 @@ A few settings in that config matter more than they look. **`autostop: "off"` is the point.** Pool Machines are idle by design. [Autostop](/docs/launch/autostop-autostart/) would helpfully shut them down, and a stopped pool Machine is a cold start with extra steps. -**`started` doesn't mean ready.** The `wait` endpoint tells you the VM booted, not that your app inside is serving. Poll your app's health endpoint through the [exec endpoint](/docs/machines/api/machines-resource/#execute-a-command-on-a-machine) before marking the entry `ready`: +**`started` doesn't mean ready.** The `wait` endpoint tells you the VM booted, not that your app inside is serving. Poll your app's health endpoint through the [exec endpoint](/docs/machines/api/machines-resource/) before marking the entry `ready`: ```bash curl -X POST "https://api.machines.dev/v1/apps/pool-x7f3k2/machines/${MACHINE_ID}/exec" \ @@ -172,18 +171,18 @@ curl -X POST "https://api.machines.dev/v1/apps/pool-x7f3k2/machines/${MACHINE_ID This also decides where per-user secrets should live: in the config file on the volume, not in Machine env vars. Env changes require a Machine update and a restart. A file write doesn't. -When you do need a real Machine update (rename, metadata, env vars), use the [lease flow](/docs/machines/api/machines-resource/#machine-leases) so concurrent updaters can't clobber each other: acquire a lease, send the update with the `fly-machine-lease-nonce` header, release the lease in a `finally`. And know that Machine updates replace the entire config; they don't merge. `GET` the current Machine first and send back its full config with only your fields changed, or you'll silently wipe mounts, services, checks, and env. +When you do need a real Machine update (rename, metadata, env vars), use the [lease flow](/docs/machines/api/machines-resource/#create-a-machine-lease) so concurrent updaters can't clobber each other: acquire a lease, send the update with the `fly-machine-lease-nonce` header, release the lease in a `finally`. And know that Machine updates replace the entire config; they don't merge. `GET` the current Machine first and send back its full config with only your fields changed, or you'll silently wipe mounts, services, checks, and env. ## Maintain the pool -A small worker, ideally its own [process group](/docs/apps/processes/), loops on an interval. Each tick does four things, in order: +A small worker, ideally its own [process group](/docs/launch/processes/), loops on an interval. Each tick does four things, in order: 1. **Cycle.** Delete unallocated entries whose `provision_version` doesn't match the current one. Bump the version whenever your image or Machine config changes, and the pool replaces itself over the next few ticks. 2. **Clean.** For `failed` entries, delete the app and the row. 3. **Validate.** `GET /machines/{id}` for each `ready` entry; discard anything whose `state` isn't `started`. Also discard rows stuck in `provisioning` for more than ~10 minutes. That's a crashed provisioner, and the incrementally-written IDs let you clean up whatever it left behind. 4. **Replenish.** Count available entries (`status = 'ready' AND allocated_at IS NULL`) plus in-flight `provisioning` entries against your target, and create the shortfall. Counting claimed-but-still-`ready` rows here is the classic bug: the pool looks full and quietly stops refilling as users claim entries. Spread the new entries across regions by weight (say, 3 in `us`, 1 in `eu`, 1 in `apac`, where those are your own buckets of concrete Fly regions like `iad`, `ams`, and `sin`). Counting in-flight `provisioning` entries stops you from over-provisioning during a burst of claims. Guard against overlapping runs. -## Pointers & Footguns +## Pointers and footguns - **Warm Machines cost money.** Pool size is a bet on your peak claim rate versus your tolerance for cold-start fallbacks. Start small (2-5), log ready/provisioning counts every tick, and grow only when you see fallbacks under real traffic. If your app boots fast, a pool of *stopped* Machines that you `start` on claim may be the better trade, since stopped Machines cost much less than running ones. - **Mint unique secrets per entry** (auth tokens, gateway passwords) at provision time and store them encrypted in your database. A claimed Machine should never be reachable with another entry's credentials. @@ -193,7 +192,7 @@ A small worker, ideally its own [process group](/docs/apps/processes/), loops on ## Related reading -- [Per-User Dev Environments with Fly Machines](/docs/blueprints/per-user-dev-environments/) The architecture this pattern plugs into, including routing with `fly-replay`. -- [Connecting to User Machines](/docs/blueprints/connecting-to-user-machines/) How to get traffic to a fleet of per-user Machines once you've assigned them. -- [Why one app per user?](/docs/machines/guides-examples/one-app-per-user-why) The isolation reasoning behind app-per-entry. -- [Machines API reference](/docs/machines/api/) Apps, Machines, volumes, leases, and exec. +- [Per-User Dev Environments with Fly Machines](/docs/blueprints/per-user-dev-environments/): The architecture this pattern plugs into, including routing with `fly-replay`. +- [Connecting to User Machines](/docs/blueprints/connecting-to-user-machines/): How to get traffic to a fleet of per-user Machines once you've assigned them. +- [Why one app per user?](/docs/machines/guides-examples/one-app-per-user-why): The isolation reasoning behind app-per-entry. +- [Machines API reference](/docs/machines/api/): Apps, Machines, volumes, leases, and exec. From 515ff0d7d842320a2de3c86ce0385854be4f900a Mon Sep 17 00:00:00 2001 From: Kristin Martin Date: Tue, 14 Jul 2026 21:56:15 +0000 Subject: [PATCH 2/7] Add warm pools guide to the guides side nav The blueprint was added to the index page but not to partials/_guides_nav.html.erb, so it never appeared in the side nav. Placed under Architecture Patterns, matching the index page order. --- partials/_guides_nav.html.erb | 1 + 1 file changed, 1 insertion(+) diff --git a/partials/_guides_nav.html.erb b/partials/_guides_nav.html.erb index 429db9d396..3c20d28eb0 100644 --- a/partials/_guides_nav.html.erb +++ b/partials/_guides_nav.html.erb @@ -16,6 +16,7 @@ links: [ { text: "Deploy OpenClaw on Fly.io", path: "/docs/blueprints/deploy-openclaw/" }, { text: "Run Hermes Agent on Fly.io", path: "/docs/blueprints/hermes-agent-on-fly-io/" }, + { text: "Warm pools of user Machines", path: "/docs/blueprints/warm-pool-user-machines/" }, { text: "Deploying Remote MCP Servers", path: "/docs/blueprints/remote-mcp-servers/" }, { text: "Resilient apps use multiple Machines", path: "/docs/blueprints/resilient-apps-multiple-machines/" }, { text: "Getting Started with N-Tier Architecture", path: "/docs/blueprints/n-tier-architecture/" }, From c4c98f6855991a8c642aaa19927d495a1dc75f00 Mon Sep 17 00:00:00 2001 From: Kyle McLaren Date: Wed, 15 Jul 2026 14:18:51 +0200 Subject: [PATCH 3/7] Fix Flycast allocation to target the app's own network The allocation call passed "network": "", which allocates the Flycast address into the organization's default network, not the app's isolated per-app network. Verified against the live Machines API: an empty (or omitted) network lands the IP in the org default 6PN segment, while passing the app's network name places it in the app's own segment alongside its Machines. As written, the Flycast address could not reach a Machine in a custom-network app. Allocate with the same network name used to create the app, and add prose explaining the field does not default to the app's network. Co-Authored-By: Claude Opus 4.8 --- blueprints/warm-pool-user-machines.html.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/blueprints/warm-pool-user-machines.html.md b/blueprints/warm-pool-user-machines.html.md index d582f0a972..f406c7a47d 100644 --- a/blueprints/warm-pool-user-machines.html.md +++ b/blueprints/warm-pool-user-machines.html.md @@ -34,13 +34,15 @@ curl -X POST "https://api.machines.dev/v1/apps" \ "network": "pool-x7f3k2" }' -# Allocate a Flycast (private) address +# Allocate a Flycast (private) address into the app's own network curl -X POST "https://api.machines.dev/v1/apps/pool-x7f3k2/ip_assignments" \ -H "Authorization: Bearer ${FLY_API_TOKEN}" \ -H "Content-Type: application/json" \ - -d '{"type": "private_v6", "network": ""}' + -d '{"type": "private_v6", "network": "pool-x7f3k2"}' ``` +Pass the same `network` name you created the app with. The `network` field on the allocation targets which 6PN network the Flycast address routes within, and it does not default to the app's own network. Leave it empty and the address lands in your organization's default network, where it cannot reach a Machine that lives in the app's isolated per-app network. + App creation fails if the name is already taken. Your worker will retry after crashes, so treat creation as idempotent: on a name-collision error, `GET /v1/apps/{name}` to confirm the app exists and reuse it. If it doesn't exist, the failure was a real validation error, so rethrow it. ## Track pool state in a database From adc89a535058e51f3c460f1a909ace3a4c7f917b Mon Sep 17 00:00:00 2001 From: Kristin Martin Date: Wed, 15 Jul 2026 09:14:44 -0700 Subject: [PATCH 4/7] add verified 60s max Co-authored-by: Kyle McLaren --- blueprints/warm-pool-user-machines.html.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/blueprints/warm-pool-user-machines.html.md b/blueprints/warm-pool-user-machines.html.md index f406c7a47d..4b6eeea91e 100644 --- a/blueprints/warm-pool-user-machines.html.md +++ b/blueprints/warm-pool-user-machines.html.md @@ -102,7 +102,7 @@ curl -X POST "https://api.machines.dev/v1/apps/pool-x7f3k2/machines" \ } }' -# Wait for the VM to boot (timeout defaults to 60s) +# Wait for the VM to boot (timeout defaults to 60s, which is also the max) curl "https://api.machines.dev/v1/apps/pool-x7f3k2/machines/${MACHINE_ID}/wait?state=started&timeout=60" \ -H "Authorization: Bearer ${FLY_API_TOKEN}" ``` From 50cc2ab1b67d5815e79c987255f6ea0ede169c44 Mon Sep 17 00:00:00 2001 From: Kristin Martin Date: Wed, 15 Jul 2026 17:31:11 +0000 Subject: [PATCH 5/7] Add 422 to the Machines API response codes table --- machines/api/working-with-machines-api.html.markerb | 1 + 1 file changed, 1 insertion(+) diff --git a/machines/api/working-with-machines-api.html.markerb b/machines/api/working-with-machines-api.html.markerb index 3d5abcd12a..5c060d6e4c 100644 --- a/machines/api/working-with-machines-api.html.markerb +++ b/machines/api/working-with-machines-api.html.markerb @@ -73,6 +73,7 @@ Typically, 2xx HTTP status codes denote successful operations, 4xx codes imply f | 401 | The API key used was missing or invalid. | | 404 | The resource was not found. | | 408 | Request timed out. | +| 422 | Validation failed (for example, an app name that is already taken). | | 5xx | Indicates an error with Fly.io API servers. | ## Rate Limits From 263e2bf7febc0149c45d27d089545a2860719633 Mon Sep 17 00:00:00 2001 From: Kristin Martin Date: Wed, 15 Jul 2026 17:39:28 +0000 Subject: [PATCH 6/7] Link exec to its page in the generated Machines API reference --- blueprints/warm-pool-user-machines.html.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/blueprints/warm-pool-user-machines.html.md b/blueprints/warm-pool-user-machines.html.md index 4b6eeea91e..3eab23dbb7 100644 --- a/blueprints/warm-pool-user-machines.html.md +++ b/blueprints/warm-pool-user-machines.html.md @@ -115,7 +115,7 @@ A few settings in that config matter more than they look. **`autostop: "off"` is the point.** Pool Machines are idle by design. [Autostop](/docs/launch/autostop-autostart/) would helpfully shut them down, and a stopped pool Machine is a cold start with extra steps. -**`started` doesn't mean ready.** The `wait` endpoint tells you the VM booted, not that your app inside is serving. Poll your app's health endpoint through the [exec endpoint](/docs/machines/api/machines-resource/) before marking the entry `ready`: +**`started` doesn't mean ready.** The `wait` endpoint tells you the VM booted, not that your app inside is serving. Poll your app's health endpoint through the [exec endpoint](https://docs.machines.dev/machines/Machines_exec+external) before marking the entry `ready`: ```bash curl -X POST "https://api.machines.dev/v1/apps/pool-x7f3k2/machines/${MACHINE_ID}/exec" \ From 599cb404628a7d8ae73801c23f0ffee922930890 Mon Sep 17 00:00:00 2001 From: Kristin Martin Date: Wed, 15 Jul 2026 10:47:21 -0700 Subject: [PATCH 7/7] add 422 info Co-authored-by: Kyle McLaren --- blueprints/warm-pool-user-machines.html.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/blueprints/warm-pool-user-machines.html.md b/blueprints/warm-pool-user-machines.html.md index 3eab23dbb7..07987abeb0 100644 --- a/blueprints/warm-pool-user-machines.html.md +++ b/blueprints/warm-pool-user-machines.html.md @@ -43,7 +43,7 @@ curl -X POST "https://api.machines.dev/v1/apps/pool-x7f3k2/ip_assignments" \ Pass the same `network` name you created the app with. The `network` field on the allocation targets which 6PN network the Flycast address routes within, and it does not default to the app's own network. Leave it empty and the address lands in your organization's default network, where it cannot reach a Machine that lives in the app's isolated per-app network. -App creation fails if the name is already taken. Your worker will retry after crashes, so treat creation as idempotent: on a name-collision error, `GET /v1/apps/{name}` to confirm the app exists and reuse it. If it doesn't exist, the failure was a real validation error, so rethrow it. +App creation returns `422` if the name is taken. Your worker will retry after crashes, so treat that as idempotent: `GET /v1/apps/{name}` to confirm the app exists and reuse it. If it doesn't exist, the `422` was a real validation error, so rethrow it. ## Track pool state in a database