fix(orchestrator): refuse a create that runs out of memory so placement retries it

[eren23]

Fixes #3689

A create right after a kill can fail with a 500 when the node's hugepage pool holds only one sandbox of that size. Delete returns before the killed Firecracker exits, and that process keeps its hugepages for up to about 0.5 s. The new sandbox then gets ENOMEM when the orchestrator maps its guest memfd. Create returned this as codes.Internal, so placement excluded the node and gave up.

What I changed

1. Create returns codes.ResourceExhausted when the error is ENOMEM. I check it with errors.Is(err, syscall.ENOMEM), not with the message text. Placement already treats ResourceExhausted as a refusal and retries the same node with backoff, so the create lands when the old process is gone.
2. Before it answers with that refusal, Create runs the start rollback and releases the sandbox ID. Placement retries within a millisecond with the same ID, and the normal error path releases the ID only after a rollback in a goroutine. A retry in that window gets AlreadyExists, which excludes the node. At this point the rollback holds only the template pin, so it is quick.
3. loadSnapshot wrapped ctx.Err() while it waited for uffd ready, so a uffd exit after a good snapshot load became a plain context.Canceled. It now wraps context.Cause(ctx), and the ENOMEM reaches Create on that path too.

I did not add a guard for a template that can never fit the pool. With this change such a create retries until the request deadline and ends as a 503. On my host that took 70 s and about 134 placement attempts. The other ResourceExhausted refusals in Create (max sandboxes, too many starting) work the same way, and TestPlaceSandbox_DeadlineEndsRefusalBackoff covers that path. A guard would need the pool size in the server, which it does not have today. The placement hugepage check from 16ec5e6 already skips such a node when it is on. I can add a guard if you prefer one.

How I tested

1. TestLoadSnapshotKeepsUffdExitCause fails on main with context canceled while waiting for uffd ready: context canceled and passes with the change, 30 of 30 runs.

2. TestMemoryRefusal checks that the error chain from the lab log gives ResourceExhausted, and that a plain error with the same text does not.

3. go test -race -count=1 ./pkg/server/ ./pkg/sandbox/fc/ in a linux/arm64 container with Go 1.26.8. All 231 tests pass.

4. golangci-lint run and golangci-lint fmt --diff (v2.14.0) on both packages report no issues.

5. The existing placement tests TestPlaceSandbox_RefusedNodeIsRetried and TestPlaceSandbox_DeadlineEndsRefusalBackoff pass.

6. I built three orchestrators from 7546658, swapped only the binary on one Embed host and ran the same cycles on each. The host runs compose 311f9fc with nested KVM, and its hugepage pool fits one 8 GiB sandbox. The middle column is this change without step 2.

Case	main	without step 2	this PR
6 GiB written, kill, create at once, 40 cycles	0 of 40, 40 × 500	40 of 40	40 of 40
2 GiB written, 30 cycles	1 of 30, 29 × 500	30 of 30	30 of 30
Idle sandbox, 30 cycles	30 of 30	30 of 30	30 of 30
18 parallel workers on 512 MiB sandboxes, 270 creates	240 of 270, 30 × 500	269 of 270, 1 × 500	270 of 270
A second 8 GiB sandbox that never fits, 3 tries	500 in 0.06 s	503 in 70 s	503 in 70 s

The numbers count creates that worked on the first API call. The one 500 without step 2 is the race above. The retry came 4 ms after the refusal and got AlreadyExists. With this PR the 6 GiB case was ready 0.56 s after the kill (median) with no retry in the client. On main the client had to retry after the 500, and it was ready after 1.00 s. After each case all hugepages were free, no Firecracker process ran and the API listed no sandbox, on all three binaries.

[cla-bot]

We require contributors to sign our Contributor License Agreement, and we don't have @eren23 on file. You can sign our CLA at https://e2b.dev/docs/cla . Once you've signed, post a comment here that says '@cla-bot check'

[eren23]

@cla-bot check

[cla-bot]

The cla-bot has been summoned, and re-checked this pull request!