diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index ddf1f63..ccefc9a 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -122,6 +122,35 @@ jobs: python-version: ${{ matrix.python }} - run: python -m unittest discover -s tests + # The real stack, nothing faked: Postgres, differently-server and an + # OpenSandbox server on the Docker runtime, driven through the public API + # (test/e2e). Sandboxes are containers here, so this proves orchestration, + # not microVM isolation; tests for what this runtime lacks skip themselves. + # The approve job waits for it: it passed twice in a row before it was added. + e2e: + name: end-to-end (docker runtime) + runs-on: ubuntu-latest + timeout-minutes: 30 + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-go@v5 + with: + go-version-file: go.mod + - name: build the server image and pull the sandbox images + run: make e2e-prepare + - name: start Postgres, OpenSandbox and the server + run: make e2e-up + - name: end-to-end tests + run: make e2e + - name: stack logs + if: failure() + run: | + docker compose -f test/e2e/compose.yaml ps -a + docker compose -f test/e2e/compose.yaml logs --no-color --tail 400 + - name: stop the stack + if: always() + run: make e2e-down + helm: name: helm chart runs-on: ubuntu-latest @@ -195,7 +224,7 @@ jobs: # again. It approves what CI checked, not what a reviewer read. approve: name: approve when CI passes - needs: [go, go-experimental, postgres, sdk-typescript, console, sdk-python, helm, images, dco, release-source] + needs: [go, go-experimental, postgres, e2e, sdk-typescript, console, sdk-python, helm, images, dco, release-source] if: >- always() && github.event_name == 'pull_request' && diff --git a/CHANGELOG.md b/CHANGELOG.md index c7c3f69..4c6b152 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,12 @@ All notable changes are recorded here. The format follows runs them locally. They used to run only on the memory store, which production does not use. The first run found no defect. A run that is meant to use Postgres now fails, instead of skipping, when it has no database. +- CI runs the end-to-end suite (`test/e2e`) on every pull request: Postgres, + `differently-server` and an OpenSandbox 1.2.0 server on the Docker runtime, + driven through the public API. It had only been run by hand, and not at all + for 0.1.0-alpha.3; its first CI run, on that release's code, passed 16 + tests and skipped the two that need a desktop image or a microVM pool. The + Kubernetes and Fast Sandbox runtimes are still run by hand. ## [0.1.0-alpha.3] — 2026-11-10 diff --git a/Makefile b/Makefile index dd31e26..e4fcbc8 100644 --- a/Makefile +++ b/Makefile @@ -94,6 +94,7 @@ e2e-prepare: docker build -q -f build/server.Dockerfile -t differently-server:e2e . >/dev/null docker pull -q opensandbox/execd:release-1.2.0 >/dev/null docker pull -q debian:bookworm-slim >/dev/null + docker pull -q python:3.12-slim >/dev/null e2e-up: docker compose -f $(E2E_COMPOSE) up -d --force-recreate --wait diff --git a/docs/maintaining.md b/docs/maintaining.md index dcb5c5a..be7c9a7 100644 --- a/docs/maintaining.md +++ b/docs/maintaining.md @@ -56,11 +56,13 @@ the one before; repeating a rung for reassurance adds nothing. | Postgres store | `make test-postgres` (Docker, or `DIFFERENTLY_TEST_POSTGRES_URL` for a database you already run): the conformance suite and the server acceptance scenarios | Changes in `internal/store`, `internal/server` or `internal/scheduler` | | Recorded runtimes | part of `go test` (`TestReplayRecordings`) | Any OpenSandbox adapter change: real responses from the Docker and Kubernetes runtimes, no server | | Live adapter | `OPENSANDBOX_URL=… go test -tags live -run TestLiveOpenSandbox ./internal/adapters/opensandbox/`; for the managed providers `DAYTONA_API_KEY=… go test -tags live -run TestLiveDaytona ./internal/adapters/daytona/` and `FREESTYLE_API_KEY=… go test -tags live -run TestLiveFreestyle ./internal/adapters/freestyle/` (a few small sandboxes, all deleted) | A new OpenSandbox call or version; re-recording; any Daytona or Freestyle adapter change | -| Real stack, local | `make e2e-prepare e2e-up e2e` | Behaviour across the API, scheduler, store and a real OpenSandbox server | +| Real stack, local and in CI | `make e2e-prepare e2e-up e2e`; CI's `end-to-end (docker runtime)` job runs the same on every PR | Behaviour across the API, scheduler, store and a real OpenSandbox server on the Docker runtime | | Real stack, a real cluster | the e2e suite pointed at an install ([test/e2e](../test/e2e/README.md#run-against-another-deployment)) | Helm, NetworkPolicy, the Kubernetes and Fast Sandbox runtimes, upgrades | -CI runs the first four on every PR. The live, e2e and cluster rungs need a -server and are run by hand; record what ran, where and with which versions in +CI runs the first four on every PR, and the real stack on the Docker +runtime. The live and cluster rungs, and the e2e suite on the Kubernetes and +Fast Sandbox runtimes, need a server and are run by hand; record what ran, +where and with which versions in [test/e2e/README.md](../test/e2e/README.md) and [test/opensandbox/README.md](../test/opensandbox/README.md), and say in the PR what did not run. diff --git a/test/e2e/README.md b/test/e2e/README.md index de64b54..ddd5f05 100644 --- a/test/e2e/README.md +++ b/test/e2e/README.md @@ -32,6 +32,10 @@ the trust report. ## Run +CI runs this suite on every pull request (the `end-to-end (docker runtime)` +job in `.github/workflows/ci.yml`), with the stack below on a GitHub-hosted +Linux runner. To run it yourself: + ```bash make e2e-prepare # build differently-server:e2e, pull execd and the sandbox image make e2e-up # Postgres, OpenSandbox (:18091), server (:18090), metrics (:19091) @@ -115,6 +119,7 @@ them precisely instead. | 2026-10-28 | k3s v1.36.5 on an Azure `Standard_D8s_v6`, x86_64; Differently from its Helm chart with embedded PostgreSQL | server, controller and `execd` release-1.2.0, Kubernetes runtime, no snapshot registry | 12/12 passed (pause and snapshots checked as refusals); sandboxes running in about 1.4–2.7 s; an unpullable image retries after OpenSandbox's 60 s pod-ready timeout with the pool healthy | | 2026-10-29 | same Azure VM; pool `opensandbox-microvm` (`runtime: fastsandbox`), template `tpl-1cbcfa94` (1 vCPU, 1 GiB, guest kernel 6.18.36+) | server, Fast Sandbox and `execd` release-1.2.0 | **Partial run**, stopped when the VM was deallocated. Sandboxes running in about 1.3 s; files and pause/resume passed (pause about 52 s for 1 GiB); sandbox env and a template other than the pool's are refused precisely (the tests now expect that). Fork failed with a bare 422 from OpenSandbox on restore, not yet diagnosed. Port URL, idle wake, memory kept and the desktop stream did not run | | 2026-10-29 | Azure, two-node k3s (`Standard_D8s_v6` + `Standard_D16s_v6` KVM node), Postgres Flexible Server, two server replicas | server 1.2.0, Fast Sandbox with a 6.18.36 guest kernel with nftables; pool `runtime: fastsandbox` with OpenWork's Workbot desktop template | 14/14 run passed (desktop, unpullable-image, idle-sweep and switching tests not run); pause about 3–4 min for the 4 GiB desktop, idle wake by a port request 2.3 s | +| 2026-11-10 | GitHub-hosted runner, Ubuntu 26.04, x86_64 (CI job `end-to-end (docker runtime)`) | server 1.2.0, `execd` release-1.2.0, Docker runtime | 16 passed, 2 skipped (no desktop image; no microVM pool), in about 2 minutes; the first run on the code of `v0.1.0-alpha.3` and the first on x86_64 with the Docker runtime | Not covered yet: