[ice-zeus]

Closes #26

Replaces #46, whose commits carried attribution lines that maestro-policy rejects. Pushed history is never rewritten, so this branch starts again from main.

What and why

The 11 store tests in src/store_tests.rs need a real Redis (REDIS_TEST_URL), and CI never ran them. Now every rust-test part runs them, and no test run reaches an image registry: one of three runs of the first design (pull from Docker Hub per run) failed on a 60 s Docker Hub timeout (https://github.com/shifter-io/web-proxy/actions/runs/42838989632).

• tests/redis-image.sh is the one pin: the redis:7.4-alpine index sha256:858f009f…3499 (linux/amd64 manifest sha256:ca0acbb137c1…), the Docker Hub and public.ecr.aws repository names and the local name web-proxy-ci/redis:7.4-alpine.
• .github/ciod/prefetch.sh (new, the repository's ciod hook; nothing in Maestro changes): the nightly CI image build runs it without Docker. It fetches the index, the linux/amd64 manifest, its config and layers through registry API v2 with an anonymous token, Docker Hub first and public.ecr.aws only if Docker Hub fails, one attempt each. Every body's sha256 is checked against its digest. It writes $CI_CACHE_DIR/web-proxy/tools/redis-<index digest>.tar (an OCI image layout) through a temp folder and a rename, plus a .complete marker. It prints redis image: prefetched <path> or redis image: installed now <path>.
• .github/scripts/ci-run.sh: rust-test runs the hook (the fallback for server images built before this merges: one download and a ::warning::), docker loads the archive and runs REDIS_IMAGE=web-proxy-ci/redis:7.4-alpine REDIS_PULL=never sh tests/allowance-redis.sh cargo nextest run … --run-ignored all. The job summary gets the redis image: line. Nothing retries: a failed download, a digest mismatch, a failed load or a Redis that does not start fails rust-test.
• tests/allowance-redis.sh keeps its interface. It reads the pin, takes REDIS_IMAGE and REDIS_PULL (defaults: the pinned image, --pull missing, as before) and logs its docker run line. One fix that rootless Docker needs: the image hands /data to its redis user, which under rootless Docker is another host user, so cleanup could not remove the fixture folder from a sticky /tmp (the part failed with 42 tests green, https://github.com/shifter-io/web-proxy/actions/runs/42837209874). The script now mounts a subfolder.
• ENGINEERING.md (regenerated docs/engineering.html) and the ci.yml header say CI runs the store tests and where the image comes from. #[ignore] is only for tests that need REDIS_TEST_URL.

Tests

• tests/ciod-prefetch.test.mjs (new) runs the hook against fake local registries (token, index, manifest, config, layer) through REGISTRY_URL, MIRROR_URL and AUTH_URL:
  • the first run prints installed now and writes a verified archive; the second prints prefetched and makes no request;
  • a digest mismatch fails, and nothing is left behind;
  • a mismatch on Docker Hub falls back to the mirror;
  • with Docker Hub down, the mirror is used;
  • with both down, it exits non-zero and leaves no partial file.
• tests/ci-run.test.mjs runs the real allowance-redis.sh with a docker stub and a stub hook. It checks docker load, --pull never --network none with no -p, the local image, REDIS_TEST_URL for nextest, the container removed after a failing test, the summary line, the warning on installed now, and that a failed download, load or Redis start fails rust-test before any test runs.
• On a disposable server (Docker 29.1.3), all 19 tests of both files passed. The hook ran twice the way the image build runs it (CI_CACHE_DIR and HOME in a temp folder): installed now, then prefetched, both exit 0, a 16 MB archive. docker load gave Loaded image: web-proxy-ci/redis:7.4-alpine, and allowance-redis.sh started Redis from it with --pull never --network none.

Coverage

Suite	main (merge queue run 42836258936)	This PR
Rust tests (nextest)	31 passed, 11 skipped	42 passed, 0 skipped
Node tests	235 passed	246 passed (11 new: 6 in ci-run.test.mjs, 5 in ciod-prefetch.test.mjs)
Everything else	unchanged	unchanged

Proofs that the store tests cannot be skipped silently

Throwaway branches, each one change on this PR's head 130b5c4; maestro-prune-branches removes them after the issue closes.

• Redis start broken (task/26-proof-redis-start, redis-server --no-such-option): https://github.com/shifter-io/web-proxy/actions/runs/42842718650. tier1 red, only rust-test failed ("Redis fixture did not become ready").
• Failing assert in one store test (task/26-proof-assert): https://github.com/shifter-io/web-proxy/actions/runs/42842725819. tier1 red, only rust-test failed: 42 run, 41 passed, 1 failed, 0 skipped.
• Wrong digest in tests/redis-image.sh (task/26-proof-digest, last hex digit changed): https://github.com/shifter-io/web-proxy/actions/runs/42842732984. tier1 red, rust-test failed. A digest that names nothing gets 404 from both registries, so the hook ends with "redis image: download failed from https://registry-1.docker.io and https://public.ecr.aws". The "digest mismatch" path (a registry sending other bytes) is covered by ciod-prefetch.test.mjs. node-test is red too, since ci-run.test.mjs checks the pinned archive name.
• No registry at test time: every run's rust-test log shows redis fixture: docker run --pull never --network none web-proxy-ci/redis:7.4-alpine after Loaded image: web-proxy-ci/redis:7.4-alpine (ci-timing artifact, parts/rust-test.log).

Stability and times (ci-8cpu, head 130b5c4)

Five dispatch runs, all green with the same counts (Rust 42 passed / 0 skipped, Node 246 passed), plus the pull-request run:

Run	Mode	Parts step	rust-test	clippy
https://github.com/shifter-io/web-proxy/actions/runs/42842741653	full	30 s	30 s	23 s
https://github.com/shifter-io/web-proxy/actions/runs/42842752776	full	30 s	30 s	23 s
https://github.com/shifter-io/web-proxy/actions/runs/42842767274	full	30 s	30 s	23 s
https://github.com/shifter-io/web-proxy/actions/runs/42842778845	full	32 s	32 s	25 s
https://github.com/shifter-io/web-proxy/actions/runs/42842785405	full	32 s	32 s	25 s
https://github.com/shifter-io/web-proxy/actions/runs/42842859056 (PR)	fast	31 s	31 s	24 s

Today's server images predate the hook, so these runs took the fallback (redis image: installed now, one download, with the warning). rust-test is now the longest part, about 7 s longer than clippy, and the parts step stays well under 90 s. After merge, the first tier1 run on a server image built after the merge should show redis image: prefetched.

Notes

• .github/scripts/changed-scope.sh does not count tests/ as Rust input, so a pull request that changes only tests/redis-image.sh skips rust-test. The merge queue runs every part, so a bad pin still cannot merge. That file is outside this issue, and Merge-queue drill: two PRs that pass alone and break together #27 touches it.

Size: about 125 production lines (the hook 90, ci-run.sh 32, ci.yml 4).

[ice-zeus]

Review of #49 at 130b5c4 (closes #26).

Blocking

None.

Checked: the hook verifies the index against the pin before it reads any digest from it, and verifies each manifest and blob before using it, so registry data can never choose a path or name an unpinned blob. The archive is built in a temp folder inside tools/ and renamed into place, with the .complete marker written last. rust_test runs as a plain part command, so set -e applies: a failed hook, docker load or Redis start stops the part before nextest runs, and tests/ci-run.test.mjs covers each of these. allowance-redis.sh keeps --network none, the Unix socket and no published port. $fixture_dir stays 0700 and only the mounted subfolder is 777. The store tests use random visitor ids, so running them in parallel against one Redis is safe. Tests cover the new behavior: 5 hook cases against fake registries and 6 ci-run cases. Size and file limits are within RULES.md. tier1 and maestro-policy are green on this head.

Notes

1. NON-BLOCKING: The prefetched check (.github/ciod/prefetch.sh:28) looks only at the marker and the archive file. That is fine because the archive appears only through a rename, but a test where .complete exists and the archive is missing would pin down that the hook downloads again.
2. NON-BLOCKING: When the pin changes, the old redis-<digest>.tar (about 16 MB) stays in tools/. It only matters if that cache lasts across nightly image builds.

Existing

• .github/scripts/changed-scope.sh does not count tests/ as Rust input, so a pull request that changes only tests/redis-image.sh or tests/allowance-redis.sh skips rust-test. The merge queue still runs it, and the PR body already says so (#27 touches that file).