Container Dev Mode: host CLI, embedded registry, and VM push path

[jetm]

Container Dev Mode is the inner dev loop for a containerized app on an
immutable Avocado OS device: edit on the host, and only the changed image
layer is hot-reloaded onto the running device — no reflash, no full re-ship.

This adds the host half:

• a per-project content-addressed OCI store, split write/read listeners, and
  a control WebSocket (three-listener model), all over a per-project pinned CA;
• Basic-gated write token vs Bearer read/control token (split-credential model);
• an engine-driver watcher (docker/podman) that syncs only changed layers and
  selects PUSH vs INGEST by host topology, plus a cross-arch guard;
• avocado container dev up/sync/status/down/prune;
• the authenticated VM push path: on the macOS/VM fast path the guest pushes to
  a routable HTTPS write listener whose per-project CA is delivered at up,
  never baked.

Verification: unit + integration suites (container_dev lib, plus
container_dev_e2e, container_dev_security, container_dev_arch) all pass;
both halves of the loop are validated against a real Avocado OS qemux86-64
target (docker 25.0.9) built from the published 2024/edge feed, booted under
QEMU and reached only over SSH. The authenticated write path passes 8/8, and the
hot-reload loop delivers a baseline and then reloads it with no auth failures.

The target used to be a Debian cloud image standing in for a device. Replacing
it found three defects that stand-in could not surface: bootstrap delivery
shelled out to base64 -d, which Avocado OS does not ship, so up could never
bootstrap the OS this feature targets; the watcher acted on its own retag and
lost its credential when a superseded push outlived its config, producing 1281
failed pushes from one rebuild; and the device restarted its container rather
than the owning unit, so a pulled image was ignored while every layer reported
success. The last of those is why container_dev.images[].service is now
carried on the wire.

Phase 0 de-risk findings are in docs/container-dev/phase0-findings.md; the lab
that produced the above is docs/container-dev/lab/.

[jetm]

Self-review (cold read of my own +10k PR). The security core is clean: TLS drops the CA key (bootstrap carries only CA cert + read token), tokens are 256-bit CSPRNG, the store closes digest/tag path-traversal, every docker/podman call is argv-only (no shell), and the bind addresses match the threat model. Six correctness/robustness findings inline - the two worth acting on first are the recycled-PID signal hazard in down/sync and the cross-arch guard never being wired into the live up path (confirm whether that's intentionally deferred - if so it shouldn't read as complete).

[jetm]

Independent second review of the increment since f69b3e2 (+444/-55, 4 files), traced cold at 2295751. Ten findings inline, two of them high.

The two I would fix before this merges are both about a guard that does not guard:

• dev.rs:365 - the arch guard this diff wires up is fail-open while the arch book is empty, and reconcile never consults the book at all. Since up spawns the watcher before it bootstraps the device, the empty-book window is the normal case, not an edge: a rebuild pushed before the device ever connects gets recorded into desired state and delivered verbatim on the device's first frame, architecture unchecked. The comment right below claims this decorator prevents exactly that.
• dev.rs:436 - SessionLock::acquire runs after the state write, the token mint, all three binds, the watcher spawn, and both SSH deliveries, so it cannot exclude a second up from any of them. Two of its consequences need no unusual configuration at all: a concurrent status/sync/down in the two-statement window makes acquire fail ENOENT (it lacks .create(true)) after everything is already bound and the device is bootstrapped, and because every read path unlinks the lock inode, mutual exclusion does not survive a single down.

Two findings are backed by running the code rather than reading it. Deleting session.touched = Instant::now(); (registry.rs:407) leaves the test suite at 10/10 passing, so patching_a_session_keeps_it_alive_past_the_ttl asserts nothing about the TTL - it trips the workspace CLAUDE.md rule that every test must fail when the implementation is wrong. And the default-config second-up really does die at EADDRINUSE first (mio sets only SO_REUSEADDR), which is why the destructive half of the lock finding is scoped to the two env overrides rather than claimed outright.

One doc contradiction worth fixing on its own: registry.rs lines 62-63 say only the gap between chunks counts against the TTL, while lines 70-72 say bodies are buffered as Bytes before anything inspects them. The second is what the code does, so the first is wrong, and the commit body repeats it as an absolute ("never evicted mid-push").

Traced and cleared, so nobody re-audits: HelloArchBook's derived Clone does share one map (the Arc is cloned, not the BTreeMap), so the book genuinely is shared between ControlServer and the guard - that was my main suspicion and it is fine. The Arc<dyn DeviceArchBook> double-wrap is the dyn coercion, not redundancy. EngineArchProbe::new(...as_ref()) on a temporary Box is sound because new copies out a &'static str. A refusal from the guard does make do_sync_and_notify skip the notify, as its comment claims. require_basic_write wraps the whole router, so an anonymous request causes no large allocation despite the body limit. The loopback-only doc rewrites are accurate in both directions. read_blob to blob_size is behavior-preserving and the digest is still validated. And I dropped one candidate outright: the new test's Instant::now() - TTL - 1s does not panic on a freshly booted host, since Linux Instant is a signed-seconds Timespec - verified on a box with 3,700s uptime.

Scope note: this was the requested light-to-moderate pass - three finder angles plus four verifiers, no separate gap sweep. avocado-cli has no repo-level CLAUDE.md, so the workspace one governs.

[jetm]

Second-reviewer pass on my own PR, cold, on the change since the last round. Twelve findings inline, most severe first. Four of them are the ones I'd land before anything else: the session lock no longer proving pid ownership (1), the arch filter that only covers one of two notify paths (2), the mid-transfer upload session teardown (3), and registry/uploads/ never being swept (4).

Findings 3, 4, 5, 6 and 12 all trace to the same shift in this round: the blob path moved from buffer-in-memory to stream-to-disk. That was the right move, but the invariants that used to fall out of buffering for free - a request either arrived whole or not at all, an abandoned upload died with the process, a blob could only reach disk if it fit in RAM, and a size cap existed - each needed an explicit replacement, and only the manifest path got one.

Findings 9, 10 and 11 are a set: three tests whose doc comments claim a falsifier they do not have. In each case I checked by making the change the comment says would fail, and the suite stayed green. Worth fixing as a group, because together they mean the lock mechanism and up's guard wiring are effectively untested.

Below the cap - confirmed, lower severity

• store.rs:390 - BlobUpload::finish re-implements blob_path (store.rs:326) inline plus write_blob's exists/create_dir_all/persist tail, so the blob layout now has two authors while only readers go through blob_path. blobs_root is also initialized from self.root and then has "blobs" joined onto it, and format!("sha256:{hex}") hardcodes an algorithm two lines above a parse_digest that accepts any.
• store.rs:341 - parse_digest's doc comment is orphaned onto pub struct BlobUpload, so cargo doc renders the new public type's summary as "Split an OCI digest into its (algorithm, hex) components", and parse_digest at store.rs:402 ends up undocumented.
• store.rs:360/:378 - unbuffered write_all per hyper frame means syscall count per GiB is set by link behavior rather than bounded by code; flush() before persist is a documented no-op on File, and with no sync_all a host crash can leave the blob path present while its tail never reached disk, which has_blob and the HEAD dedup probe both report as complete.
• watcher.rs:601 - record_hello has no production callers left (only watcher.rs:962/990 and six sites in tests/container_dev_arch.rs) and plants holders: 1 with no lease, so its entry can never be released. The struct doc at watcher.rs:579 still says reconnects "overwrite" while record_session six lines below says they refcount.
• ws.rs:441 - the inner Option<DeviceArchLease> is never None, since the only Some(..) return arm always supplies a lease, so if lease.is_some() is a branch that cannot be taken.
• ws.rs:1015 - notify_refuses_an_event_with_no_image_id labels frames.is_empty() "The decisive assertion", but an empty desired digest compares equal to the empty running_digest, so that assertion passes with the digest.is_empty() bail deleted. Only the third assertion is decisive. assert!(result.is_err()) also never checks which error.
• tests/container_dev_arch.rs:323 - pinned_ca_connector is now the fourth byte-identical copy (also container_dev_security.rs:131, container_dev_e2e.rs:353, ws.rs:1064), and tests/common/mod.rs already exists as the shared home.
• dev.rs:949 + watcher.rs:740 - two engine spawns per image per manual sync, to read two fields of the same image inspect object, awaited serially per image.

Checked and refuted

Recording these so nobody re-derives them. Digest verification and path traversal in the new write path are sound: finish compares against the incrementally computed hash before any filesystem move, the dedup branch is only reachable after the hash matched, and parse_digest's permissive algorithm parsing is unreachable because a non-sha256: digest fails equality first. Auth is not bypassable - require_basic_write is a from_fn_with_state layer over the whole router, applied outside routing, so it precedes the /v2/ ping and unrouted methods too. No listener moved this round. Secret file permissions are fine: session.lock is 0644 but always empty (it exists only as a stable flock inode), and persisted blobs keep NamedTempFile's 0600. No DeviceArchLease::drop self-deadlock and no lock-order cycle between image_arches and desired. Instant subtraction in the test helpers does not panic on Linux, and the Windows job is cargo check only. The missing -- before image in resolve_image_id is real in isolation but byte-identical to the pre-existing EngineArchProbe::image_arch shape this round does not touch.

No rust.md violation: edition 2021, anyhow + thiserror matching the CLI row, and no [lints] table, rustfmt.toml, clippy.toml or workspace introduced.

[jetm]

One finding, carried over from avocado-os#46 after tracing it to the code that actually owns it. The os#46 side is resolved; this is the residual half, and it partly corrects what I said there.

[jetm]

Second-reviewer pass over the Rust sources of this branch, read cold. Reviewed src/commands/container/dev.rs, all of src/utils/container_dev/, and the src/main.rs / src/commands/mod.rs / src/utils/{mod,config,container,remote,runs_on,runtime}.rs and src/commands/runtime/{build,deploy}.rs hunks the diff touches. Deliberately out of scope: Cargo.lock and everything under docs/ (read only as evidence for what the lab exercises, never reviewed as an artifact), and the device-side agent, which lives in avocado-os#46.

Reviewed at ad111ee; the head has since moved to 2c8e854, which is a docs-only commit (docs/container-dev/verify-vm-write-path.sh, +13/-11). No Rust source changed between the two, so all eleven inline findings are current.

The first three are what I would fix before anything else. Two images sharing a tag silently serve each other's manifest; the TLS leaf has no SAN for the LAN address up actually advertises, so the documented hardware path cannot connect at all; and both are invisible in the lab because setup-lab.sh:361 pins AVOCADO_CONTAINER_DEV_HOST=10.0.2.2, the one address that is in the SAN set.

One finding could not be posted inline because its line is not inside a diff hunk. src/main.rs:2004, needs_vm_routing(): it is a positive allow-list of command variants and Commands::Container is absent, so ensure_routed_for_process never runs for container dev, DOCKER_HOST is never set, and HostTopology::vm_routing is false by default. On macOS or Windows that means sync_mode() returns Ingest instead of Push, and VmWriteSetup::docker, the routable write bind, the TLS write task, and deliver_vm_ca are all skipped. It was never hit because docs/container-dev/verify-vm-write-path.sh:29 requires a manual DOCKER_HOST export. Not a one-line fix: flipping it true makes dev.rs:274 hard-fail up for anyone without AVOCADO_CONTAINER_DEV_VM set. It pairs with the ingest.tar finding below, which is the path this gap sends every non-Linux user down.

On the red Windows compile check: it is not a code problem. Run 30858362570 job 91834557875 died in Install build pre-reqs (.github/workflows/pr.yml:20-23), where choco install nasm cmake -y got a 504 from the V2 feed on cmake.install 4.4.2 and then failed dependency resolution against cmake.install 4.4.0. cargo check --target x86_64-pc-windows-msvc never ran. Every std::os::unix / libc / tokio::signal::unix use this branch adds is properly cfg-gated (flock, SessionLock, session_is_live, the signal handlers, and watcher.rs:483's PermissionsExt block, with the dev.rs test module at :1099 gated #[cfg(all(test, unix))]). A re-run should clear it; pinning the cmake version, or dropping it since the windows-2025-vs2026 image ships CMake, would stop it recurring.

Verified but below the inline cut, in rough severity order: dev.rs:643 down removes session.json while up still holds the flock, so status prints "not running" with all three listeners bound. dev.rs:366 seeds ControlServer with DesiredState::default() and derive_from_watched_tags has zero production callers, so after down/rebuild/up a reconnecting device reconciles against an empty map and silently keeps the stale image. collect_garbage's doc claims invocation from down and no such caller exists, so one orphaned manifest plus config plus changed layer accumulates per rebuild until a manual prune. dev.rs:71 hardcodes DEFAULT_CONFIG and the ContainerDevCommands variants take no -C, so this is the one command family that cannot be pointed at a non-default config. registry.rs:610 never compares a digest-shaped reference to the computed digest (spec wants 400 DIGEST_INVALID; low reach, real clients compute it themselves). registry.rs:458 ignores Content-Range and there is no 416 path anywhere on the write listener, so a same-Location retry doubles the bytes and surfaces as a 400 at the final PUT. registry.rs:654's .max(start) reports one byte too many for a zero-length chunk; deleting the clamp gives the correct answer. config.rs:43 makes ContainerDevImage::service required while nothing but tests reads it, and setup-lab.sh:297 states outright that nothing delivers it to the device. runs_on.rs:445 adds an unconditional print_failure_notice on a path whose contract is "returns None if it failed", so a benign probe now dumps up to 15 lines of remote stderr over the TUI mid-build.

Two latent ones worth knowing about rather than fixing now: bootstrap.rs:141 write_bootstrap writes the Bearer token at the ambient umask, but it has zero production callers (the live SSH path uses umask 077 with three tests pinning 0600); and watcher.rs:434 puts the write token on podman's argv where run_engine interpolates argv into an anyhow context printed with {e:#}, but dev.rs:401 hardcodes let engine = DEFAULT_ENGINE and nothing selects podman. Both are one line from mattering.

Test-integrity offenders, since CLAUDE.md is explicit here: registry.rs:1731 a_failed_chunk_stream_leaves_the_session_resumable asserts only status codes and map length, never issues a PUT, and never compares a digest, so it passes with resume broken. tls.rs:352 discards server_config() into _config and asserts a substring on the CA cert, so swapping the leaf's key for the CA's still passes. config.rs:130 compares a const to a literal and exercises no code. Against ~178 new tests these are the exceptions, not the pattern.

Cleanup, not defects: four stale #[allow(dead_code)] module attributes in container_dev/mod.rs that suppress nothing (checked with --force-warn dead_code) plus ~20 lines of "wired by a later task" rationale for tasks that have landed; image-ref normalization implemented three times (watcher.rs:287, watcher.rs:332, ws.rs:135) with the copies already diverged, which is the arch-book finding below; hex encoding hand-rolled with a format! per byte in four places when utils::jcs::hex_encode exists; resolve_endpoint formats host:port and dev.rs immediately re-parses it twice; two docker image inspect subprocesses per image per manual sync where one --format '{{.Id}} {{.Architecture}}' would do.

Things I chased that do not hold, named so nobody re-raises them: non-constant-time token comparison (the doc comment justifies via TLS plus a single-developer host and never claims unreachability); write_blob trusting a client-supplied digest (its one production caller computes it server-side); the docker config.json write-then-chmod window (the parent is a 0700 tempdir_in); the minted CA being dangerously unconstrained (docker scopes it per-registry via certs.d/, and nothing installs it in a system trust store); RecvError::Lagged dropping sync frames (tokio advances to the oldest retained message, so the newest idempotent Sync still lands); and IPv6 endpoint mis-splitting (rsplit_once(':') handles both forms, and both listeners bind IPv4-only anyway).

Verified clean: no second rustls crypto provider is linked, no let _ = on a lock guard anywhere in the diff, no danger_accept_invalid_certs, error handling matches the crate's existing anyhow-plus-thiserror split, auth layers cover every route on both routers with WS auth checked before the upgrade and no query-string token path, and RuntimeConfig has no deny_unknown_fields so container_dev cannot break an existing config parse.

[jetm]

Verified all twelve findings from review 4849090587 against the pushed head 2badb34. Eight are fully closed and I have nothing to add on those: the ingest bail, the begin_pull deletion (the flock via session_is_live is the cross-process proof the finding wanted, and all five overstating doc comments were rewritten), sweep_uploads following from the same change, early signal registration, the image_ref module collapsing the three normalization copies, the stray-.tmp skip, and the serve_tls backoff. On that last one I checked the other accept loop at ws.rs:400 that still has the else { return; } shape - it is #[cfg(test)]-gated with no production caller, so it is not a live second instance.

Three are partial and one was not fixed. Seven inline; the eighth cannot anchor.

needs_vm_routing() still omits Commands::Container. src/main.rs is not in the diff; main.rs:2004-2030 still lists fifteen variants plus Commands::Connect { command: ConnectCommands::Upload } and no Container arm. This one lived in the review body rather than a thread, so it had no thread to answer, and there is no PR-level comment either. The coupled side moved instead - ingest now bails loudly rather than writing a tar nobody reads - which is a genuine improvement, but it turns the gap into a misleading remedy string rather than closing it. Detail on the watcher.rs thread.

Test integrity. All three named offenders are untouched. a_failed_chunk_stream_leaves_the_session_resumable (registry.rs:1738) still asserts only status codes and map length, never issues a PUT, never compares a digest. mint_builds_a_server_config_from_the_leaf (tls.rs:436-445) still discards server_config() into _config and asserts on the CA cert PEM, so swapping the leaf's key for the CA's still passes - only the mint call signature was threaded through. config.rs is not in the diff at all, so default_registry_port_is_not_5000 still compares a const to a literal.

I reasoned about the revert edit for each new test rather than assuming they bite. Most do: the two-repository tag test, the slash-key test, the GC tag-walk test, all three SAN tests, the arch-recording test, the store-side prune-refusal tests, and the stray-.tmp test all fail on a coherent revert. Three do not, and they are worth knowing about. a_registry_qualified_ref_is_still_arch_filtered_on_the_broadcast pins the recording side only - the frame's reference is already my-app:dev, so canonical(&reference) is the identity there and reverting ws.rs:543 leaves it green. Reverting dev.rs:705 to a hardcoded SessionActivity::Idle leaves everything green, so the wiring of the flock into prune is unpinned even though the store-side check is well covered. And the GC size-guard test is a new offender in its own right - detail inline. Findings 7, 11, 12, and 4 got no test at all, so reverting any of those four is silent.

CI is green, including Windows compile check - run 30864480584, job 91853231339, SUCCESS, so the previous red really was the Chocolatey 504 and a re-run cleared it. The workflow itself was not changed, so .github/workflows/pr.yml:20-23 still runs the unpinned choco install nasm cmake -y and the same flake can recur. Worth pinning the version, or dropping cmake since windows-2025-vs2026 ships it.

On thread resolution. All 40 threads report resolved, including all eleven from that review, each with a substantive reply. Three of those resolutions are premature given the partials above. Two of the replies are accurate about what was done but silent on what was not - the migration question goes unmentioned, and the lab gap is acknowledged as the reason the bug was invisible without either fixing it or stating it as a deferral. One small correction: the tag reply says reverting tag_path to the flat form fails ten tests. That holds only for an incoherent partial revert with the nested list_tags still in place; a coherent revert of both fails two. Still falsifiable, just not ten.

Everything above is from reading the head plus the diff between the two SHAs. I did not run the suite myself - CI's green is the evidence for that - and no real pinned-CA handshake from a device was performed, which is what would actually settle the SAN fix.

[jetm]

Second-reviewer pass on the incremental change since the last review. It touches only docs/container-dev/lab/demo.sh, no Rust, so this is a shell review.

One substantive finding: the version assertion the new comment describes as airtight is vacuous for dev-shaped versions, because write_ctx puts image=my-app:dev into the same log line the check substring-matches against. So demo.sh all dev prod passes whatever the target is running. That is the same false-pass class the rest of this revision is fixing, so it is worth closing here rather than later.

Two smaller ones: reading /version by executing the image makes a build-only step require host-executability, which breaks the cross-arch flow the file's own header advertises and then emits an error telling you to redo the step that just worked; and seed is still missing from the help block whose signature this change alters.

The : >"$UP_LOG" addition is a bigger correctness win than its comment claims - it also closes a stale-log false-pass in the up gate. Detail in the third comment, along with an AVOCADO_BIN quoting inconsistency at :463 that is one way to trigger it.

All 8 checks are SUCCESS on fdfcf86, but none is required, so ci_conclusion is none and CI is not evidence the lab script works. Behavioral proof pending: an actual demo.sh all <v1> <v2> run against a target, which is exactly what finding 1 says the current assertion would not catch anyway.