P36b+P36c: WAL checkpoints + bucket-backed sc serve — P36 complete

CLAUDE.md

@@ -150,7 +150,7 @@ the code, those win.
 | P33 | Randomized protected sealing (fresh DEK + nonce; `RANDOMIZED` perms bit); dual-read of pre-P33 convergent ciphertext; per-checkout keyed stat cache; `sc rewrap` upgrades convergent blobs at the tip | [0043](docs/adr/0043-randomized-protected-encryption.md) |
 | P34 | Private branches: ref points at a sealed-branch manifest; every commit/tree/blob individually sealed (copy-on-write) under a per-branch KEK wrapped per recipient + escrow; `sc branch --private/grant/revoke/publish`; opaque to non-recipients (content, paths, messages); grant O(1), revoke rotates the KEK; publish replays to public with a scanner gate; git bridge + private→public integration refused; `PROTOCOL_VERSION` 4 | [0044](docs/adr/0044-per-branch-access-control.md) |
 | P35 | Native Tauri desktop browser: opens `.sc` repositories through `scl-repo`, shows local/remote refs, all-parent snapshot DAG + provenance, public trees and first-parent diffs; protected content is locked and private branches remain opaque; no mutation or identity surface | [0045](docs/adr/0045-native-desktop-read-model.md) |
-| P36 | P36a built: bucket WAL remotes (sc+wal://, sc+s3://) — immutable packs + CAS'd manifest, multi-writer safe, no coordinator; checkpoints (P36b) and bucket-backed serve (P36c) pending | [0046](docs/adr/0046-wal-bucket-remotes.md) |
+| P36 | Bucket WAL remotes (sc+wal://, sc+s3://): immutable packs + CAS'd manifest, checkpoints + log-tail cold start, bucket-backed `sc serve --store` with disposable instances | [0046](docs/adr/0046-wal-bucket-remotes.md) |
 
 ## Standing boundaries & gotchas
 
@@ -172,6 +172,9 @@ transport-adjacent. The rest, imperatively:
 - **Bucket remotes hold public content plaintext at rest** — bucket ACL is
   the perimeter (sealed content stays ciphertext, unchanged); partial-clone
   `filter` against bucket remotes is refused.
+- **`sc serve --store` still requires a local serve home with `.sc/`** —
+  tokens, TLS identity, and pack spills live there; the bucket holds all
+  served content.
 - **Protected sealing is randomized since P33.** Pre-P33 convergent ciphertext
   dual-reads forever and stays equality-confirmable forever (rotation ≠
   erasure). Identical independent edits on two branches now genuinely

ROADMAP.md

@@ -930,24 +930,20 @@ scale-&-reach horizon):
 
 - **Bucket compaction/gc (P36a follow-on).** The WAL log (`log/<seq>` keys,
   each a hand-rolled versioned binary `walfmt` entry, not protobuf) grows
-  unboundedly with no compaction — every reader walks the full parent chain
-  back from `head_seq`, and no entry or superseded pack is ever removed.
-  Deferred until checkpointing (below) gives a safe compaction cutoff.
+  unboundedly with no compaction — a cold-start reader now stops at the
+  latest checkpoint (P36b) instead of walking to `0`, but no log entry,
+  checkpoint, or superseded pack is ever actually removed from the bucket.
+  Deferred until a safe pruning cutoff (e.g. "no reader can still need
+  anything before checkpoint N") is designed. Compaction is also what
+  relieves `walfmt::MAX_LIST` (65536): a checkpoint fold with more refs or
+  packs than that cap is skipped outright (`maybe_fold_checkpoint`'s guard,
+  P36b review) rather than writing an object `Checkpoint::decode` would then
+  refuse to read back, so a remote whose live ref/pack count grows past the
+  cap loses folding entirely until compaction can retire entries below it.
 - **Leases (P36a follow-on).** The only cross-writer coordination today is
   the manifest's compare-and-swap; there is no lease/TTL primitive for
   operations that need to hold exclusive intent across more than one bucket
   round-trip (e.g. a long-running compaction). Deferred.
-- **Checkpoint fold (P36b, next).** `walfmt::Manifest.checkpoint_seq`
-  already reserves a field for a future compaction cutoff, but nothing
-  writes a `checkpoints/<seq>/` object yet — cold start is always a full
-  log walk from `head_seq`. Folding the log into a periodic ref snapshot so
-  cold start becomes snapshot + short tail is the next bucket-remote phase.
-- **Bucket-backed `sc serve` (P36c, next).** `sc serve` cannot host a
-  bucket as its backing store today — a bucket remote is written to
-  directly by every client's `sc push`/`sc fetch`, not brokered through a
-  server process. Wiring `BucketTransport` in as an `sc serve` backend
-  (so a bucket remote can also sit behind access control / resource limits
-  the way `.sc/` repos do via P29/P31) is deferred to a follow-on phase.
 - **Partial clone from bucket remotes (P36a follow-on).** `sc clone
   --filter` against `sc+wal://`/`sc+s3://` is refused outright
   (`BucketTransport` has no per-prefix negotiation); teaching the WAL format
@@ -964,17 +960,23 @@ scale-&-reach horizon):
   individually, but a transfer moves a whole pack at a time, so memory use
   is pack-sized, not object-sized. Streaming the S3 request/response bodies
   instead of buffering them whole is deferred.
-- **Incremental `refresh()` for bucket remotes (P36a follow-on).**
-  `BucketTransport::refresh` short-circuits on an unchanged manifest tag,
-  but whenever the manifest *has* changed it re-walks the full parent chain
-  from `head_seq` and re-fetches every `idx_key` on that chain from
-  scratch — O(chain) GETs per manifest change, which gets expensive under
-  fleet-frequency pushes. An incremental refresh that picks up from the
-  last-seen manifest/seq instead of re-walking from scratch is deferred.
+- **Incremental `refresh()` for bucket remotes (P36a follow-on, narrowed by
+  P36b).** `BucketTransport::refresh` short-circuits on an unchanged
+  manifest tag, but whenever the manifest *has* changed it re-walks the log
+  chain from `head_seq` back to the manifest's checkpoint (P36b bounds this
+  to the tail instead of the full chain to `0`) and still re-fetches every
+  `idx_key` in the cumulative pack list from scratch each time — expensive
+  under fleet-frequency pushes even with the checkpoint bound. An
+  incremental refresh that picks up from the last-seen manifest/seq instead
+  of rebuilding the whole index every time is deferred.
 - **Bucket-aware push negotiation (P36a follow-on).** Push negotiation today
   issues a `has_object` round trip per object over S3; batching those probes
   into one `refresh()` plus local index lookups (instead of one S3 round
   trip per object) is deferred.
+- **Serve-side persistent pack cache (P36c follow-on).** A bucket-backed
+  serve instance re-downloads packs per connection; a content-addressed
+  on-disk cache in the serve home would make warm instances cheap without
+  affecting correctness.
 
 ## How a phase gets built
 

crates/cli/src/main.rs

@@ -359,6 +359,13 @@ enum Cmd {
         /// --tls-cert (enforced in run_serve, not by clap).
         #[arg(long)]
         tls_key: Option<PathBuf>,
+        /// Serve a bucket WAL remote (`sc+wal://…` or `sc+s3://…`) instead of
+        /// this repo's own object store (P36c). `<path>` remains the serve
+        /// home: its `.sc/` still provides access tokens, the TLS identity,
+        /// and scratch space — but all served content lives in the bucket,
+        /// making this instance disposable.
+        #[arg(long)]
+        store: Option<String>,
         /// Repo root to serve (the directory containing `.sc/`). Required
         /// unless a `token`/`fingerprint` subcommand is used.
         path: Option<PathBuf>,
@@ -948,6 +955,7 @@ fn main() -> Result<()> {
             tls,
             tls_cert,
             tls_key,
+            store,
             path,
         } => match sub {
             Some(ServeSub::Token { op }) => run_serve_token(op),
@@ -975,6 +983,7 @@ fn main() -> Result<()> {
                     tls,
                     tls_cert,
                     tls_key,
+                    store,
                     path,
                 )
             }
@@ -3476,7 +3485,11 @@ fn run_clone_git(url: &str, dst: &std::path::Path) -> Result<()> {
 /// `--read-only`/`--allow-public` are `--http`-only (P29): `--stdio`
 /// delegates auth/access entirely to ssh, so combining them is refused
 /// rather than silently ignored. `--max-connections`/`--timeout` are also
-/// `--http`-only (P31). `--max-pack-size` applies to both (P31).
+/// `--http`-only (P31). `--max-pack-size` applies to both (P31). `--store`
+/// (P36c) redirects served content to a bucket WAL remote; `path` remains
+/// the serve home (tokens/TLS identity/scratch) either way. A malformed
+/// `--store` URL is validated up front, before any bind — same fail-fast
+/// idiom as `run_remote`'s `BucketUrl::parse` check.
 fn run_serve(
     stdio: bool,
     http: Option<String>,
@@ -3488,8 +3501,12 @@ fn run_serve(
     tls: bool,
     tls_cert: Option<PathBuf>,
     tls_key: Option<PathBuf>,
+    store: Option<String>,
     path: PathBuf,
 ) -> Result<()> {
+    if let Some(url) = &store {
+        scl_repo::BucketUrl::parse(url)?; // fail fast on malformed URLs, before any bind
+    }
     match (stdio, http) {
         (true, None) => {
             if read_only || allow_public {
@@ -3503,6 +3520,24 @@ fn run_serve(
                     "--tls applies only to --http (ssh already provides --stdio's confidential channel)"
                 );
             }
+            // `--http`'s `handle_http_connection` gates on `.sc/` presence
+            // unconditionally (404 before any dispatch, store mode included).
+            // `--stdio` has no such gate upstream — enforce it here, before
+            // touching stdin, so a `--store` session never auto-vivifies
+            // `<path>/.sc/tmp/` under an uninitialized directory (which
+            // `TempServeDir::create_in` would otherwise do via
+            // `create_dir_all`) and never leaves an empty `.sc/tmp/` behind
+            // after teardown (its `Drop` only removes the leaf spool dir).
+            // Local (non-`--store`) `--stdio` needs no separate check here:
+            // `LocalTransport::open` already fails closed on a missing
+            // `.sc/` inside `serve_with_policy` itself.
+            if store.is_some() && !path.join(".sc").is_dir() {
+                anyhow::bail!(
+                    "sc serve --store requires an initialized serve home (run `sc init` in {} first): \
+                     tokens, TLS identity, and pack spills live under its .sc/",
+                    path.display()
+                );
+            }
             let max_pack = max_pack_size.unwrap_or(scl_repo::wire::DEFAULT_MAX_PACK_SIZE);
             scl_repo::wire::validate_max_pack_size(max_pack)?;
             let policy = scl_repo::wire::WirePolicy {
@@ -3512,7 +3547,16 @@ fn run_serve(
             };
             let mut stdin = std::io::stdin().lock();
             let mut stdout = std::io::stdout().lock();
-            scl_repo::wire::serve_with_policy(&path, &mut stdin, &mut stdout, policy)?;
+            match &store {
+                Some(url) => scl_repo::wire::serve_bucket_with_policy(
+                    url,
+                    &path,
+                    &mut stdin,
+                    &mut stdout,
+                    policy,
+                )?,
+                None => scl_repo::wire::serve_with_policy(&path, &mut stdin, &mut stdout, policy)?,
+            }
             Ok(())
         }
         (false, Some(addr)) => {
@@ -3542,6 +3586,7 @@ fn run_serve(
                 allow_public,
                 limits,
                 tls_mode,
+                store.as_deref(),
             )?;
             Ok(())
         }

crates/cli/tests/bucket_remote.rs

@@ -2,8 +2,9 @@
 //! is proven in scl-repo's bucket_transport tests; this exercises CLI
 //! plumbing: remote add validation, push, clone, fetch.
 
+use std::io::BufRead;
 use std::path::{Path, PathBuf};
-use std::process::{Command, Output};
+use std::process::{Child, Command, Output, Stdio};
 
 fn sc(dir: &Path, args: &[&str]) -> Output {
     let mut cmd = Command::new(env!("CARGO_BIN_EXE_sc"));
@@ -18,6 +19,39 @@ fn tmp(tag: &str) -> PathBuf {
     d
 }
 
+/// Spawn `sc serve --http 127.0.0.1:0 <extra…> <path>` and return the child
+/// plus the OS-assigned `host:port` it reports on its first stdout line
+/// (`listening on <addr>`). Copied from `crates/cli/tests/http_remote.rs`'s
+/// `spawn_http_server` — same readiness contract (the announce line prints
+/// only after `TcpListener::bind` returns) — parameterized with `extra` so
+/// this file's tests can pass `--store <url>`.
+fn spawn_http_server_with(root: &Path, extra: &[&str]) -> (Child, String) {
+    let mut args = vec!["serve", "--http", "127.0.0.1:0"];
+    args.extend_from_slice(extra);
+    args.push(root.to_str().unwrap());
+    let mut child = Command::new(env!("CARGO_BIN_EXE_sc"))
+        .args(&args)
+        .stdout(Stdio::piped())
+        .spawn()
+        .expect("spawn sc serve --http");
+    let stdout = child.stdout.take().expect("child stdout is piped");
+    let mut reader = std::io::BufReader::new(stdout);
+    let mut line = String::new();
+    let n = reader
+        .read_line(&mut line)
+        .expect("read serve startup line");
+    if n == 0 {
+        let status = child.wait().ok();
+        panic!("sc serve --http exited before announcing a bound address: {status:?}");
+    }
+    let addr = line
+        .trim()
+        .strip_prefix("listening on ")
+        .unwrap_or_else(|| panic!("unexpected serve startup line: {line:?}"))
+        .to_string();
+    (child, addr)
+}
+
 #[test]
 fn bucket_clone_push_fetch_round_trip_and_url_validation() {
     let a = tmp("a");
@@ -51,3 +85,126 @@ fn bucket_clone_push_fetch_round_trip_and_url_validation() {
         assert!(!d.exists());
     }
 }
+
+/// `sc serve --http --store <sc+wal://…>` serves a bucket instead of the
+/// serve-home's own object store (P36c): a repo pushed straight to the
+/// bucket is clonable through the server, and a push through the server is
+/// visible to a completely separate server instance pointed at the same
+/// bucket (proving durable state lives in the bucket, not the server
+/// process). A malformed `--store` URL must be refused before any bind.
+#[test]
+fn serve_store_serves_a_bucket_and_second_instance_sees_pushes() {
+    let bucket = tmp("srv-bucket");
+    let home = tmp("srv-home");
+    assert!(sc(&home, &["init"]).status.success());
+    let store = format!("sc+wal://{}", bucket.display());
+
+    // seed: a repo pushed straight to the bucket
+    let seed = tmp("srv-seed");
+    assert!(sc(&seed, &["init"]).status.success());
+    std::fs::write(seed.join("f.txt"), b"served from bucket").unwrap();
+    assert!(sc(&seed, &["commit", "-m", "c1"]).status.success());
+    assert!(sc(&seed, &["remote", "add", "origin", &store])
+        .status
+        .success());
+    assert!(sc(&seed, &["push", "origin"]).status.success());
+
+    // malformed store URL refused before binding
+    let bad = sc(
+        &home,
+        &[
+            "serve",
+            "--http",
+            "127.0.0.1:0",
+            "--store",
+            "sc+s3://",
+            home.to_str().unwrap(),
+        ],
+    );
+    assert!(!bad.status.success());
+
+    let (mut child, addr) = spawn_http_server_with(&home, &["--store", &store]);
+    let parent = tmp("srv-clone");
+    let dst = parent.join("d");
+    let url = format!("sc+http://{addr}/repo");
+    assert!(sc(&parent, &["clone", &url, dst.to_str().unwrap()])
+        .status
+        .success());
+    assert_eq!(
+        std::fs::read(dst.join("f.txt")).unwrap(),
+        b"served from bucket"
+    );
+    // push through the server, then read it back via a SECOND instance
+    std::fs::write(dst.join("g.txt"), b"hop").unwrap();
+    assert!(sc(&dst, &["commit", "-m", "c2"]).status.success());
+    assert!(sc(&dst, &["push", "origin"]).status.success());
+    child.kill().ok();
+    let _ = child.wait();
+    let (mut child2, addr2) = spawn_http_server_with(&home, &["--store", &store]);
+    let parent2 = tmp("srv-clone2");
+    let d2 = parent2.join("d2");
+    assert!(sc(
+        &parent2,
+        &[
+            "clone",
+            &format!("sc+http://{addr2}/repo"),
+            d2.to_str().unwrap()
+        ]
+    )
+    .status
+    .success());
+    assert_eq!(std::fs::read(d2.join("g.txt")).unwrap(), b"hop");
+    child2.kill().ok();
+    let _ = child2.wait();
+
+    for p in [&bucket, &home, &seed, &parent, &parent2] {
+        std::fs::remove_dir_all(p).unwrap();
+        assert!(!p.exists());
+    }
+}
+
+/// Regression (P36c review): `sc serve --stdio --store <url> <path>` must
+/// fail closed when `<path>` has no `.sc/` yet, exactly like the `--http`
+/// path's unconditional 404 gate — not silently `create_dir_all` one into
+/// existence via `TempServeDir::create_in`'s spool-dir creation and leave an
+/// empty `.sc/tmp/` behind after teardown. The check runs before any stdin
+/// read, so the child exits immediately on its own (no hang, no need to
+/// feed it a HELLO frame).
+#[test]
+fn stdio_serve_with_store_refuses_an_uninitialized_serve_home() {
+    let bucket = tmp("stdio-uninit-bucket");
+    let store = format!("sc+wal://{}", bucket.display());
+    // `tmp()` creates the directory itself but never runs `sc init` in it —
+    // exactly the "uninitialized dir" this gate must reject.
+    let home = tmp("stdio-uninit-home");
+    assert!(!home.join(".sc").exists());
+
+    let out = sc(
+        &home,
+        &[
+            "serve",
+            "--stdio",
+            "--store",
+            &store,
+            home.to_str().unwrap(),
+        ],
+    );
+    assert!(
+        !out.status.success(),
+        "must refuse an uninitialized serve home: {out:?}"
+    );
+    let stderr = String::from_utf8_lossy(&out.stderr);
+    assert!(
+        stderr.contains("serve home"),
+        "stderr must name the serve home as the problem: {stderr}"
+    );
+    assert!(
+        !home.join(".sc").exists(),
+        "refusing must never auto-vivify .sc/ under the uninitialized home"
+    );
+
+    for p in [&bucket, &home] {
+        std::fs::remove_dir_all(p).unwrap();
+        assert!(!p.exists());
+    }
+}