Fix checkpoint publication and recovery boundaries

.github/workflows/ci.yml

@@ -138,6 +138,15 @@ jobs:
       - name: Watchdog Lua CM e2e
         run: just test-watchdog-e2e
 
+      - name: Upload E2E failure logs
+        if: ${{ failure() }}
+        uses: actions/upload-artifact@v6
+        with:
+          name: rollups-e2e-logs-${{ github.run_attempt }}
+          path: tests/e2e/results/*.log
+          retention-days: 7
+          if-no-files-found: ignore
+
   watchdog-docker:
     name: Watchdog Docker image smoke
     runs-on: ubuntu-latest

README.md

@@ -167,6 +167,12 @@ Most queue sizes, polling intervals, and safety limits are now internal runtime
 
 ## API
 
+JSON `sender` fields in successful `POST /tx` responses and WebSocket messages
+use EIP-55 checksum casing. Address fields in `/history` and `sender` fields in
+`/historical-l1-inputs` use lowercase hex. Clients must compare decoded 20-byte
+addresses and use one normalized encoding for account or projection keys across
+these routes.
+
 ### `POST /tx`
 
 Request shape:
@@ -234,6 +240,8 @@ After each successfully applied input at offset `X`, persist the claim with
   `HISTORY_UNAVAILABLE`, or `AHEAD_OF_HEAD`. Rebootstrap on a history mismatch.
 - A claim exactly at the head waits for the next input. Replay uses bounded
   pages and queues, with no total catch-up limit. The subscriber cap is `64`.
+- Before upgrade, capacity exhaustion returns `429 OVERLOADED`; shutdown or an
+  operational subscription failure returns `503 UNAVAILABLE`.
 - Messages are JSON text frames; binary fields are `0x`-prefixed hex.
   Direct-input `block_timestamp` values are Unix seconds.
 - Batch envelopes are absent. Offsets count executed application inputs,
@@ -406,8 +414,23 @@ and `X-Executed-Input-Count`, selected atomically with the artifact lease.
 Streaming holds the lease until the response ends or the client disconnects.
 The accepted endpoints return `404` until a comparable checkpoint exists:
 genesis is comparable at block zero; a rebuilt baseline is restorable but only
-a later accepted batch establishes a comparison point. Divergence blocks
-publication of the accepted checkpoint. See [snapshot lifecycle](docs/snapshots/lifecycle.md).
+a later accepted batch establishes a comparison point. Known divergence makes
+all three finalized endpoints return `503 UNAVAILABLE`, including conditional
+state requests. The check shares the checkpoint-selection transaction, before
+any lease or archive is created. See [snapshot lifecycle](docs/snapshots/lifecycle.md).
+
+### Health probes (internal only)
+
+- `GET /livez` returns `200` whenever the handler responds, with an empty body.
+- `GET /readyz` returns `200` while the inclusion-lane receiver is open and
+  shutdown has not been requested; otherwise `503`. Its body is empty.
+- `GET /healthz` uses the same status as `/readyz` and returns JSON:
+  `{ "status": "ok", "inclusion_lane": "ok" }`. `status` becomes `"degraded"`
+  for either failure condition; `inclusion_lane` becomes `"stopped"` only when
+  its receiver is closed, so it can remain `"ok"` during shutdown.
+
+These probes cover process reachability, the lane channel, and shutdown state.
+They do not certify L1 freshness, submitter balance, or canonical agreement.
 
 ## Storage Model
 

docs/invariants.md

@@ -80,12 +80,12 @@ by writer and are write-once (`0001_schema.sql`).
 |---|---|
 | inclusion lane | `batches` (insert + `sealed_at_ms`), `frames`, `user_ops`, `application_inputs`, `dumps`/`snapshots` (batch close) |
 | input reader | `safe_inputs`, `l1_safe_head`, `safe_accepted_batches`, `canonical_divergence` (the divergence poison marker) |
-| recovery (startup) | `batches.invalidated_at_ms`, Tip reopen, current `application_inputs` suffix deletion |
-| history metadata (setup/recovery) | `history_state` — complete era/application-count/L1-block baseline; generation advance and immutable preserved-prefix cut in a non-empty standard-recovery cascade |
+| recovery (startup) | `batches.invalidated_at_ms`, Tip reopen, current `application_inputs` suffix deletion and replacement direct-input rows |
+| history metadata (setup/recovery) | `history_state` — complete era/application-count/L1-block baseline and generation; `history_generation_cuts` — immutable preserved-prefix cuts written with non-empty standard-recovery cascades |
 | batch submitter and mempool flusher | `wallet_nonce_watermark` — deliberately shared under one protocol: each raises it before its first broadcast (write-before-broadcast, I14) |
 | egress (HTTP) | `dumps.lease_count` (leases); `run`'s startup hygiene resets it to zero as the crash backstop |
-| setup | `deployment_identity` (pinned once), `batch_tree_anchor` (the root nonce, frozen once setup completes), the initial `dumps` + `snapshots` rows (genesis or rebuild registration, atomic with the complete history baseline), the `setup_complete` fact (written once), `batch_policy.log_gas_price` + `log_gas_price_updated_at_ms` (first write; Fixed and Uniswap) |
-| snapshot GC (the lane after reconciliation, `run`'s startup hygiene) | unreferenced `dumps` row deletion (`gc_unreferenced_dumps`) |
+| setup | `deployment_identity` (pinned once), `batch_tree_anchor` (the root nonce, frozen once setup completes), the initial `dumps` + `snapshots` rows and rebuild root `batches`/`frames` (atomic with the complete history baseline), the `setup_complete` fact (written once), `batch_policy.log_gas_price` + `log_gas_price_updated_at_ms` (first write; Fixed and Uniswap) |
+| snapshot GC (the lane after reconciliation, `run`'s startup hygiene) | obsolete `snapshots` and unreferenced `dumps` row deletion (`gc_unreferenced_dumps`), including a superseded baseline artifact |
 | command brackets (run, setup, flush) | `terminal_faults` (append-only, best-effort at settlement) |
 | admin | `batch_policy` alpha knobs (`log_alpha`, `log_one_plus_alpha`) |
 | fee oracle | `batch_policy.log_gas_price` + `log_gas_price_updated_at_ms` (Uniswap mode only; stamps on every successful refresh) |
@@ -224,9 +224,11 @@ by writer and are write-once (`0001_schema.sql`).
   remedy is cockroach recovery.
 - **Completeness boundary:** the check completely enforces the accepted-batch
   identity predicate above; it is intentionally not a general canonical/application
-  divergence oracle. It trusts collapsed history below the anchor and the
-  checkpoint application state, shares `scheduler_accepts` (including its
-  documented self-trust omissions), and does not independently detect bugs in
+  divergence oracle. The entire L1 prefix through baseline block `C` is opaque,
+  including previously rejected future-nonce batches; the check trusts the
+  checkpoint state and continuation nonce instead of reinterpreting that prefix.
+  It shares `scheduler_accepts` (including its documented self-trust omissions),
+  and does not independently detect bugs in
   direct-input/user-op execution. A wrong-high cockroach checkpoint nonce is a
   known example that can escape it. Absence of the marker therefore does not
   prove global agreement. Conversely, a structurally malformed foreign landing
@@ -253,9 +255,11 @@ by writer and are write-once (`0001_schema.sql`).
   boundary selects external directs by the setup-pinned submitter address;
   only those inputs and included user ops enter `application_inputs`.
 - **Enforced by:** classified direct reads and complete receipt validation at
-  append. Startup/recovery derive the initial direct rows before catch-up,
-  which must execute them successfully before admission. Replay and WS need
-  no envelope filter because every row executes.
+  append. Standard startup recovery attributes undrained directs to the new Tip;
+  lane catch-up executes them before processing queued user operations. Manual
+  rebuild represents the folded prefix through `C` in its baseline snapshot,
+  with no application-history rows for that prefix. Replay and WS need no
+  envelope filter because every row executes.
 - **Depended on by:** application replay and replicated state correctness.
 
 ### I12. Safe head advances only on real observation; `synced_at_ms` is genuine progress time
@@ -341,9 +345,12 @@ by writer and are write-once (`0001_schema.sql`).
   conflicting batch-tree writes; the detector and next typed read
   stop the process. A chunk committed before either runtime observation may
   acknowledge and later roll back.
-- **Watchdog boundary:** the freeze blocks accepted-checkpoint publication before the
-  offending landing becomes a comparable sequencer checkpoint. Because the
-  watchdog skips replay when the finalized inclusion block is unchanged, it
+- **Watchdog boundary:** accepted-checkpoint selection checks for divergence
+  in the same transaction as selection and any download lease, refusing while
+  the marker is present. A matching batch
+  before a divergent acceptance in the same L1 block cannot represent that
+  block's final state. Because the watchdog skips replay when the finalized
+  inclusion block is unchanged, it
   does not subsume this wire-identity detector. Conversely, the check does
   not subsume the watchdog's broader independent application-state
   comparison.
@@ -429,10 +436,14 @@ by writer and are write-once (`0001_schema.sql`).
   before replacement directs. The entire transition commits in the cascade
   transaction. Clean restart changes neither token. Every intervening cut is
   required to authorize reusing a checkpoint from an older generation.
-- **Enforced by:** `complete_baseline_setup`, immutable history triggers,
-  exact-`+1` generation trigger, and `cascade_and_reopen`.
-- **Depended on by:** mandatory snapshot-derived WS claims. Identity is validated
-  before the requested count, including for empty history.
+- **Enforced by:** `complete_baseline_setup`, immutable baseline and
+  `history_generation_cuts` triggers, the exact-`+1` generation trigger requiring
+  its cut, and `cascade_and_reopen`. `preserved_input_count_in` asserts that
+  every intervening generation has a cut before computing compatibility.
+- **Depended on by:** mandatory snapshot-derived WS claims and `/history`
+  checkpoint compatibility across standard recoveries. Identity is validated
+  before the requested count, including for empty history. Cuts remain available
+  for the era's lifetime; their absence must never authorize a partial minimum.
 - **Breaks:** a client silently resumes a replaced suffix or inaccessible prefix.
 - **Operational boundary:** rebuilding uses a fresh/wiped data directory.
   Checkpoint state, inclusion block, and next nonce are trusted operator inputs;

docs/plans/2026-07-coordination-tracks.md

@@ -76,7 +76,10 @@ Remaining checks need the actual consumer:
   for production readiness: the old native state is unavailable, the exported
   bundle restores correctly, and execution after rebuild matches the canonical
   machine. For the DEX, pin the designated state drive/memory region and derive
-  resume metadata from canonical execution. Add the integration check to the
+  resume metadata from canonical execution. The exporter must check
+  [pending-direct eligibility](../recovery/cockroach.md#checkpoint-eligibility)
+  and the drill must exercise refusal and earlier-checkpoint fallback, alongside
+  eligible pending-queue recovery. Add the integration check to the
   release validation once the actual artifacts are available; no generic trait
   or deployment gate currently enforces this requirement.
 - Repeat snapshot-to-live bootstrap and canonical comparison with the private

docs/protocol/application-contract.md

@@ -209,6 +209,10 @@ layout, and extraction procedure for each supported image. Other applications
 may require a different mapping. The recovery bundle also needs the exact L1
 boundary and next scheduler nonce, obtained from trusted canonical execution;
 these are separate from merely extracting application bytes.
+The exporter must also verify the canonical pending-direct queue satisfies
+[checkpoint eligibility](../recovery/cockroach.md#checkpoint-eligibility).
+An accurate application clock and `A < B` do not prove that condition after
+faulty sequencing.
 
 Each integration supplies a versioned export command and operator procedure,
 and demonstrates recovery from a non-genesis canonical checkpoint before

docs/protocol/projection-replay.md

@@ -66,8 +66,11 @@ share a count, and generations can reuse replaced offsets.
 
 Such a backup contains core state and projection at count `X`, inclusion block
 `B`, next scheduler nonce `N`, and the application's own clock `A`. The
-[manual recovery contract](../recovery/cockroach.md#replay-boundaries) requires
-`A < B`, except known empty genesis, and `B <= C` for the target rebuild:
+[manual recovery contract](../recovery/cockroach.md#checkpoint-eligibility)
+requires no pending canonical direct at or below `A`, as well as `A < B`
+(except known empty genesis) and `B <= C` for the target rebuild. Establish
+queue eligibility against the canonical checkpoint; application-state equality
+and the scalar bounds alone cannot prove it after faulty sequencing:
 
 1. Independently establish trust in the backup, projection implementation, and
    checkpoint boundary under the [incident playbook](../recovery/cockroach.md#application-specific-reader-state).