Skip to content

Improve recovery docs and doc policy #42

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
GCdePaula wants to merge 4 commits into from
Closed

Improve recovery docs and doc policy #42

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
254d236
docs: clarify recovery and improve agent reading paths
GCdePaula Sep 17, 2026
15f7998
docs: ground automatic recovery in implementation
GCdePaula Sep 17, 2026
3ef56ec
docs: align history snapshots and watchdog contracts
GCdePaula Sep 17, 2026
3569769
docs: retire stale reviews and define review lifecycle
GCdePaula Sep 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
350 changes: 171 additions & 179 deletions AGENTS.md
View file
Open in desktop

Large diffs are not rendered by default.

74 changes: 4 additions & 70 deletions CLAUDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,73 +1,7 @@
# CLAUDE.md

Quick reference for working in this repository. For the full guide — architecture, duality, recovery, invariants, threat model, and rules — read [`AGENTS.md`](AGENTS.md).
Read [`AGENTS.md`](AGENTS.md) before working in this repository. It contains the
shared mental model, safety constraints, and contribution rules for all agents.

## Shell Environment

This project uses Nix + direnv. Before running any command that needs project tools (Foundry, TLA+, etc.), activate the direnv environment:

```bash
eval "$(direnv export bash 2>/dev/null)"
```

This makes `anvil`, `forge`, `cast`, `tlc`, and other Nix-provided tools available. Cargo and rustc are available without direnv.

## Commands

```bash
cargo check # compile check
cargo test --workspace --exclude canonical-test # run tests (canonical-test needs libslirp)
cargo fmt --all # format
cargo clippy --all-targets --all-features -- -D warnings # lint
cargo test -p sequencer --lib # includes Anvil-backed tests (needs Foundry on PATH)
```

## What This Is

Off-chain sequencer for an app-specific DeFi rollup. Accepts signed user operations, issues low-latency soft confirmations, and posts batches to L1. Currently backed by a placeholder wallet app (transfer, withdrawal). **Security-critical infrastructure** — handle every change accordingly.

Rust edition 2024 / Axum API / SQLite (rusqlite, WAL) / EIP-712 signing / SSZ encoding.

## Workspace Layout

- `sequencer/` — sequencer library (no binary; app crates build the binary).
- `sequencer-core/` — shared domain types consumed by both sequencer and scheduler.
- `examples/app-core/` — placeholder wallet app implementing `Application`.
- `examples/wallet-sequencer/` — binary crate: wallet app + sequencer library.
- `bindings/c-app-engine/` — reusable native engine adapter implementing `Application` through a C ABI.
- `bindings/c-app-sequencer/` — optional C-engine CLI host and external-archive binary.
- `examples/c-wallet-engine/` — reference C ABI exports, genesis tool, and conformance tests.
- `examples/c-wallet-sequencer/` — binary composing the C-engine host with the reference wallet engine.
- `examples/canonical-app/` — on-chain scheduler reference implementation.
- `examples/canonical-test/` — e2e test harness for the canonical app.
- `sdk/rust-client/` — Rust client library for the sequencer API.
- `tests/{benchmarks,e2e,harness}/` — test infrastructure.

## Sequencer Module Layout

`sequencer/src/` is organized by writer role; `storage/<role>.rs` holds each role's storage half.

- `commands/` — the operator command brackets (`run/` plus its worker
supervisor, `setup/`, `flush`) and their command-scoped `config` and
`error` taxonomy (incl. exit-code projection).
- `runtime/` — the runtime authority capabilities, consumed crate-wide:
the exclusive process lock and the runtime scope/shutdown machinery.
- `ingress/` — public-facing: `api.rs` (`POST /tx`, `GET /fee`) + `inclusion_lane/` (hot path).
- `egress/` — internal read path: `api/` (WS subscribe + health) + `l2_tx_feed/`.
- `l1/` — reader, submitter, fee oracle, provider, partition helper.
- `recovery/` — startup preemptive-recovery procedure, runtime danger detector, mempool flusher.
- `storage/` — SQLite persistence, split per writer role.
- `http.rs` — shared HTTP error type + `axum::serve` orchestration; `clock.rs` — the crate-wide wall clock.

## Before You Start Real Work

- **[`AGENTS.md`](AGENTS.md)** — mission, requirements, invariants, duality, recovery, conventions, rules.
- **[`docs/protocol/`](docs/protocol/)** — the authoritative protocol contracts: [`scheduler-semantics.md`](docs/protocol/scheduler-semantics.md) (canonical acceptance algorithm), [`application-contract.md`](docs/protocol/application-contract.md) (the `Application` trait), and [`c-application-binding.md`](docs/protocol/c-application-binding.md) (the native C binding). Read before touching the scheduler, the gold frontier, the fold, or an `Application` impl.
- **[`docs/invariants.md`](docs/invariants.md)** — cross-module invariants register + the fail-loud check policy. Check it before changing anything it lists as load-bearing.
- **[`docs/review/register.md`](docs/review/register.md)** — the review register: open findings, settled decisions, refuted proposals (do-not-re-propose). Check it for open findings in code you're about to touch, and before proposing a mechanism or simplification.
- **[`docs/plans/`](docs/plans/)** — the [authority-boundary ADR](docs/plans/2026-08-authority-boundary-adr.md), active coordination tracks, and in-flight design handoffs. Check before starting work that might belong to a track.
- **[`docs/threat-model/README.md`](docs/threat-model/README.md)** — trust boundaries and in-scope threats.
- **[`docs/recovery/README.md`](docs/recovery/README.md)** — preemptive recovery design + TLA+ proofs.
- **[`docs/snapshots/lifecycle.md`](docs/snapshots/lifecycle.md)** — snapshot lifecycle design + invariants (take/promote/GC, crash-safety). Read before touching the inclusion lane's safe-frontier/snapshot path.
- **[`docs/watchdog/operator-deployment.md`](docs/watchdog/operator-deployment.md)** — watchdog on live L1 (Sepolia / mainnet, production-like).
- **[`docs/watchdog/getting-started.md`](docs/watchdog/getting-started.md)** — local dev: watchdog + `sequencer-devnet` on Anvil.
- [Shell and commands](AGENTS.md#shell-and-commands) — toolchain selection and validation commands.
- [Reading routes](AGENTS.md#reading-routes) — the contracts to read for the work at hand.
91 changes: 59 additions & 32 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Sequencer

A sequencer for Cartesi app-specific rollups. Provides low-latency soft confirmations for user operations, posts them to L1 in batches, and maintains a deterministic replay feed that matches the application's final execution order.
A sequencer for Cartesi app-specific rollups. Provides low-latency soft confirmations for user operations, posts them to L1 in batches, and exposes its current application execution order for replica replay.

**Security-critical infrastructure.** Handle every change with the care financial systems demand.

Expand All @@ -26,9 +26,9 @@ Sequencer (off-chain) Scheduler (on-chain)
```

When things go well, the sequencer's chain and the scheduler's view converge.
When batches are becoming stale on L1, the sequencer detects the doomed suffix
and runs standard recovery. Terminal canonical divergence is the distinct
content-identity case below.
When batches risk becoming stale on L1, the sequencer stops serving and startup
determines the required repair. A lost or untrustworthy local state instead
requires an operator rebuild. Both recovery modes are described below.

## Trust Model

Expand Down Expand Up @@ -60,18 +60,33 @@ backstop, not proof that arbitrary application or scheduler divergence cannot
exist, and it does not replace the watchdog. The mechanism and its bounds are
recorded in [`docs/invariants.md`](docs/invariants.md) (I9 and I15).

The third case is handled by the recovery subsystem. Batches that are too old when they reach L1 (`inclusion_block − safe_block ≥ MAX_WAIT_BLOCKS`) are skipped by the scheduler. This "staleness" poisons the nonce counter: all subsequent batches become unreachable regardless of their individual freshness. The sequencer detects this via a danger-zone threshold, preemptively goes offline, flushes the L1 mempool, and cascade-invalidates the doomed chain. See [`docs/recovery/`](docs/recovery/) for the full design, TLA+ formal verification, and design history.
## Recovery

The sequencer trusts its own code is bug-free. Recovery means recovery from liveness failures, which can legitimately happen even in the absence of bugs (infrastructure outages, network failures, gateway failure). Code-level bugs are a separate problem handled by tests and review. See [`docs/threat-model/README.md`](docs/threat-model/README.md) for the complete threat model applied across the codebase.
**[Standard recovery](docs/recovery/README.md)** runs automatically at startup
using the existing database. It handles liveness failures such as outages and
extended downtime: reconcile L1 outcomes, invalidate the affected optimistic
suffix, and resume from retained state. Stale batches do not consume the
scheduler's expected nonce, so their successors cannot be accepted until recovery
supplies a replacement at that nonce.

**[Cockroach recovery](docs/recovery/cockroach.md)** is an operator-triggered
rebuild when the local database is lost or cannot be trusted. This includes a
sequencer bug that corrupted state or emitted malformed batches: fix the bug,
choose a trusted canonical application checkpoint, then rebuild in a fresh data
directory. The command processes historical L1 inputs through the canonical
scheduler and prepares a baseline for resuming normal operation.

The [threat model](docs/threat-model/README.md#self-trust) explains the boundary
between normal operation's self-trust and manual repair after a bug.

## Failure Modes

The sequencer is designed to handle:

- **L1 provider outages** — workers retry with exponential backoff. The inclusion lane and API continue operating locally. A wall-clock fallback detects when an outage pushes batches into the danger zone.
- **Undiagnosed interruptions (OOM, SIGKILL, reboot)** — restart can recover automatically: every boot derives any required recovery from SQLite and L1 safe state through startup recovery, never assuming the previous exit was clean. Terminal errors returned through a command bracket best-effort record their cause in `terminal_faults`; terminal runtime aborts leave only process diagnostics.
- **Extended downtime** — startup syncs to the current L1 safe head, flushes if needed, and recovers before admission; restart policy is the exit-code contract (a terminal exit means: do not restart, page an operator — the one manual remedy is a fresh-directory `setup --recovery` after canonical divergence).
- **Adversarial L1 mempool** — block builders and private mempools are treated as adversarial. The recovery flusher consumes every pending nonce slot with a no-op so delayed "zombie" submissions cannot land later.
- **Extended downtime** — startup syncs to the current L1 safe head, flushes if needed, and recovers before admission. A terminal exit requires operator investigation; rebuilding untrustworthy state follows the cockroach recovery procedure above.
- **Adversarial L1 mempool** — block builders and private mempools are treated as adversarial. Recovery waits until every covered wallet-nonce slot is consumed at safe depth, whether the original transaction or a flush no-op wins, so delayed "zombie" submissions cannot land later.

## Interfaces

Expand All @@ -81,7 +96,13 @@ Users submit signed operations via `POST /tx` (JSON). Operations are signed with

### Sequenced Transaction Feed

Subscribers connect via `GET /ws/subscribe?era_id=<uuid>&recovery_generation=<u64>&next_input=<u64>` (WebSocket). The feed delivers all sequenced transactions (user ops + direct inputs) in deterministic order, matching the on-chain execution order. This is the primary interface for downstream consumers (frontends, indexers). The endpoint is designed for a small number of indexer subscribers, which serve users directly.
Subscribers restore an HTTP snapshot, then use one WebSocket stream to replay
application inputs and follow the optimistic tip. Recovery can replace that
history; the snapshot's era, generation, and input count bind a resume request
to the state the consumer actually holds. The endpoint serves a small number of
infrastructure subscribers, which serve users directly. See the
[bootstrap workflow](docs/protocol/application-history.md#replica-bootstrap-and-resume)
and [wire contract](#api).

### Batch Submission

Expand All @@ -91,11 +112,15 @@ The batch submitter posts closed batches to L1's InputBox contract. Each batch c

The sequencer runs in two phases. **`setup`** pins the
deployment identity (including the reviewed fee-oracle source), does the initial L1 sync, and registers the genesis
snapshot — run it once. It is L1-read-only: it takes the batch-submitter
snapshot — run it once. Plain `setup` is L1-read-only: it takes the batch-submitter
*address*, never the signing key. **`run`** boots the sequencer from the
set-up DB, reading identity from it (so chain id / app address are not `run`
arguments); it holds the signing key because it submits.

For rebuilding from a trusted checkpoint, follow the
[cockroach recovery procedure](docs/recovery/cockroach.md#run-a-rebuild).
`setup --recovery` also needs the submitter key because it flushes transactions.

```bash
# Phase A — set up the data dir (run once; idempotent).
CARTESI_SEQUENCER_BLOCKCHAIN_HTTP_ENDPOINT=http://127.0.0.1:8545 \
Expand Down Expand Up @@ -168,6 +193,16 @@ Notes:
- queue capacity is an internal runtime constant tuned alongside inclusion-lane chunking to absorb short bursts; if this starts triggering persistently, it is a signal to revisit runtime sizing or throughput rather than add another admission layer.
- Browser wallets can call `POST /tx` and `GET /fee` from any origin with any request headers; preflight permits GET and POST and is cached for one hour. CORS is applied only to ingress. Egress routes remain operator-only and require network access controls.

Success response after inclusion:

```json
{
"ok": true,
"sender": "0x...",
"nonce": 0
}
```

### `GET /fee`

Fee quote for setting signed user-op `max_fee` before `POST /tx`. All three fields are log-space exponents (base 129/128), the same encoding as `max_fee`. Inclusion rejects any op with `max_fee` below the open-frame `fee`.
Expand All @@ -186,7 +221,7 @@ Notes:

### `GET /ws/subscribe?era_id=<uuid>&recovery_generation=<u64>&next_input=<u64>`

WebSocket stream of canonical application inputs, replaying from the inclusive
WebSocket stream of the current application history, replaying from the inclusive
`next_input` offset and then following the optimistic tip. Fetch and restore
`/latest_snapshot` first; its headers supply the complete subscription claim.
After each successfully applied input at offset `X`, persist the claim with
Expand All @@ -205,6 +240,9 @@ After each successfully applied input at offset `X`, persist the claim with
including business failures and malformed-direct no-ops.
- Recovery stops the process and disconnects subscribers. A reconnect must
present its saved claim; offsets alone cannot distinguish a replaced suffix.
- Shutdown or a feed read/send failure may disconnect without a WebSocket
Close frame. Resume from the saved claim after an unexpected disconnect;
a clean close is not required for safe replay.

Message shapes:

Expand All @@ -216,16 +254,6 @@ Message shapes:
{ "kind": "direct_input", "offset": 11, "sender": "0x...", "block_number": 123, "block_timestamp": 1700000000, "transaction_hash": "0x...", "payload": "0x...", "input_index": 42, "batch_nonce": 4 }
```

Success response:

```json
{
"ok": true,
"sender": "0x...",
"nonce": 0
}
```

### Operator snapshot endpoints (internal only)

These serve application state to the operator's watchdog and indexers.
Expand All @@ -246,7 +274,7 @@ api split lands).
adding a coherent `checkpoint.toml` receipt with its L1 inclusion block and
next batch nonce for trusted recovery.

All state/archive responses include `X-History-Era`, `X-Recovery-Generation`,
Successful state/archive downloads include `X-History-Era`, `X-Recovery-Generation`,
and `X-Executed-Input-Count`, selected atomically with the artifact lease.
Streaming holds the lease until the response ends or the client disconnects.
The accepted endpoints return `404` until a comparable checkpoint exists:
Expand Down Expand Up @@ -310,21 +338,20 @@ docker pull ghcr.io/cartesi/sequencer-watchdog:vX

## Development

```bash
cargo check # compile
cargo test --workspace --exclude canonical-test # test (canonical-test needs libslirp)
cargo fmt --all # format
cargo clippy --all-targets --all-features -- -D warnings # lint
```

Some tests require [Foundry](https://getfoundry.sh) (`anvil` on PATH). They run by default and fail with a clear message if unavailable. This project uses Nix + direnv for tooling — `direnv allow` provides Foundry, TLA+, and other dependencies.
The shared [development commands](AGENTS.md#shell-and-commands) cover Rust
toolchain selection, Nix/direnv tooling, compilation, tests, formatting, and
linting. Read the [testing guidance](AGENTS.md#testing-guidance) before choosing
validation for a change; some tests require Anvil or libslirp.

## Further Reading

- [`AGENTS.md`](AGENTS.md) — developer guide: architecture, conventions, duality, recovery, invariants, rules.
- [`CLAUDE.md`](CLAUDE.md) — quick reference for shell setup and commands.
- [`CLAUDE.md`](CLAUDE.md) — Claude entrypoint to the shared agent guide.
- [`docs/threat-model/README.md`](docs/threat-model/README.md) — trust boundaries, in-scope and out-of-scope threats.
- [`docs/recovery/README.md`](docs/recovery/README.md) — recovery design, TLA+ formal verification, design history.
- [`docs/recovery/README.md`](docs/recovery/README.md) — automatic recovery, TLA+ formal verification, design history.
- [`docs/recovery/cockroach.md`](docs/recovery/cockroach.md) — manual rebuild after lost state or a sequencer bug.
- [Application history and replay](docs/protocol/application-history.md) — progress, history identity, and replica bootstrap/resume.
- [Snapshots](docs/snapshots/README.md) — engine checkpoints, lifecycle, accepted comparison, and wallet encoding.
- [`docs/watchdog/getting-started.md`](docs/watchdog/getting-started.md) — step-by-step: run the watchdog with a local sequencer.
- [`docs/watchdog/operator-deployment.md`](docs/watchdog/operator-deployment.md) — watchdog on live L1 (Sepolia staging, mainnet production).
- [`docs/watchdog/README.md`](docs/watchdog/README.md) — watchdog architecture, modules, and test commands.
Expand Down
Loading