From 254d2365330846e35fb8bd57879c42d6c25d7c73 Mon Sep 17 00:00:00 2001 From: gcdepaula Date: Thu, 17 Sep 2026 11:11:34 -0300 Subject: [PATCH 1/4] docs: clarify recovery and improve agent reading paths --- AGENTS.md | 347 ++++++++++++++++++------------------ CLAUDE.md | 74 +------- README.md | 50 ++++-- docs/recovery/README.md | 55 +++--- docs/recovery/cockroach.md | 245 +++++++++++++------------ docs/snapshots/README.md | 25 ++- docs/threat-model/README.md | 15 +- 7 files changed, 388 insertions(+), 423 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 22add5b..720c22d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,6 +1,9 @@ # AGENTS.md -This file tells AI coding agents and human contributors how to work effectively in this repository. Start here. +Start here for the repository's mental model and working rules. Read this baseline +once, then use [Reading Routes](#reading-routes) to follow the contracts relevant +to the work. Those documents own detailed behavior; this guide explains why it +matters and where to look. ## Mission @@ -72,7 +75,10 @@ Scheduler-acceptance semantics exist in exactly three implementations that must 2. the off-chain acceptance predicate — `ProtocolTiming::scheduler_accepts` ([`sequencer-core/src/protocol.rs`](sequencer-core/src/protocol.rs)), which feeds `safe_accepted_batches`; 3. the inclusion lane's live prediction (drain + execution order). -The expected-nonce fold is homed next to `scheduler_accepts` as `advance_expected_batch_nonce` (same file); the submitter's `decide_submit_start` consumes it, and `populate_safe_accepted_batches` keeps a deliberate inline copy (its advance is interleaved with storage-only side effects — the content-identity check and the divergence freeze — that can't move below the protocol layer). Touching any of these means re-checking the others — their agreement is the system's most load-bearing invariant (see [`docs/invariants.md`](docs/invariants.md)). +The submitter's expected-nonce scan also depends on this agreement. The +[scheduler contract](docs/protocol/scheduler-semantics.md#the-three-implementations-and-why-they-agree) +maps the implementations and the deliberate storage-local copy. Changing one +requires checking the others. Two mechanical facts the agreement rests on: @@ -88,44 +94,40 @@ A batch is **stale** when `inclusion_block - first_frame.safe_block >= MAX_WAIT_ 1. **Liveness failure** — the sequencer went offline and failed to submit batches in time. 2. **Censorship** — the sequencer kept submitting batches but froze `safe_block` to hold back direct inputs. -When the scheduler encounters a stale batch, it **skips it entirely** — no nonce consumed, no state change. This is the **censorship-resistance backstop**: the sequencer cannot hold write priority indefinitely without advancing the drain cursor. Direct inputs are force-drained at `MAX_WAIT_BLOCKS`, guaranteeing deposit availability within ~4h even under adversarial conditions. +When the scheduler encounters a stale batch, it skips its frames without +consuming the batch nonce. The overdue-direct backstop still runs. Together, +these rules prevent the sequencer from holding write priority indefinitely +without advancing the drain cursor; direct inputs are force-drained at +`MAX_WAIT_BLOCKS`, giving the ~4h censorship-resistance bound. ### Cascading invalidation If a batch is stale, all existing subsequent batches are also invalid. The scheduler's expected-nonce counter does not advance on a stale skip, so every subsequent batch arrives at an unexpected nonce and is rejected. Invalidation is a suffix operation: marking batch `N` invalid cascades to `N+1`, `N+2`, …, including the open batch. New batches created after recovery are unaffected. -### Preemptive recovery - -Rather than waiting for a batch to go stale on L1, the sequencer uses a **danger threshold** (`MAX_WAIT_BLOCKS − MARGIN`). The threshold is *only a trigger*: it tells the system "stop running, hand off to recovery." It does not encode "this batch is doomed" — that decision belongs to the post-flush cascade. - -The cycle crosses a process boundary by design: the in-process -[`DangerDetector`](sequencer/src/recovery/detector.rs) polls -`Storage::check_danger` on a cadence and returns a non-`Safe` worker exit; the -runtime closes intake and drains before the command returns non-zero (stopping -the process is how the sequencer goes offline). Diagnosed terminal runtime -faults abort immediately; expected-recovery and retryable exits are graceful; -the orchestrator respawns; on every boot startup recovery re-derives the -response from local facts. - -The authoritative dispatch table, phase ordering, boot-local witnesses, the -admission sequence, the "everything past gold is doomed" model, and the -per-path rationale live in -[`docs/recovery/README.md`](docs/recovery/README.md) — that document **owns** -the recovery design; this section is only the map. Do not restate dispatch -details here. - -### Detection: safe-only, with wall-clock fallback - -Staleness is only checked against L1 **safe** state, never latest. Stale batches in latest that haven't reached safe yet will eventually become safe, and the check will fire at that point. This avoids reacting to L1 reorgs. - -When the sequencer's view of L1 stops advancing — most often because the RPC gateway is stalled or returning stale reads, occasionally because L1 itself is unhealthy — the DB-based staleness check sees a frozen `current_safe_block` and may fail to trigger. The danger detector uses two wall-clock signals: the recorded L1 safe block timestamp must remain younger than `CARTESI_SEQUENCER_L1_READ_STALE_AFTER_BLOCKS`, and unresolved batches are also checked with `estimated_missed_blocks = (now − last_safe_progress_ms) / seconds_per_block` by adjusting the danger threshold downward. This prevents silently issuing doomed soft confirmations during stale-provider periods or L1 outages. - -### Formal verification - -The recovery design is verified by two bounded TLA+ models; -[`docs/recovery/README.md`](docs/recovery/README.md) "Formal Verification" -says what each proves. When touching recovery code, read both current models -first. +### Two recovery paths + +**Automatic recovery** repairs optimistic history after liveness failures. The +danger detector signals service to stop when a danger check fires; startup recovery +settles outstanding submissions and replaces the invalid suffix using local +SQLite facts and safe L1 history. The danger threshold is a trigger, not proof +that a batch is doomed. Detection uses safe state, with wall-clock checks when +the L1 view stops advancing. Expected recovery exits gracefully; diagnosed +terminal runtime faults abort immediately. The [automatic recovery design](docs/recovery/README.md) +owns detection, startup ordering, dispatch, and admission. + +**Manual cockroach recovery** (`setup --recovery`) rebuilds from a trusted +canonical application checkpoint and L1 history into a fresh data directory. +It also applies after sequencer bugs: fix the bug, choose a trusted canonical +checkpoint, and then run recovery. Historical batches, +including malformed ones, receive the canonical scheduler's treatment. The +result accounts for a fixed input prefix from which sequencing can resume; it +does not need to catch the moving L1 tip or preserve prior soft confirmations. +The [cockroach recovery guide](docs/recovery/cockroach.md) owns the checkpoint +requirements, stopping boundary, and rebuild procedure. + +Before changing recovery code, read its guide and both current bounded TLA+ +models. The automatic recovery guide's [Formal Verification](docs/recovery/README.md#formal-verification) +section explains their scopes; the models are not a proof of every recovery path. ## Threat Model (brief) @@ -134,13 +136,19 @@ See [`docs/threat-model/README.md`](docs/threat-model/README.md) for the full mo - **Trusted:** InputBox contract, our own Ethereum node (fail-stop, not byzantine), operator config, batch-submitter key. - **Adversarial:** `POST /tx` callers, direct-input senders, the L1 mempool and block builders (zombie transactions are a first-class threat). - **RPC endpoint:** single (`CARTESI_SEQUENCER_BLOCKCHAIN_HTTP_ENDPOINT`), trusted fail-stop, **must be one consistent node** — no fallback tier exists yet (see the threat model's actor table). -- **Self-trust:** the sequencer trusts its own code is correct. Bugs that emit malformed batches are fault states requiring manual intervention, not threats to defend against at runtime. +- **Self-trust:** normal operation assumes the sequencer's own code is correct. + Invariant violations fail loud. Bug-induced malformed batches require fixing + the bug and, when rebuilding is necessary, manual cockroach recovery; automatic + recovery does not repair software defects. - **In scope:** correctness bugs *and* exploitation. Under rollup semantics, a correctness bug that causes scheduler/sequencer state divergence is as severe as direct theft. ## Architecture Map Top-level layout follows the system's data flow. Each sequencer module corresponds to a writer role; the matching `storage/.rs` holds its storage half. +The implementation uses Rust edition 2024, Axum, SQLite (rusqlite/WAL), EIP-712 +signing, and SSZ batch encoding. + ### Workspace - `sequencer/` — sequencer **library** (no binary). App crates compose it into a binary. @@ -158,25 +166,17 @@ Top-level layout follows the system's data flow. Each sequencer module correspon ### Sequencer module layout -- `sequencer/src/lib.rs` — public sequencer API. The thin binary entrypoints live in `examples/wallet-sequencer/`. -- `sequencer/src/harness.rs` — CLI harness: the `setup`/`run`/`flush-mempool` subcommand parser, `dispatch`, and the exit-code projection. An app's `main` is ~5 lines (`run_main` + a genesis-app closure). -- `sequencer/src/http.rs` — shared HTTP error type, JSON `ErrorResponse`, `ApiConfig`, and `axum::serve` orchestration. -- `sequencer/src/commands/` — the operator command brackets: `setup` (phase A — pin identity, initial sync, genesis snapshot, atomic `setup_complete` fact), `run` (phase B — recover, prepare, admit, and boot workers; its `workers` supervisor lives beside it), and `flush` (`flush-mempool`). `sequencer/src/commands/` also owns the command-scoped `config` and `error` taxonomy (incl. the exit-code projection); `sequencer/src/runtime/` is exactly the runtime authority capabilities — the process lock and `shutdown` (runtime scope and graceful notification) — consumed crate-wide. `L1Config` lives in `sequencer/src/l1/`; the crate-wide wall clock is `sequencer/src/clock.rs`. -- `sequencer/src/ingress/` — public-facing HTTP + inclusion lane. - - `api.rs` — `POST /tx` and `GET /fee` handlers, JSON-rejection mapping. - - `inclusion_lane/` — single-lane hot-path loop (`mod.rs`), catch-up replay, config, error types. -- `sequencer/src/egress/` — internal read path. - - `api/` — `/ws/subscribe`, `/livez`, `/readyz`, `/healthz`. - - `l2_tx_feed/` — DB-backed ordered-tx feed. -- `sequencer/src/l1/` — L1 client surface. - - `reader.rs` — safe-input ingestion from InputBox into SQLite. - - `submitter/` — batch submitter (`worker.rs` + `poster.rs`); re-estimates fees every tick without carrying a fee floor from earlier attempts. The rationale, accepted liveness limits, and revisit criteria live in [`docs/l1-fee-policy.md`](docs/l1-fee-policy.md). - - `fee_oracle/` — setup-pinned L1 Uniswap V3 TWAP → `batch_policy.log_gas_price` (+ `log_gas_price_updated_at_ms`); fixed mode writes once at setup and has no worker. - - `eip1559.rs` — shared EIP-1559 fee estimation (poster, oracle, flusher). - - `provider.rs` — alloy provider construction. - - `partition.rs` — long-block-range retry helper. -- `sequencer/src/recovery/` — preemptive recovery startup procedure (`mod.rs`), runtime danger detector (`detector.rs`), and mempool flusher (`flusher.rs`). -- `sequencer/src/storage/` — SQLite persistence, split by writer role (`ingress`, `egress`, `l1_inputs`, `l1_submission`, `recovery`, `admin`, `safe_accepted_batches`, `snapshot_dumps`, plus shared `history`, `mod`, `open`, `convert`, `queries`, `mutations`, and `migrations/`). +Paths below are relative to `sequencer/src/`: + +- `lib.rs` and `harness.rs` — public API and shared CLI harness; app binaries supply their genesis-app constructor. +- `commands/` — `setup`, `run` (including worker supervision), and `flush`, with command configuration and exit-code classification. +- `runtime/` — exclusive process ownership and runtime scope/shutdown. +- `ingress/` — public HTTP handlers and the single inclusion lane. +- `egress/` — internal HTTP/WS API and DB-backed application-input feed. +- `l1/` — safe-input reader, batch submitter, fee oracle, shared EIP-1559 estimation, and provider access. +- `recovery/` — automatic startup repair, danger detector, and mempool flusher; the manual rebuild lives in `commands/setup/`. +- `storage/` — each writer role's persistence operations, shared queries, and schema. +- `http.rs` and `clock.rs` — shared HTTP errors/server setup and the crate-wide wall clock. `L1Config` lives in `l1/`. ## Key Concepts @@ -185,8 +185,8 @@ Top-level layout follows the system's data flow. Each sequencer module correspon - **Batch** — list of frames posted on-chain as one L1 transaction (SSZ-encoded). - **Inclusion lane** — the single ordering lane, with a latency-critical user-op regime and a slower L1-reconciliation regime ([ADR mechanism 4](docs/plans/2026-08-authority-boundary-adr.md)); the only writer of open batch/frame state ([I17](docs/invariants.md)) and the system's execution bottleneck. - **Batch submitter** — stateless worker that bulk-submits all pending batches each tick. Nonces are assigned by storage (structural `parent.nonce + 1`) when batches are closed; the submitter just reads them. -- **Danger detector** — background worker that polls `Storage::check_danger` on a fixed cadence and exits with `RecoveryRequired` when any non-`Safe` danger status fires. Never writes to the DB; never talks to L1. Crashes the process so startup recovery or refusal can run. -- **Fee oracle** — setup pins either a fixed exponent or a reviewed Uniswap V3 WETH/X TWAP tuple into deployment identity, and writes the first `log_gas_price` (+ observation stamp) in both modes. Setup requires a successful live quote; `run` performs no fee-source read before recovery/admission. Fixed mode has no worker; Uniswap launches a lazy refresher that immediately attempts a quote, persists successes, and retains the last price while logging and retrying transient source failures. The stamp is telemetry, not a runtime-admission or expiry gate. A shared-endpoint outage/stale view is already detected from L1 safe-head progress; a fee-source-only outage is an accepted economic residual (stale-low may subsidize DA, stale-high may reject users), not a canonical-correctness fault. Deterministic source misconfiguration, fatal arithmetic, and persistent storage faults remain terminal. The 10× margin lives in `batch_policy.log_slack`; it is a buffer rather than a bound on market movement, and frame fees stay immutable until the next frame opens. +- **Danger detector** — polls `Storage::check_danger` and signals the process to stop so startup can recover or refuse. It reads local facts; it never writes the DB or talks to L1. +- **Fee oracle** — setup pins and bootstraps a fixed price or Uniswap V3 TWAP source. The price informs future frame fees; an oracle-only outage is an accepted economic risk. The [threat model's actor table](docs/threat-model/README.md#actors-and-trust) owns the source assumptions and failure policy. - **Input reader** — ingests safe inputs from L1 InputBox and maintains the durable safe head, accepted-batch projection, and divergence marker in one atomic transaction (`sequencer/src/storage/l1_inputs.rs`); it hands the lane no in-memory cursor. - **L2 tx feed** — DB-backed application-input stream. HTTP snapshot headers provide `(EraId, RecoveryGeneration, ExecutedInputCount)`; WS validates that @@ -211,8 +211,13 @@ Top-level layout follows the system's data flow. Each sequencer module correspon - Rejections (`InvalidNonce`, `InvalidMaxFee`, `InsufficientFeeBalance`) produce no state mutation and are not persisted. These are protocol-level rejection semantics every app must implement: nonces prevent user-op replay, fees prevent spam against the sequencer's DA budget. ("Fee", not "gas" — the fee tracks DA; compute metering, if it ever exists, is a separate future concept.) - Included txs are persisted as frame/batch data in `batches`, `frames`, `user_ops`, `safe_inputs`, and `application_inputs`. Recovery metadata lives in `safe_accepted_batches`; batch lifecycle state (sealed/invalidated) lives on the `batches` row itself as write-once timestamps. - Frame fee is persisted in `frames.fee` and is fixed for the lifetime of that frame. The next frame's fee is currently sampled from `batch_policy_derived.recommended_fee` at rotation; oracle bootstrap writes the price before any Tip can sample it, and `log_slack` applies the 10× margin in log space. This is present behavior, not a reason for the five-block clock policy; hoisting fee to the batch is a later design with its own trade-offs. -- Wallet state (balances, nonces) is in-memory today — not persisted. -- **EIP-712 domain fields:** `name`, `version`, `chainId`, `verifyingContract`. `chainId` and `verifyingContract` come from `CARTESI_SEQUENCER_BLOCKCHAIN_ID` and `CARTESI_SEQUENCER_APP_ADDRESS` (validated against the RPC chain id at startup). All four fields must be present on both sides — both the sequencer and the on-chain scheduler construct the domain via `sequencer_core::build_input_domain`, the canonical shared constructor. +- Wallet balances and nonces live in memory between checkpoints; restart restores + a dump and replays persisted application inputs. +- **EIP-712 domain fields:** `name`, `version`, `chainId`, `verifyingContract`. + Setup pins the chain id and app address from `CARTESI_SEQUENCER_BLOCKCHAIN_ID` + and `CARTESI_SEQUENCER_APP_ADDRESS`, validating the chain id against RPC. All + four fields must be present on both sides; the sequencer and canonical + scheduler share `sequencer_core::build_input_domain`. ### InputBox payload classification @@ -244,41 +249,23 @@ User ops are executed only through `sequencer_core::application::validate_and_ex `Application` requires `Send`, with neither `Clone` nor `Sync`. Dumps must be durable and immutable, and restored engines must remain independent after source deletion. The opaque app prefix may be a file or directory; checkpoint disposal uses ordinary recursive filesystem deletion. Canonical inspection belongs to the separate `CanonicalState` trait; the native sequencer serves the comparison file in the checkpoint. The [C binding guide](docs/protocol/c-application-binding.md) maps the contract to native engines. -## Hot-Path Invariants - -The hot-path rules are owned elsewhere; this section is only the map. - -- Drain attribution, frame-clock monotonicity, the - content-identity check and the divergence freeze, history metadata, the - `WriteHead` cache, and the application-input sequence are registered in - [`docs/invariants.md`](docs/invariants.md) (the fail-loud check policy plus - I2, I3, I9, I10, I12–I18, I20) — that register owns them; do not restate - them here. -- Command admission, terminal stop, the two-regime lane and its - acknowledgement rule, and role-local authority are owned by the - [authority-boundary ADR](docs/plans/2026-08-authority-boundary-adr.md) - (mechanisms 1, 2, and 4); startup recovery by - [`docs/recovery/README.md`](docs/recovery/README.md). -- The frame-clock policy (five newly-safe blocks, one frame at the observed - tip, never interpolated, and its revisit trigger) is owned by - [`docs/protocol/scheduler-semantics.md`](docs/protocol/scheduler-semantics.md); - the no-preemption digestibility assumption by - [`docs/protocol/application-contract.md` §5](docs/protocol/application-contract.md#5-operational-capacity-for-l1-reconciliation). -- Queue admission (`429 OVERLOADED`), batch closure, and every other API or - storage-model shape are owned by [`README.md`](README.md). - -One rule lives here because nothing else owns it: preserve single-lane -deterministic ordering. Do not introduce extra concurrency in hot-path -ordering logic without explicit approval. - -## Storage Invariants - -Owned by [`docs/invariants.md`](docs/invariants.md): the writer-role table -(one writer role per fact), the `valid_*` view rule, `WriteHead` coherence -(I17), history metadata (I18), the application-input sequence and its canonical offsets (I10, I20). The schema -(`sequencer/src/storage/migrations/0001_schema.sql`) owns the write-once -batch lifecycle, the Tip's uniqueness, and the user-op identity rule. Do not -restate them here. +## Ordering and Storage + +Preserve single-lane deterministic ordering. Do not introduce extra concurrency +in hot-path ordering logic without explicit approval. + +The inclusion lane combines a latency-critical user-op regime with complete L1 +reconciliation. The [authority-boundary ADR](docs/plans/2026-08-authority-boundary-adr.md) +owns that split, acknowledgement rules, runtime ownership, and command admission. +The [scheduler contract](docs/protocol/scheduler-semantics.md#sequencer-frame-clock-policy) +owns the five-safe-block frame clock; the [Application contract](docs/protocol/application-contract.md#5-operational-capacity-for-l1-reconciliation) +owns the assumption that accumulated directs are digestible without preemption. + +Storage changes cross writer boundaries even when the SQL looks local. Read the +[invariant register](docs/invariants.md) for writer ownership, `valid_*` reads, +drain attribution, the content-identity/divergence freeze, `WriteHead` coherence, +and application-history offsets. The [schema](sequencer/src/storage/migrations/0001_schema.sql) +enforces write-once batch lifecycle, Tip uniqueness, and user-op identity. ## Type Boundaries @@ -294,41 +281,35 @@ restate them here. ## HTTP Endpoints - **Ingress** (public-facing): `POST /tx`, `GET /fee`. -- **Egress** (internal indexers/watchdog): `GET /ws/subscribe`, `GET /finalized_state`, `GET /finalized_state/inclusion_block`, `GET /latest_snapshot`, `GET /finalized_snapshot`, `GET /livez`, `GET /readyz`, `GET /healthz`. The snapshot/state endpoints are **operator-only** (no auth) and must not be exposed publicly; the streaming routes hold a GC lease for the response lifetime ([`docs/snapshots/lifecycle.md`](docs/snapshots/lifecycle.md)). +- **Egress** (internal indexers/watchdog): application-input subscriptions, + snapshot/state downloads, and health probes. Snapshot/state endpoints have no + authentication and **must not be exposed publicly**. Downloads hold a GC lease + for their response lifetime ([snapshot lifecycle](docs/snapshots/lifecycle.md)). Today both sides serve from one listener; the planned API split puts each side on its own port (same binary) so internal probes and subscribers can be firewalled from public submit traffic. -Message shapes, caps, close codes, and health semantics are **owned by [`README.md`](README.md)** (the API contract) — do not restate them here. +The [README API contract](README.md#api) owns routes, message shapes, caps, +close codes, and health semantics. -## Environment Variables +## Command Configuration -Split by subcommand (the phase split). **`setup`** (required): +Configuration follows the command phases: -- `CARTESI_SEQUENCER_BLOCKCHAIN_HTTP_ENDPOINT` -- `CARTESI_SEQUENCER_BLOCKCHAIN_ID` -- `CARTESI_SEQUENCER_APP_ADDRESS` -- `CARTESI_SEQUENCER_BATCH_SUBMITTER_ADDRESS` (the submitter address — `setup` is L1-read-only and never signs). **Must be a dedicated address**: `setup`'s detection gate refuses if the submitter's wallet nonce is unsettled, so reusing a busy address (e.g. the contract deployer, whose deploy-tx tail isn't safe at setup time) false-positives. The devnet uses anvil account 9 (`DEVNET_SEQUENCER_ADDRESS`), distinct from the account-0 deployer. -- `CARTESI_SEQUENCER_CHECKPOINT_BLOCK` (optional, default `0` = genesis) — the trusted checkpoint machine's L1 inclusion block. `setup` refuses (typed `SetupRefuse`, exit 40 = run `setup --recovery`) if a previous instance left work past it; plain `setup` detects only, and loading a non-genesis checkpoint machine is `setup --recovery`. +- Plain **`setup`** is L1-read-only and never signs. It pins chain/app/submitter + identity and the fee source. **`setup --recovery`** also needs the submitter + key to flush its outstanding transactions; see the [rebuild guide](docs/recovery/cockroach.md). +- **`run`** takes the RPC endpoint and signing key (or key file). It reads the + pinned chain id, app address, and submitter address from the database. -**`run`** (required) — chain id / app address / submitter address are read from the DB `setup` pinned, not from args: +**Use a dedicated submitter address.** Plain setup refuses an unsettled wallet +nonce, so sharing a busy contract-deployer address can trip the detection gate +while its deployment transactions are not yet safe. The devnet uses Anvil +account 9 (`DEVNET_SEQUENCER_ADDRESS`), separate from the account-0 deployer. -- `CARTESI_SEQUENCER_BLOCKCHAIN_HTTP_ENDPOINT` -- `CARTESI_SEQUENCER_AUTH_PRIVATE_KEY` or `CARTESI_SEQUENCER_AUTH_PRIVATE_KEY_FILE` - -**Optional** (names only — defaults and semantics are **owned by -[`sequencer/src/commands/config.rs`](sequencer/src/commands/config.rs)**; a -defaults list here drifted once already): `CARTESI_SEQUENCER_HTTP_ADDR`, `CARTESI_SEQUENCER_DATA_DIR`, -`CARTESI_SEQUENCER_LONG_BLOCK_RANGE_ERROR_CODES`, `CARTESI_SEQUENCER_BATCH_SUBMITTER_IDLE_POLL_INTERVAL_MS`, -`CARTESI_SEQUENCER_BATCH_SUBMITTER_CONFIRMATION_DEPTH`, `CARTESI_SEQUENCER_PREEMPTIVE_MARGIN_BLOCKS`, -`CARTESI_SEQUENCER_L1_READ_STALE_AFTER_BLOCKS` (fixed default, independent of the margin; -must be strictly below the danger threshold or startup refuses), -`CARTESI_SEQUENCER_SECONDS_PER_BLOCK`, and the runtime-only -`CARTESI_SEQUENCER_FEE_ORACLE_POLL_INTERVAL_MS`. Setup-only fee-oracle source knobs are -(`CARTESI_SEQUENCER_FEE_ORACLE_FIXED_LOG_GAS_PRICE`, -`CARTESI_SEQUENCER_FEE_TOKEN_ADDRESS`, -`CARTESI_SEQUENCER_WETH_ADDRESS`, `CARTESI_SEQUENCER_UNISWAP_V3_POOL`, -`CARTESI_SEQUENCER_FEE_ORACLE_TWAP_WINDOW_SECS` — mainnet/Sepolia default to - pinned USDC pool presets; other chains require an explicit source). +The [Running guide](README.md#running) gives invocation examples; +[`commands/config.rs`](sequencer/src/commands/config.rs) owns environment-variable +names, defaults, and validation. Check configuration there before documenting or +changing it, including checkpoint and fee-source selection. ## Coding Conventions @@ -343,23 +324,33 @@ must be strictly below the danger threshold or startup refuses), ## Documentation Practice -The corpus has two tenses, kept strictly apart: - -- **Living docs are timeless.** This file, `README.md`, `docs/protocol/`, - `docs/invariants.md`, `docs/recovery/`, `docs/snapshots/`, - `docs/threat-model/`, `docs/watchdog/`, and `docs/plans/` describe what is - true now and why — present tense, reasoning inline, no dates, no amendment - banners, no review codenames, no "previously/no longer". Each doc owns its - topic; others point at it rather than restating it. -- **History lives only in `docs/review/` and commit messages.** A review - ledger is append-only while its review is open. When it closes, distill it: - promote conclusions into the living docs, record settled decisions and - refuted proposals in [`docs/review/register.md`](docs/review/register.md), - and delete the process narration. Conclusions with reasoning outlive the - path taken to them. -- **Record deliberate absence once**, at the seam where someone would re-add - the mechanism, phrased as a positive design statement with its reason — - never as removal notices scattered across documents. +Write for a reader building a mental model. Start with the purpose, input, +result, and governing constraint; introduce implementation detail when it +explains a necessary behavior. Keep an intentional baseline here so important +concepts are discoverable before anyone knows to ask about them. Summaries +should point to the owner of a contract rather than becoming a second copy. + +Keep three kinds of material distinct: + +- **Current contracts and designs** describe what holds now and why. Use present + tense and reasoning inline, without amendment banners, review codenames, or + "previously/no longer" narration. Each topic has one owner. Some current + architecture documents live in `docs/plans/`; the directory name does not + make their established contracts optional. +- **Active plans** name open decisions, dependencies, and remaining work. They + must distinguish proposed behavior from implemented contracts. On completion, + put the durable design in its owner and reduce the plan to its remaining work + and links. +- **Historical evidence** lives in review ledgers, explicitly marked historical + documents, and commit history. A review ledger is append-only while open. + When it closes, promote conclusions into current docs, record settled and + refuted proposals in the [review register](docs/review/register.md), and remove + process narration. Retained superseded proposals must say they are historical + and link to the current contract; they are evidence, not instructions. + +**Record deliberate absence once**, at the seam where someone would re-add the +mechanism, phrased as a positive design statement with its reason. Avoid removal +notices scattered across documents. ## Testing Guidance @@ -374,34 +365,30 @@ Focus tests on: Prefer black-box tests around `POST /tx` and commit outcomes for integration. -Some `sequencer` tests use Anvil (Foundry). They run by default and fail with a clear message if `anvil` is not on PATH. Install Foundry or use `nix develop`. +Some `sequencer` tests use Anvil (Foundry). They run by default and fail with a +clear message if `anvil` is not on PATH. Use the configured Nix/direnv environment +or install Foundry. `canonical-test` additionally needs libslirp. -## Fast Start Commands +## Shell and Commands -See [`CLAUDE.md`](CLAUDE.md) for shell setup and the full command list. In short: +Use the configured Nix/direnv environment for Foundry, TLA+, and other project +tools. For noninteractive commands, prefer `direnv exec . `. Rust is +pinned to **1.95.0** in [`rust-toolchain.toml`](rust-toolchain.toml); verify both +`cargo --version` and `rustc --version` in the environment you use. A Nix-provided +Cargo or rustc may bypass rustup, so direnv alone does not guarantee the pinned +compiler is selected. Correct the toolchain selection before interpreting build +or dependency errors. ```bash -cargo check -cargo test --workspace --exclude canonical-test -cargo fmt --all -cargo clippy --all-targets --all-features -- -D warnings +direnv exec . cargo check +direnv exec . cargo test --workspace --exclude canonical-test +direnv exec . cargo test -p sequencer --lib # includes Anvil-backed tests +direnv exec . cargo fmt --all +direnv exec . cargo clippy --all-targets --all-features -- -D warnings ``` -Run server (two phases — `setup` once, then `run`; see `README.md` "Running"): - -```bash -# setup (L1-read-only; takes the submitter ADDRESS, not the key) -CARTESI_SEQUENCER_BLOCKCHAIN_HTTP_ENDPOINT=http://127.0.0.1:8545 \ -CARTESI_SEQUENCER_BLOCKCHAIN_ID=31337 \ -CARTESI_SEQUENCER_APP_ADDRESS=0x1111111111111111111111111111111111111111 \ -CARTESI_SEQUENCER_BATCH_SUBMITTER_ADDRESS=0xf39Fd6e51aad88F6F4ce6aB8827279cffFb92266 \ -cargo run -p wallet-sequencer -- setup - -# run (keyed; reads identity from the set-up DB) -CARTESI_SEQUENCER_BLOCKCHAIN_HTTP_ENDPOINT=http://127.0.0.1:8545 \ -CARTESI_SEQUENCER_AUTH_PRIVATE_KEY=0xac0974bec39a17e36ba4a6b4d238ff944bacb478cbed5efcae784d7bf4f2ff80 \ -cargo run -p wallet-sequencer -- run -``` +The shared command harness is in [`sequencer/src/harness.rs`](sequencer/src/harness.rs). +See [Running](README.md#running) for the two-phase `setup` / `run` workflow. ## Always / Ask First / Never @@ -411,14 +398,15 @@ cargo run -p wallet-sequencer -- run - Preserve API error shape and status code mapping unless intentionally changing the API contract. - Add or update tests when logic changes. - Run at least `cargo check` before finishing. -- Read `docs/recovery/` before touching recovery code, and `docs/threat-model/` before touching trust-boundary code. +- Read the relevant recovery guide and both current TLA+ models before touching + recovery code, and the threat model before touching trust-boundary code. - Check [`docs/invariants.md`](docs/invariants.md) before changing anything it lists as load-bearing, and [`docs/review/register.md`](docs/review/register.md) for open findings in the code you're about to touch and for decisions already settled or refuted. ### Ask First - Changing tx wire format (`UserOp`, SSZ payload layout, EIP-712 domain fields). - Changing DB schema or migration strategy. -- Altering rejection semantics (what consumes nonce/gas vs what is rejected). +- Altering rejection semantics (what consumes nonce/fee vs what is rejected). - Introducing concurrency changes to commit ordering. - Changing chunk/frame/batch closure or ack semantics. @@ -444,18 +432,23 @@ Before finishing a change, ensure: 3. Formatting and lints are clean, or list any unresolved warnings explicitly. 4. PR summary includes **what changed**, **why it changed**, and **risk / compatibility notes**. -## Related Documents - -- [`README.md`](README.md) — product framing, user-facing trust model, **API contract** (endpoint shapes, caps, close codes, health semantics). -- [`CLAUDE.md`](CLAUDE.md) — shell setup, quick reference, pointer back here. -- [`docs/protocol/`](docs/protocol/) — the authoritative protocol contracts: [`scheduler-semantics.md`](docs/protocol/scheduler-semantics.md) (the canonical acceptance algorithm, I1), [`application-contract.md`](docs/protocol/application-contract.md) (the `Application` trait contract), and [`c-application-binding.md`](docs/protocol/c-application-binding.md) (the native C binding). -- [`docs/invariants.md`](docs/invariants.md) — register of cross-module invariants (what's load-bearing across files) + the fail-loud check policy. -- [`docs/review/register.md`](docs/review/register.md) — the review register: open findings, settled decisions, refuted proposals (do-not-re-propose), and the review history table; the dated ledgers beside it carry the evidence the table points at. -- [`docs/plans/`](docs/plans/) — the architecture decision record ([`2026-08-authority-boundary-adr.md`](docs/plans/2026-08-authority-boundary-adr.md)), active coordination tracks, and in-flight design handoffs. -- [`docs/threat-model/README.md`](docs/threat-model/README.md) — trust boundaries, in-scope and out-of-scope threats. -- [`docs/recovery/README.md`](docs/recovery/README.md) — recovery design, TLA+ formal verification, design history. -- [`docs/snapshots/`](docs/snapshots/) — app snapshots: [`format.md`](docs/snapshots/format.md) (dump trait + wire format) and [`lifecycle.md`](docs/snapshots/lifecycle.md) (creation/acceptance/GC/lease design + crash-safety). -- [`docs/watchdog/operator-deployment.md`](docs/watchdog/operator-deployment.md) — production-like watchdog (Sepolia / mainnet; internal snapshot API). -- [`docs/watchdog/getting-started.md`](docs/watchdog/getting-started.md) — local dev: watchdog + `sequencer-devnet` on Anvil. -- [`docs/watchdog/README.md`](docs/watchdog/README.md) — watchdog architecture, compare vs advance modes, test commands. -- [`sequencer-core/`](sequencer-core/) — shared domain types and protocol contracts. +## Reading Routes + +Follow the rows that intersect the change. Each destination explains the +cross-module consequences to check before editing; follow its links when the +work reaches another boundary. + +| Work | Read first and why | +|---|---| +| Scheduler acceptance, batch nonces, direct-input ordering, or frame clock | [Scheduler semantics](docs/protocol/scheduler-semantics.md) — canonical algorithm and the implementations that must agree. | +| Inclusion lane, storage writes, or runtime concurrency | [Invariant register](docs/invariants.md) — enforcement and consumers; [authority-boundary ADR](docs/plans/2026-08-authority-boundary-adr.md) — ownership, acknowledgement, admission, and terminal stop. | +| Application implementation, execution, or native integration | [Application contract](docs/protocol/application-contract.md) — determinism, progress, failure, capacity, and checkpoints; [C binding](docs/protocol/c-application-binding.md) for native engines. | +| Automatic recovery or danger detection | [Automatic recovery](docs/recovery/README.md), then [preemptive.tla](docs/recovery/preemptive.tla) and [admission.tla](docs/recovery/admission.tla) — repair ordering and the models' bounded guarantees. | +| Manual rebuild after lost state or a sequencer bug | [Cockroach recovery](docs/recovery/cockroach.md) — trusted checkpoint, fixed input boundary, and fresh baseline. | +| API, subscriber replay, or application-history coordinates | [README API contract](README.md#api) — wire behavior; [application history](docs/plans/application-history.md) — era, generation, offsets, and recovery boundaries. | +| Snapshots, restart, export, retention, or watchdog checkpoints | [Snapshot lifecycle](docs/snapshots/lifecycle.md) — durable publication, accepted comparison points, leases, and GC; [wallet format](docs/snapshots/format.md) when changing wallet bytes. | +| Trust boundaries, provider behavior, or hostile L1 input | [Threat model](docs/threat-model/README.md) — actor assumptions, supported failures, and residual risks. | +| Submission fees or oracle pricing | [L1 fee policy](docs/l1-fee-policy.md) — estimation and replacement limits; [threat-model actor table](docs/threat-model/README.md#actors-and-trust) — oracle source and outage assumptions. | +| Command setup or deployment configuration | [Running](README.md#running) and [config.rs](sequencer/src/commands/config.rs) — invocation, identity pinning, defaults, and validation. | +| Watchdog development or operation | [Architecture](docs/watchdog/README.md); [local dev](docs/watchdog/getting-started.md) for Anvil; [operator deployment](docs/watchdog/operator-deployment.md) for Sepolia/mainnet. | +| A new mechanism, simplification, or work spanning an active track | [Review register](docs/review/register.md) — relevant open findings and settled/refuted reasoning; [coordination tracks](docs/plans/2026-07-coordination-tracks.md) — remaining work and dependencies. Consult dated evidence when its reasoning is needed. | diff --git a/CLAUDE.md b/CLAUDE.md index 037795a..a10e1d3 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,73 +1,7 @@ # CLAUDE.md -Quick reference for working in this repository. For the full guide — architecture, duality, recovery, invariants, threat model, and rules — read [`AGENTS.md`](AGENTS.md). +Read [`AGENTS.md`](AGENTS.md) before working in this repository. It contains the +shared mental model, safety constraints, and contribution rules for all agents. -## Shell Environment - -This project uses Nix + direnv. Before running any command that needs project tools (Foundry, TLA+, etc.), activate the direnv environment: - -```bash -eval "$(direnv export bash 2>/dev/null)" -``` - -This makes `anvil`, `forge`, `cast`, `tlc`, and other Nix-provided tools available. Cargo and rustc are available without direnv. - -## Commands - -```bash -cargo check # compile check -cargo test --workspace --exclude canonical-test # run tests (canonical-test needs libslirp) -cargo fmt --all # format -cargo clippy --all-targets --all-features -- -D warnings # lint -cargo test -p sequencer --lib # includes Anvil-backed tests (needs Foundry on PATH) -``` - -## What This Is - -Off-chain sequencer for an app-specific DeFi rollup. Accepts signed user operations, issues low-latency soft confirmations, and posts batches to L1. Currently backed by a placeholder wallet app (transfer, withdrawal). **Security-critical infrastructure** — handle every change accordingly. - -Rust edition 2024 / Axum API / SQLite (rusqlite, WAL) / EIP-712 signing / SSZ encoding. - -## Workspace Layout - -- `sequencer/` — sequencer library (no binary; app crates build the binary). -- `sequencer-core/` — shared domain types consumed by both sequencer and scheduler. -- `examples/app-core/` — placeholder wallet app implementing `Application`. -- `examples/wallet-sequencer/` — binary crate: wallet app + sequencer library. -- `bindings/c-app-engine/` — reusable native engine adapter implementing `Application` through a C ABI. -- `bindings/c-app-sequencer/` — optional C-engine CLI host and external-archive binary. -- `examples/c-wallet-engine/` — reference C ABI exports, genesis tool, and conformance tests. -- `examples/c-wallet-sequencer/` — binary composing the C-engine host with the reference wallet engine. -- `examples/canonical-app/` — on-chain scheduler reference implementation. -- `examples/canonical-test/` — e2e test harness for the canonical app. -- `sdk/rust-client/` — Rust client library for the sequencer API. -- `tests/{benchmarks,e2e,harness}/` — test infrastructure. - -## Sequencer Module Layout - -`sequencer/src/` is organized by writer role; `storage/.rs` holds each role's storage half. - -- `commands/` — the operator command brackets (`run/` plus its worker - supervisor, `setup/`, `flush`) and their command-scoped `config` and - `error` taxonomy (incl. exit-code projection). -- `runtime/` — the runtime authority capabilities, consumed crate-wide: - the exclusive process lock and the runtime scope/shutdown machinery. -- `ingress/` — public-facing: `api.rs` (`POST /tx`, `GET /fee`) + `inclusion_lane/` (hot path). -- `egress/` — internal read path: `api/` (WS subscribe + health) + `l2_tx_feed/`. -- `l1/` — reader, submitter, fee oracle, provider, partition helper. -- `recovery/` — startup preemptive-recovery procedure, runtime danger detector, mempool flusher. -- `storage/` — SQLite persistence, split per writer role. -- `http.rs` — shared HTTP error type + `axum::serve` orchestration; `clock.rs` — the crate-wide wall clock. - -## Before You Start Real Work - -- **[`AGENTS.md`](AGENTS.md)** — mission, requirements, invariants, duality, recovery, conventions, rules. -- **[`docs/protocol/`](docs/protocol/)** — the authoritative protocol contracts: [`scheduler-semantics.md`](docs/protocol/scheduler-semantics.md) (canonical acceptance algorithm), [`application-contract.md`](docs/protocol/application-contract.md) (the `Application` trait), and [`c-application-binding.md`](docs/protocol/c-application-binding.md) (the native C binding). Read before touching the scheduler, the gold frontier, the fold, or an `Application` impl. -- **[`docs/invariants.md`](docs/invariants.md)** — cross-module invariants register + the fail-loud check policy. Check it before changing anything it lists as load-bearing. -- **[`docs/review/register.md`](docs/review/register.md)** — the review register: open findings, settled decisions, refuted proposals (do-not-re-propose). Check it for open findings in code you're about to touch, and before proposing a mechanism or simplification. -- **[`docs/plans/`](docs/plans/)** — the [authority-boundary ADR](docs/plans/2026-08-authority-boundary-adr.md), active coordination tracks, and in-flight design handoffs. Check before starting work that might belong to a track. -- **[`docs/threat-model/README.md`](docs/threat-model/README.md)** — trust boundaries and in-scope threats. -- **[`docs/recovery/README.md`](docs/recovery/README.md)** — preemptive recovery design + TLA+ proofs. -- **[`docs/snapshots/lifecycle.md`](docs/snapshots/lifecycle.md)** — snapshot lifecycle design + invariants (take/promote/GC, crash-safety). Read before touching the inclusion lane's safe-frontier/snapshot path. -- **[`docs/watchdog/operator-deployment.md`](docs/watchdog/operator-deployment.md)** — watchdog on live L1 (Sepolia / mainnet, production-like). -- **[`docs/watchdog/getting-started.md`](docs/watchdog/getting-started.md)** — local dev: watchdog + `sequencer-devnet` on Anvil. +- [Shell and commands](AGENTS.md#shell-and-commands) — toolchain selection and validation commands. +- [Reading routes](AGENTS.md#reading-routes) — the contracts to read for the work at hand. diff --git a/README.md b/README.md index 3df30f2..68c1a0a 100644 --- a/README.md +++ b/README.md @@ -26,9 +26,9 @@ Sequencer (off-chain) Scheduler (on-chain) ``` When things go well, the sequencer's chain and the scheduler's view converge. -When batches are becoming stale on L1, the sequencer detects the doomed suffix -and runs standard recovery. Terminal canonical divergence is the distinct -content-identity case below. +When batches risk becoming stale on L1, the sequencer stops serving and startup +determines the required repair. A lost or untrustworthy local state instead +requires an operator rebuild. Both recovery modes are described below. ## Trust Model @@ -60,9 +60,24 @@ backstop, not proof that arbitrary application or scheduler divergence cannot exist, and it does not replace the watchdog. The mechanism and its bounds are recorded in [`docs/invariants.md`](docs/invariants.md) (I9 and I15). -The third case is handled by the recovery subsystem. Batches that are too old when they reach L1 (`inclusion_block − safe_block ≥ MAX_WAIT_BLOCKS`) are skipped by the scheduler. This "staleness" poisons the nonce counter: all subsequent batches become unreachable regardless of their individual freshness. The sequencer detects this via a danger-zone threshold, preemptively goes offline, flushes the L1 mempool, and cascade-invalidates the doomed chain. See [`docs/recovery/`](docs/recovery/) for the full design, TLA+ formal verification, and design history. +## Recovery -The sequencer trusts its own code is bug-free. Recovery means recovery from liveness failures, which can legitimately happen even in the absence of bugs (infrastructure outages, network failures, gateway failure). Code-level bugs are a separate problem handled by tests and review. See [`docs/threat-model/README.md`](docs/threat-model/README.md) for the complete threat model applied across the codebase. +**[Standard recovery](docs/recovery/README.md)** runs automatically at startup +using the existing database. It handles liveness failures such as outages and +extended downtime: reconcile L1 outcomes, invalidate the affected optimistic +suffix, and resume from retained state. Stale batches do not consume the +scheduler's expected nonce, so their successors cannot be accepted until recovery +supplies a replacement at that nonce. + +**[Cockroach recovery](docs/recovery/cockroach.md)** is an operator-triggered +rebuild when the local database is lost or cannot be trusted. This includes a +sequencer bug that corrupted state or emitted malformed batches: fix the bug, +choose a trusted canonical application checkpoint, then rebuild in a fresh data +directory. The command processes historical L1 inputs through the canonical +scheduler and prepares a baseline for resuming normal operation. + +The [threat model](docs/threat-model/README.md#self-trust) explains the boundary +between normal operation's self-trust and manual repair after a bug. ## Failure Modes @@ -70,7 +85,7 @@ The sequencer is designed to handle: - **L1 provider outages** — workers retry with exponential backoff. The inclusion lane and API continue operating locally. A wall-clock fallback detects when an outage pushes batches into the danger zone. - **Undiagnosed interruptions (OOM, SIGKILL, reboot)** — restart can recover automatically: every boot derives any required recovery from SQLite and L1 safe state through startup recovery, never assuming the previous exit was clean. Terminal errors returned through a command bracket best-effort record their cause in `terminal_faults`; terminal runtime aborts leave only process diagnostics. -- **Extended downtime** — startup syncs to the current L1 safe head, flushes if needed, and recovers before admission; restart policy is the exit-code contract (a terminal exit means: do not restart, page an operator — the one manual remedy is a fresh-directory `setup --recovery` after canonical divergence). +- **Extended downtime** — startup syncs to the current L1 safe head, flushes if needed, and recovers before admission. A terminal exit requires operator investigation; rebuilding untrustworthy state follows the cockroach recovery procedure above. - **Adversarial L1 mempool** — block builders and private mempools are treated as adversarial. The recovery flusher consumes every pending nonce slot with a no-op so delayed "zombie" submissions cannot land later. ## Interfaces @@ -91,11 +106,15 @@ The batch submitter posts closed batches to L1's InputBox contract. Each batch c The sequencer runs in two phases. **`setup`** pins the deployment identity (including the reviewed fee-oracle source), does the initial L1 sync, and registers the genesis -snapshot — run it once. It is L1-read-only: it takes the batch-submitter +snapshot — run it once. Plain `setup` is L1-read-only: it takes the batch-submitter *address*, never the signing key. **`run`** boots the sequencer from the set-up DB, reading identity from it (so chain id / app address are not `run` arguments); it holds the signing key because it submits. +For rebuilding from a trusted checkpoint, follow the +[cockroach recovery procedure](docs/recovery/cockroach.md#run-a-rebuild). +`setup --recovery` also needs the submitter key because it flushes transactions. + ```bash # Phase A — set up the data dir (run once; idempotent). CARTESI_SEQUENCER_BLOCKCHAIN_HTTP_ENDPOINT=http://127.0.0.1:8545 \ @@ -310,21 +329,18 @@ docker pull ghcr.io/cartesi/sequencer-watchdog:vX ## Development -```bash -cargo check # compile -cargo test --workspace --exclude canonical-test # test (canonical-test needs libslirp) -cargo fmt --all # format -cargo clippy --all-targets --all-features -- -D warnings # lint -``` - -Some tests require [Foundry](https://getfoundry.sh) (`anvil` on PATH). They run by default and fail with a clear message if unavailable. This project uses Nix + direnv for tooling — `direnv allow` provides Foundry, TLA+, and other dependencies. +The shared [development commands](AGENTS.md#shell-and-commands) cover Rust +toolchain selection, Nix/direnv tooling, compilation, tests, formatting, and +linting. Read the [testing guidance](AGENTS.md#testing-guidance) before choosing +validation for a change; some tests require Anvil or libslirp. ## Further Reading - [`AGENTS.md`](AGENTS.md) — developer guide: architecture, conventions, duality, recovery, invariants, rules. -- [`CLAUDE.md`](CLAUDE.md) — quick reference for shell setup and commands. +- [`CLAUDE.md`](CLAUDE.md) — Claude entrypoint to the shared agent guide. - [`docs/threat-model/README.md`](docs/threat-model/README.md) — trust boundaries, in-scope and out-of-scope threats. -- [`docs/recovery/README.md`](docs/recovery/README.md) — recovery design, TLA+ formal verification, design history. +- [`docs/recovery/README.md`](docs/recovery/README.md) — automatic recovery, TLA+ formal verification, design history. +- [`docs/recovery/cockroach.md`](docs/recovery/cockroach.md) — manual rebuild after lost state or a sequencer bug. - [`docs/watchdog/getting-started.md`](docs/watchdog/getting-started.md) — step-by-step: run the watchdog with a local sequencer. - [`docs/watchdog/operator-deployment.md`](docs/watchdog/operator-deployment.md) — watchdog on live L1 (Sepolia staging, mainnet production). - [`docs/watchdog/README.md`](docs/watchdog/README.md) — watchdog architecture, modules, and test commands. diff --git a/docs/recovery/README.md b/docs/recovery/README.md index 32dc38f..6513d28 100644 --- a/docs/recovery/README.md +++ b/docs/recovery/README.md @@ -1,8 +1,20 @@ # Batch Recovery -This document describes the recovery design for the sequencer: how the system detects that batches are failing to land on L1, how startup recovers to a consistent state, and where runtime authority begins. Two complementary bounded TLA+ models cover the design: [`preemptive.tla`](preemptive.tla) for batch/slot safety and [`admission.tla`](admission.tla) for startup phase ordering and admission. They do not currently model the external era/generation/base metadata or the canonical `application_inputs` projection or snapshot artifact/GC lifecycle; their crash atomicity is enforced by the SQLite transaction boundaries and schema triggers described below. - -See `AGENTS.md` "Batch Staleness and Recovery" for quick-reference tables and function names. +There are two recovery paths: + +- **Standard recovery**, described on this page, runs automatically at startup. + It uses the existing database to repair an optimistic suffix that can no + longer be relied on and decides whether the sequencer can resume serving. +- **[Cockroach recovery (`setup --recovery`)](cockroach.md)** is an operator-triggered + rebuild from a trusted canonical application checkpoint and L1. Use it after + database loss or unusable local state, including after fixing a sequencer bug. + +Two complementary bounded TLA+ models cover standard recovery: +[`preemptive.tla`](preemptive.tla) for batch/slot safety and +[`admission.tla`](admission.tla) for startup phase ordering and admission. They +do not currently model the external era/generation/base metadata, the canonical +`application_inputs` projection, or snapshot artifact/GC lifecycle; their crash +atomicity is enforced by SQLite transaction boundaries and schema triggers. ## Runtime lifecycle at a glance @@ -33,7 +45,8 @@ Batches form a tree where each node is a batch and edges point from child to par Batches have two identifiers: - **Index** (`batch_index`): monotonically increasing, unique, never reused. Creation order. -- **Nonce** (`batch_nonce`): depth of the node in the tree. Assigned by the batch submitter to valid closed batches. +- **Nonce** (`batch_nonce`): scheduler sequence number. Storage derives it as + `parent.nonce + 1`, or the deployment's anchor nonce for a parentless root. In normal operation the tree degenerates into a list -- index and nonce increase in lockstep. Branches appear only after recovery, when a suffix of the chain is invalidated and a new batch forks from the last valid ancestor. @@ -49,9 +62,11 @@ The implementation handles the nonce-0 case **structurally**: `open_fresh_tip_in #### Cockroach recovery generalizes the root nonce (the anchor) -Cockroach recovery (`setup --recovery`) rebuilds a wiped DB from a trusted checkpoint and must resume submitting at nonce `N'` without replaying history — so the rebuilt tree is rooted at `N'`, not 0. Rather than plant a fake "sentinel" batch at `N'-1`, the batch-tree anchor generalizes the structural root: a `batch_tree_anchor` singleton holds the nonce the parentless root carries (default `0`; recovery sets `N'`). The same `open_fresh_tip_in_tx` / `compute_next_nonce(parent = None)` path then roots `run`'s first tip at `N'`, and `trg_enforce_nonce_contiguity` validates the root against the anchor (exact match) instead of a hard-coded 0. There is **no sentinel batch row** — the root tip *is* the anchored batch. Normal deployments keep anchor `0` and are byte-identical. See [I16](../invariants.md) and the [cockroach-recovery design](#cockroach-recovery-setup---recovery) below. - -A sealed `N'-1` sentinel was considered and rejected: a valid closed batch at `N'-1` is a legal cascade pivot, so a runtime cascade could invalidate it and leave the tree re-rooting at 0 (ABORTed by the unchanged contiguity trigger) — an unguarded reliance on "the frontier never drops to `N'-1`". The anchor has no such hidden dependency. +A fresh deployment roots its batch tree at nonce 0. Cockroach recovery roots it +at the scheduler's next nonce after replay. The `batch_tree_anchor` preserves +that starting nonce even if a later cascade removes the whole local branch. +[I16](../invariants.md#i16-the-batch-tree-has-exactly-one-valid-parentless-root-carrying-the-deployments-anchor-nonce) +owns the root invariant and its enforcement. ## Coloring @@ -372,25 +387,15 @@ Dead batches occupy `w_nonce` slots strictly below `walletNonce`. Recovery batch ## Cockroach recovery (`setup --recovery`) -Everything above is **standard recovery**: the sequencer's own bookkeeping -(the batch tree, pending dumps) lets startup cascade a doomed suffix and -resume. The repair decision is automatic, not an operator-designed -reconstruction: recovery crosses a process boundary, and the next boot -inspects fresh facts regardless of how the prior process -died. Admission and the terminal-fault black box are owned by -[ADR mechanism 2](../plans/2026-08-authority-boundary-adr.md#2-fact-derived-admission-and-the-terminal-fault-black-box). - -**Cockroach recovery** is the catastrophe path — the local DB is lost or has diverged (`CanonicalDivergence`, [I15](../invariants.md)). There is no tree to cascade; the operator supplies a fresh or explicitly wiped data directory and rebuilds canonical logical state from a trusted checkpoint plus L1. It is an operator-driven, one-shot `setup` mode, not a runtime action. There is no automated DB replacement, clone detection, distributed fencing, or partial-fill resume state machine. The summary: - -Given a trusted checkpoint machine `S` at block `B` (a finalized `dumps//` dir, carrying `N` = its resume nonce and `A` = its last-executed safe block), `setup --recovery --checkpoint-block B --checkpoint-dump-dir ` runs **flush → fold → fill**: - -1. **Flush** the wallet nonce (keyed — recovery, unlike plain `setup`, signs) so every previous-instance batch resolves at safe depth `≤ C`, the post-flush safe head. Re-sync `safe_inputs` through `C`. -2. **Fold** (the pure `sequencer-core` engine, shared with the on-chain scheduler so it is consistent by construction): seed the fridge from the `(A, B]` directs (drop batches — already in `S`), replay the `(B, C]` stream, drain the leftover fridge at `C`. Yields `(S', N')` = the advanced app state and the resume nonce. -3. **Fill** a consistent DB: write the recovered application dump first, then atomically register its complete `(era, generation = 0, K, C)` history baseline, anchor `N'`, parentless root frame at `C`, snapshot, and `setup_complete`. The collapsed prefix creates no application rows. The first later application input has offset `K`; ordinary recovery falls back to immutable `C` if the root is invalidated. The terminal-drained baseline is a local restore point and is not automatically a canonical comparison checkpoint at `C`. - -During rebuild the accepted frontier is deferred until the baseline exists. The first `run` sync seeds expected nonce `N'` and scans only inputs after `C`, explicitly excluding the trusted prefix. Replaying the old prefix with a later expected nonce could reinterpret a rejected future-nonce batch as accepted. Checkpoint state and nonce remain operator-trusted; the export receipt checks metadata agreement rather than independently verifying the checkpoint. Rebuild is one-shot after completion. File-first creation plus atomic registration removes partial-baseline resume states; a failed transaction leaves only an orphan artifact. See [cockroach recovery](cockroach.md) for the full contract. +Use [cockroach recovery](cockroach.md) when the database is lost or local state +is unusable, including after a sequencer bug. Fix the bug first, then rebuild in +a fresh data directory from a trusted, sufficiently advanced canonical +application checkpoint and L1. The recovery guide owns checkpoint requirements, +the replay procedure, and the resulting resume baseline. -The detect-and-refuse gate is the *trigger*: a fresh `setup` that finds a previous instance's batches past the checkpoint refuses with exit `40` (`EXIT_SETUP_NEEDS_RECOVERY`), pointing the operator here. +Plain `setup` also directs the operator to this path when it detects a previous +instance's batches past the checkpoint: it refuses with exit `40` +(`EXIT_SETUP_NEEDS_RECOVERY`). ## Canonical divergence (terminal, outranks every arm) diff --git a/docs/recovery/cockroach.md b/docs/recovery/cockroach.md index f02510f..0016e26 100644 --- a/docs/recovery/cockroach.md +++ b/docs/recovery/cockroach.md @@ -1,119 +1,131 @@ -# Cockroach recovery (`setup --recovery`) - -When the local DB is lost or has diverged, the operator rebuilds from a trusted -application checkpoint and L1 in a fresh data directory. This is a one-shot -setup operation. [Standard recovery](README.md) instead keeps the database and -invalidates an unaccepted suffix. - -The [canonical scheduler fold](../../sequencer-core/src/scheduler/fold.rs) -reconstructs the state. Its terminal drain also prepares the next local frame: -the result is a **resume baseline**, which can be ahead of canonical application -execution at the stopping block. It is not exposed as a finalized comparison -checkpoint merely because the L1 inputs used to construct it are safe. - -## Data dictionary - -| Symbol | Meaning | Source | -|---|---|---| -| `S` | Trusted application state at checkpoint block `B`. | Restored application dump. | -| `A` | Last executed application safe block in `S`; pending directs are seeded from `(A, B]`. | Application progress in the dump. | -| `B` | Checkpoint inclusion block. | Exported `checkpoint.toml`; must equal the configured checkpoint block. | -| `N` | Scheduler's next batch nonce at `B`. | Exported receipt, checked against immutable `info.toml`. | -| `C` | Post-flush safe stopping block. | Flusher result. | -| `N'` | Next batch nonce after folding through `C`; the new batch-tree anchor. | Fold result. | -| `K` | Application count after the terminal drain; first local history entry is `K`. | Recovered application progress. | -| `E`, `g` | New UUIDv4 era and generation zero. | Atomic completed-baseline registration. | - -The checkpoint contract requires `A < B`, checked at load. The sole exception -is the known empty genesis checkpoint (`B = 0`, next nonce and app count zero). -At a non-genesis `A = B`, a direct arriving after the accepted batch in block -`B` can still be pending; the empty `(A, B]` seed would silently omit it. -A recovery export carries a canonical application dump and a separate receipt; -baseline downloads and ordinary optimistic snapshots have no such receipt. - -### Trusted checkpoint boundary - -The application state, resume nonce, and relationship between the checkpoint and -L1 are operator-trusted. The receipt catches accidentally mixing an artifact, -nonce, or configured inclusion block; it does not independently verify state -against L1. A wrong checkpoint nonce, whether low or high, is outside the -supported model. The content-identity check verifies newly observed acceptance -after the baseline, not the opaque prefix or checkpoint correctness. - -An independent verification would need a trusted canonical-machine checkpoint -or replay from an independently trusted origin. The infrastructure subscriber's -application dump is not a substitute for that watchdog trust boundary. - -## The procedure: flush → fold → fill - -1. **Load the checkpoint.** Restore `S`, read both metadata files, verify their - nonce agreement and the configured `B`, then derive `A` and require `A < B` - or the known empty genesis checkpoint. -2. **Flush stranded transactions.** Consume unresolved wallet nonce slots and - wait for safe finality, obtaining `C`. The lost database cannot supply its - previous watermark, so the flush uses the provider's pool view. A dropped - transaction alive elsewhere can evade that view; a later accepted foreign or - mismatched landing after `C` freezes the new instance and requires another - rebuild. The trusted provider is fail-stop, not Byzantine. -3. **Re-sync raw L1 inputs.** The safe head `H1` must cover `C`; it can be later. - Acceptance projection is deferred while the new local tree is absent. -4. **Source disjoint fold ranges.** Seed external directs in `(A, B]`, then - replay all raw inputs in `(B, C]`. Sender classification excludes own batch - envelopes from the direct-input seed queue. -5. **Fold and drain.** The scheduler processes the stream with expected nonce - `N`, then drains every remaining direct through `C`, producing `(S', N')`. - A young direct still waiting in the canonical scheduler may therefore already - be present in `S'`. The resumed frame covers it before executing new user ops. -6. **Write the baseline artifact, then publish it.** First create and durably - sync the immutable dump. One SQLite transaction then creates history - `(E, 0, K, C)`, sets anchor `N'`, opens its parentless root frame at `C`, - registers the baseline artifact, and records `setup_complete`. It creates no - application-input rows for the collapsed prefix. - -On the first `run` sync, acceptance starts at `N'` and scans only raw inputs -whose block is **strictly greater than `C`**. Nonce filtering alone is unsound: -a previously rejected future-nonce batch inside the old prefix could match the -new expected nonce. The opaque prefix is never classified again. - -Inputs in `(C, H1]` remain available to the inclusion lane. Its next complete -reconciliation executes them once and records application entries beginning at -`K`. Raw L1 input indices and application offsets remain separate coordinates. - -## Recovery and retention - -`C` is the immutable fallback reconciliation boundary. While valid frames -survive, their latest `safe_block` gives the already-reconciled boundary. If -standard recovery invalidates the original root, it falls back to `C`, so -inputs represented by the baseline are never executed again. Canonical -application rows belonging to invalidated batches are deleted atomically with -the generation change and suffix invalidation. - -Startup loads the latest surviving batch-close snapshot, falling back to the -baseline. Admission requires a **rollback-safe checkpoint**: either that -baseline or a retained accepted batch snapshot. An optimistic snapshot alone -cannot satisfy this requirement because a cascade may discard its whole suffix. - -Once an accepted post-baseline batch snapshot exists, standard recovery cannot -invalidate it or return to the original baseline. GC can retire the baseline -artifact, subject to download leases. Immutable baseline metadata remains. -Snapshots with equal application counts remain distinct artifacts associated -with distinct batches; acceptance and retention never infer identity from count. - -## Crash-safety & idempotency - -A completed rebuild refuses another `setup --recovery`. Before completion, -there is no partially registered history or recovery root to resume: - -- A failure during artifact creation leaves setup incomplete. -- A failed registration transaction leaves neither baseline history, root, - anchor update, snapshot row, nor completion marker; any durable file is an - orphan for cleanup. -- A successful transaction establishes all those facts together. There are no - nullable baseline coordinates and no physical replay padding. - -Early identity pinning and raw L1 ingestion can survive an incomplete attempt. -They do not establish an application-history era. The process lock and setup -admission exclude runtime serving before the complete baseline exists. +# Cockroach recovery: rebuild from L1 + +Cockroach recovery (`setup --recovery`) creates a fresh sequencer starting state +from a trusted application checkpoint and historical L1 inputs. Use it when the +local database is lost or cannot be trusted, including after a sequencer bug has +corrupted its state or produced malformed batches. **Find and fix the bug before +rebuilding.** The operator initiates recovery; the command automates the rebuild. + +The procedure is **flush → fold → fill**: + +1. **Flush** outstanding submitter transactions and choose a fixed safe L1 + stopping block. +2. **Fold** the input history through the canonical scheduler, starting from the + trusted checkpoint. Every input receives its normal scheduler treatment: + accepted batches execute, malformed or rejected batches are skipped, and + direct inputs are queued and drained. Finally, drain every remaining direct + input through the stopping block. +3. **Fill** a fresh database with the recovered application state and next batch + nonce, ready for `run`. + +The result has accounted for the whole input prefix and carries no inherited +speculative user-operation suffix. It does not need to reach the moving tip: +normal operation handles inputs after the stopping block. + +This is a **resume baseline**. The final drain can execute young direct inputs +that the canonical scheduler still has queued at that block. The resumed frame +covers them before new user operations; the baseline itself is not a canonical +comparison checkpoint at that block. + +[Standard recovery](README.md) instead uses the existing database to repair an +optimistic suffix automatically. It assumes the local bookkeeping is trustworthy. + +## Run a rebuild + +Stop the old sequencer and resolve the cause of the failure. Choose a trusted +canonical application checkpoint; a recent one reduces replay work. Its state, +inclusion block, and next batch nonce must agree. After a bug, establish that +trust independently of the faulty local state, using a trusted canonical-machine +checkpoint or replay from a trusted origin. + +The loader requires an application artifact, `info.toml`, and `checkpoint.toml`. +The [recovery export workflow](../snapshots/lifecycle.md#http-and-recovery-exports) +describes this bundle. An export receipt checks metadata agreement; it does not +prove the checkpoint correct. An ordinary optimistic snapshot, a subscriber's +dump, or a bare local `dumps//` directory is insufficient. + +With the deployment's [setup configuration](../../README.md#running) and +batch-submitter signing key configured, use a fresh data directory: + +```sh +cargo run -p wallet-sequencer -- setup --recovery \ + --data-dir \ + --checkpoint-block \ + --checkpoint-dump-dir +``` + +Recovery signs L1 transactions, so the key must match the configured submitter. +After success, start `run` with that same data directory. A completed rebuild +refuses another `setup --recovery`; failures before completion publish no partial +baseline. + +## Implementation contract + +Read this section when changing checkpoint loading, replay, or baseline +publication. The [scheduler contract](../protocol/scheduler-semantics.md) owns +input interpretation; recovery uses that same scheduler implementation. + +### Data dictionary + +The replay boundaries are: + +| Value | Meaning | +|---|---| +| `S`, `B`, `N` | Trusted checkpoint state, inclusion block, and next batch nonce. | +| `A` | Last executed application safe block reported by `S`. | +| `C` | Fixed post-flush safe stopping block. | +| `S'`, `N'` | Recovered state and next batch nonce. | +| `K` | Application count in `S'`; the first later application input has offset `K`. | + +Loading checks the receipt's block against configured `B` and its nonce against +`info.toml`. It requires `A < B`, except for known empty genesis (`B`, nonce, and +application count all zero). At non-genesis `A = B`, a direct arriving after the +accepted batch in block `B` could still be pending but disappear from the seed +range. Checkpoint state and nonce remain operator-trusted; the later +content-identity check does not verify this prefix. + +### Flush and stopping block + +The lost database cannot supply its previous wallet-nonce watermark. Flushing +therefore depends on the provider's pool view. A transaction dropped there but +alive elsewhere may escape; a later accepted foreign or mismatched landing +freezes the rebuilt instance and requires another rebuild. The provider is +trusted fail-stop, as specified in the [threat model](../threat-model/README.md). + +After flushing, raw L1 ingestion must reach at least `C`. It may advance farther, +but the fold stops at `C`. Accepted-batch projection is deferred until the new +baseline and batch tree exist. + +### Replay boundaries + +Seed the scheduler's pending-direct queue from `(A, B]`, excluding inputs sent by +the batch submitter. Then replay **all raw inputs** in `(B, C]` in L1 order with +expected nonce `N`. Drain the remaining directs through `C` to obtain `(S', N')`. +The disjoint ranges preserve pending directs without executing the checkpoint's +accepted batches again. + +On the first `run` sync, acceptance starts at nonce `N'` and scans only blocks +**strictly after `C`**. Nonce filtering alone would let a previously rejected +future-nonce batch in the old prefix be reinterpreted as accepted. Raw inputs +ingested beyond `C` remain for normal reconciliation. Newly executed application +inputs are recorded beginning at `K`; raw L1 indices and application offsets are +separate coordinates. + +### Publish the baseline + +Write and durably sync the immutable application dump first. One SQLite +transaction then registers a fresh UUIDv4 era at generation zero, count `K`, +boundary `C`, anchor nonce `N'`, a parentless root frame at `C`, the snapshot, and +`setup_complete`. The collapsed prefix creates no application-input rows. + +Artifact failure leaves setup incomplete; transaction failure leaves at most an +orphan artifact. Identity pinning and raw L1 ingestion may survive an incomplete +attempt, but the lock and setup admission prevent serving a partial baseline. + +`C` remains the fallback reconciliation boundary if standard recovery invalidates +the root. The [history contract](../plans/application-history.md#era-baseline) +owns these immutable coordinates; [snapshot lifecycle](../snapshots/lifecycle.md) +owns restore selection, rollback-safe retention, and eventual baseline disposal. ## Code map @@ -124,4 +136,3 @@ admission exclude runtime serving before the complete baseline exists. | Atomic baseline completion | [`storage/lifecycle.rs`](../../sequencer/src/storage/lifecycle.rs) | | Scheduler fold | [`scheduler/fold.rs`](../../sequencer-core/src/scheduler/fold.rs) | | Accepted-prefix boundary | [`storage/safe_accepted_batches.rs`](../../sequencer/src/storage/safe_accepted_batches.rs) | -| Snapshot selection and GC | [`storage/snapshot_dumps.rs`](../../sequencer/src/storage/snapshot_dumps.rs) | diff --git a/docs/snapshots/README.md b/docs/snapshots/README.md index 48e8e7e..5b983d3 100644 --- a/docs/snapshots/README.md +++ b/docs/snapshots/README.md @@ -1,9 +1,9 @@ # Snapshots -Application snapshots are durable copies of the app's canonical state at a known -point in the L2-tx stream. They let the inclusion lane resume on startup with a -single *load-then-replay* instead of replaying all history, and they back the -operator's watchdog (`/finalized_state`) and indexers (`/latest_snapshot`). +Application snapshots are immutable, durable copies of application state at a +known execution boundary. They let the inclusion lane resume with load and +replay. A snapshot may contain optimistic state; L1 acceptance determines which +artifact can back a canonical comparison or recovery export. Two documents, split by concern: @@ -11,13 +11,12 @@ Two documents, split by concern: trait (`from_dump` / `create_dump` / `state_file_in_dump`) and the toy wallet's SSZ wire encoding. What a dump *is*. -- **[`lifecycle.md`](lifecycle.md)** — the *lifecycle* and its rationale: take at - batch close, pending → finalized promotion (per-range, atomic with the drain), - garbage collection, HTTP leasing, recovery interaction, and the crash-safety - reasoning (including the promote/drain wedge and why the design closes it). - When and how dumps move through the system, and *why*. +- **[`lifecycle.md`](lifecycle.md)** — creation at batch close, restart selection, + acceptance-derived comparison checkpoints, recovery exports, retention, + download leases, and crash safety. Acceptance is a separate durable fact; + artifacts are never promoted or rewritten. -Related: [`../recovery/README.md`](../recovery/README.md) (danger-zone recovery, -which clears cascade-doomed pendings), [`../../AGENTS.md`](../../AGENTS.md) -(architecture), and the root [`../../README.md`](../../README.md) (endpoint -shapes). +For automatic startup repair, see [standard recovery](../recovery/README.md). +For rebuilding after database loss or a sequencer bug, see +[cockroach recovery](../recovery/cockroach.md). The root +[README](../../README.md) owns endpoint shapes. diff --git a/docs/threat-model/README.md b/docs/threat-model/README.md index 077b091..08df4af 100644 --- a/docs/threat-model/README.md +++ b/docs/threat-model/README.md @@ -23,7 +23,7 @@ What we are protecting: | Operator env / CLI flags | Trusted, **mistakes foot-gun-guarded** | Setup configuration is authoritative — including the reviewed Uniswap V3 WETH/fee-token pool the fee oracle quotes. The complete source is pinned in deployment identity; run cannot replace it. The operator is trusted, not infallible: the supported operator-mistake class is *accidental concurrent or stale use of one data directory* — two processes on one dir (kernel process lock), a mistyped `--data-dir` (open refuses paths with no database). Deliberate operator subversion, copied-directory coordination, and distributed fencing remain out of scope (the lock is a cheap local foot-gun guard, not distributed fencing — [ADR mechanism 1](../plans/2026-08-authority-boundary-adr.md#1-runtimescope-structured-process-ownership)); mechanisms defending this class are judged against this boundary rather than re-litigated per review. | | Uniswap V3 pool (fee oracle) | Semi-trusted L1 state | Spot manipulation of a deep pool is mitigated by a 30-minute TWAP plus 10× slack in `batch_policy.log_slack`. Residual risks: TWAP lag during real moves and thin/wrong pool misconfiguration. Multi-hop pricing is out of scope. Setup writes the first Uniswap quote under the same hard L1 requirement as the rest of setup; a failed quote leaves setup incomplete. `run` does not gate recovery or admission on another quote: it starts from the persisted `batch_policy.log_gas_price`, constructs the source from setup-validated identity without RPC, and launches a refresher that logs/retries transient quote failures indefinitely while retaining that price. `log_gas_price_updated_at_ms` records successful observation for telemetry; it is not an expiry gate. A shared-endpoint outage or stale view is already caught by safe-head progress, while a pool/`observe`-specific failure with a healthy input reader is accepted as an unbounded economic residual: stale-low pricing can subsidize DA/weaken the fee spam barrier and stale-high pricing can reject users, but neither changes canonical execution because the frame's persisted fee is immutable and enforced by both sides. The 10× slack is a margin, not a proof against arbitrary market movement. Deterministic setup-time source misconfiguration (`WrongTokenPair`, `MissingPoolCode`, chain-id mismatch), fatal arithmetic, and persistent storage faults remain terminal. | | Batch-submitter private key | Private | Held in operator infra. Not reachable by the network. | -| Sequencer's own code | Trusted (bug-free is a precondition) | Bugs are prevented through tests/review and contained by fail-loud runtime invariant checks; they are not treated as adversarial behavior that the protocol can recover around. See "self-trust" below. | +| Sequencer's own code | Trusted during normal operation | Tests/review prevent bugs; runtime invariant checks fail loud. Bugs require diagnosis and correction, followed by an operator rebuild if local state cannot be trusted. See "self-trust" below. | | **L1 mempool and block builders** | **Fully adversarial** | May reorder, delay, drop, or selectively include submitted transactions. Private mempools mean "dropped" is indistinguishable from "delayed indefinitely." | | HTTP clients at `POST /tx` and `GET /fee` | Untrusted | Arbitrary public callers. May submit malformed, malicious, or replay payloads. `GET /fee` is an intentional public quote of the open-frame fee. | | WebSocket subscribers at `/ws/subscribe` | Internal, but untrusted for data-exposure | Intended for internal indexers. Treat as public for what is exposed. | @@ -31,7 +31,14 @@ What we are protecting: ### Self-trust -The sequencer trusts its own code in a specific sense: **impossible states are never *handled*.** There are no graceful fallback paths, no re-validation of a neighbor module's answer, no code that keeps running past a violated internal contract. If the sequencer emits a malformed batch, frame, or user op, it is in a bug state that requires manual intervention; normal preemptive recovery addresses liveness failures (infrastructure outages, network partitions, gateway failure), not bug-induced malformed state. Cockroach recovery is the separate operator-directed rebuild path when durable state cannot be trusted. +The sequencer trusts its own code in a specific sense: **impossible states are never *handled*.** There are no graceful fallback paths, no re-validation of a neighbor module's answer, no code that keeps running past a violated internal contract. Normal preemptive recovery addresses liveness failures such as infrastructure outages, network partitions, and gateway failure. + +If a bug corrupts local state or causes the sequencer to emit malformed batches, +frames, or user operations, the operator must diagnose and fix it. Then +[cockroach recovery](../recovery/cockroach.md) can rebuild from a trusted canonical +application checkpoint and L1. The checkpoint must be trusted independently of +the faulty local state. Historical inputs, including malformed batches, receive +the canonical scheduler's normal treatment during replay. This is **not** a prohibition on checking. Internal invariants are enforced loudly wherever a check is near-free — the type system, SQL constraints and triggers, boundary assertions — because failing loud preserves safety, while a silently-tolerated bug that externalizes (a signed batch, an ack, a feed event) is state divergence: as severe as theft and undefendable at runtime. Loud failure is not automatically self-healing: transient faults may clear on restart, but persistent invalid state is terminal and may require inspection or cockroach recovery. The rule, in short: **assert real invariants, fail loud, never absorb silently, never handle gracefully.** The decision test and the register of cross-module invariants live in [`docs/invariants.md`](../invariants.md). @@ -69,11 +76,11 @@ blocking production diagnostics would require revisiting that assumption rollbackable soft confirmations, the watchdog byte-compare, and the I15 divergence freeze. - **Adversarial mempool:** reorder, delay, drop, selective inclusion by builders -- **Zombie transactions:** a submitted batch may sit in a private mempool indefinitely and land long after we believed it was gone. Two load-bearing defenses: the recovery flusher consumes every wallet-nonce slot this deployment ever used (anchored by the persisted watermark, I14) so zombies cannot claim them; and the content-identity check (I9/I15) compares every at/above-anchor *simulated-accepted* landing against the valid closed batch we sealed at that nonce. A foreign or byte-different landing records divergence when it becomes safe and is ingested, freezes the accepted frontier, and requires cockroach recovery. This is trust-boundary validation of external input (the mempool replaying our own stale transactions at times we don't control), not defense-in-depth against self-bugs or a general canonical-state oracle. In cockroach recovery the watermark does not survive the wipe, so that flush is best-effort by construction; the content-identity check is what keeps the residual zombie detected-and-frozen rather than silent (see `docs/recovery/cockroach.md`, step 2). +- **Zombie transactions:** a submitted batch may sit in a private mempool indefinitely and land long after we believed it was gone. Two load-bearing defenses: the recovery flusher consumes every wallet-nonce slot this deployment ever used (anchored by the persisted watermark, I14) so zombies cannot claim them; and the content-identity check (I9/I15) compares every at/above-anchor *simulated-accepted* landing against the valid closed batch we sealed at that nonce. A foreign or byte-different landing records divergence when it becomes safe and is ingested, freezes the accepted frontier, and requires cockroach recovery. This is trust-boundary validation of external input (the mempool replaying our own stale transactions at times we don't control), not defense-in-depth against self-bugs or a general canonical-state oracle. In cockroach recovery the watermark does not survive the wipe, so that flush is best-effort by construction; the content-identity check is what keeps the residual zombie detected-and-frozen rather than silent (see [the flush boundary](../recovery/cockroach.md#flush-and-stopping-block)). - L1 reorgs up to safe depth - Malicious `POST /tx` callers: malformed signatures, spoofed sender, replay across chains or apps, nonce manipulation - Malicious direct-input senders: arbitrary payload, any intent; sender authenticity is guaranteed by InputBox -- Scheduler/sequencer protocol divergence of any kind (ordering, nonce rules, signature validity, fee semantics) is an in-scope correctness consequence. The content-identity check detects accepted-batch identity failures only; there is no complete runtime detector for the broader class. Shared semantics, review, and tests are preventative, and cockroach recovery is the remedy only after another signal or operator investigation diagnoses divergence. +- Scheduler/sequencer protocol divergence of any kind (ordering, nonce rules, signature validity, fee semantics) is an in-scope correctness consequence. The content-identity check detects accepted-batch identity failures only; there is no complete runtime detector for the broader class. Shared semantics, review, and tests are preventative, and cockroach recovery rebuilds state after another signal or operator investigation diagnoses divergence and its cause has been corrected. ## Out of scope From 15f799849c06624049fe277afc8adf0b945526cb Mon Sep 17 00:00:00 2001 From: gcdepaula Date: Thu, 17 Sep 2026 15:22:58 -0300 Subject: [PATCH 2/4] docs: ground automatic recovery in implementation --- AGENTS.md | 2 +- docs/invariants.md | 8 +- docs/l1-fee-policy.md | 2 +- docs/recovery/README.md | 798 ++++++++---------- docs/recovery/history/README.md | 89 +- docs/snapshots/lifecycle.md | 9 +- sequencer/src/commands/config.rs | 6 +- sequencer/src/commands/run/startup_hygiene.rs | 2 +- sequencer/src/commands/run/workers.rs | 2 +- .../src/ingress/inclusion_lane/catch_up.rs | 2 +- sequencer/src/recovery/flusher.rs | 8 +- sequencer/src/storage/ingress.rs | 5 +- sequencer/src/storage/mutations.rs | 8 +- sequencer/src/storage/recovery.rs | 193 +---- 14 files changed, 454 insertions(+), 680 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 720c22d..ba2494d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -184,7 +184,7 @@ Paths below are relative to `sequencer/src/`: - **Frame** — ordering boundary; commits `safe_block` + user ops. - **Batch** — list of frames posted on-chain as one L1 transaction (SSZ-encoded). - **Inclusion lane** — the single ordering lane, with a latency-critical user-op regime and a slower L1-reconciliation regime ([ADR mechanism 4](docs/plans/2026-08-authority-boundary-adr.md)); the only writer of open batch/frame state ([I17](docs/invariants.md)) and the system's execution bottleneck. -- **Batch submitter** — stateless worker that bulk-submits all pending batches each tick. Nonces are assigned by storage (structural `parent.nonce + 1`) when batches are closed; the submitter just reads them. +- **Batch submitter** — stateless worker that bulk-submits all pending batches each tick. Storage assigns each batch's scheduler nonce at creation (`parent.nonce + 1`, or the deployment anchor for a root); the submitter reads it and selects L1 wallet nonces for submission. - **Danger detector** — polls `Storage::check_danger` and signals the process to stop so startup can recover or refuse. It reads local facts; it never writes the DB or talks to L1. - **Fee oracle** — setup pins and bootstraps a fixed price or Uniswap V3 TWAP source. The price informs future frame fees; an oracle-only outage is an accepted economic risk. The [threat model's actor table](docs/threat-model/README.md#actors-and-trust) owns the source assumptions and failure policy. - **Input reader** — ingests safe inputs from L1 InputBox and maintains the durable safe head, accepted-batch projection, and divergence marker in one atomic transaction (`sequencer/src/storage/l1_inputs.rs`); it hands the lane no in-memory cursor. diff --git a/docs/invariants.md b/docs/invariants.md index 802fe88..f9e44ac 100644 --- a/docs/invariants.md +++ b/docs/invariants.md @@ -154,8 +154,8 @@ by writer and are write-once (`0001_schema.sql`). - **Holds:** `check_danger` checks `ClosedBatchInDanger` before `TipInDanger`. With monotonic frame clocks, the closed frontier is at least as old as the Tip. - **Enforced by:** arm order in `storage/recovery.rs` and I3. -- **Depended on by:** dispatch: a Tip-only recovery can skip flushing because - there is no doomed closed work. +- **Depended on by:** dispatch: closed-frontier danger selects flush before + a Tip-only repair can be selected. The Tip itself has no L1 footprint. ### I5. Recovery removes exactly the invalidated application suffix @@ -296,8 +296,8 @@ by writer and are write-once (`0001_schema.sql`). raises through `WalletNonceWatermarkSink` before its first send; `MempoolFlusher::flush_and_wait` likewise before its no-ops, and refuses to complete until `safe >= watermark + 1`. -- **Depended on by:** flush completeness, TLA+ Implementation Constraint 1, - cascade soundness (I9). +- **Depended on by:** [flush completeness](recovery/README.md#closed-batches-flush-sync-cascade) + and cascade soundness (I9). - **Breaks:** zombie txs evade the flush — a dropped-locally but network-surviving batch tx re-lands at a slot the recovery batch reuses, and the scheduler executes invalidated content. diff --git a/docs/l1-fee-policy.md b/docs/l1-fee-policy.md index 40ed078..863c625 100644 --- a/docs/l1-fee-policy.md +++ b/docs/l1-fee-policy.md @@ -38,7 +38,7 @@ expectation to measure, not a bound this policy establishes. The danger detector stops normal operation when the configured danger threshold is reached. That bounds continued soft-confirmation issuance under the detector's assumptions; it does not bound inclusion or recovery duration. -[Recovery](recovery/README.md#step-4-post-flush-state) requires all covered +[Recovery](recovery/README.md#closed-batches-flush-sync-cascade) requires all covered wallet slots to resolve at safe depth before a cascade can proceed. The flusher's fixed headroom can also fail to replace an unmineable original. The sequencer remains offline until recovery succeeds or the operator acts. diff --git a/docs/recovery/README.md b/docs/recovery/README.md index 6513d28..a1bdf1e 100644 --- a/docs/recovery/README.md +++ b/docs/recovery/README.md @@ -1,491 +1,353 @@ -# Batch Recovery - -There are two recovery paths: - -- **Standard recovery**, described on this page, runs automatically at startup. - It uses the existing database to repair an optimistic suffix that can no - longer be relied on and decides whether the sequencer can resume serving. -- **[Cockroach recovery (`setup --recovery`)](cockroach.md)** is an operator-triggered - rebuild from a trusted canonical application checkpoint and L1. Use it after - database loss or unusable local state, including after fixing a sequencer bug. - -Two complementary bounded TLA+ models cover standard recovery: -[`preemptive.tla`](preemptive.tla) for batch/slot safety and -[`admission.tla`](admission.tla) for startup phase ordering and admission. They -do not currently model the external era/generation/base metadata, the canonical -`application_inputs` projection, or snapshot artifact/GC lifecycle; their crash -atomicity is enforced by SQLite transaction boundaries and schema triggers. - -## Runtime lifecycle at a glance - -The sequencer's recovery loop spans two process lifetimes: - -1. **In-process detection.** The `DangerDetector` polls `Storage::check_danger`. Expected-recovery and retryable exits close intake and drain workers before returning non-zero. A terminal fault aborts the process immediately ([ADR mechanism 1](../plans/2026-08-authority-boundary-adr.md#1-runtimescope-structured-process-ownership)). -2. **External respawn.** An orchestrator (systemd, k8s, …) restarts expected-recovery and retryable exits. A terminal exit (30 or SIGABRT) requires operator investigation before a deliberate restart. -3. **Startup recovery.** Under the process lock, before workers exist, startup checks local terminal facts, attempts an initial L1 Sync, then selects at most one repair from a consistent `RecoveryInspection`: open a missing Tip, replace an aging Tip, or Flush → Sync → Cascade. Repair is followed by a fresh check. -4. **Prepare, admit, launch.** A clean result permits task-free, fallible preparation. A final current inspection must still be clean to mint the single-use `RuntimeAdmission` witness; worker launch consumes it synchronously. - -The detector trip and the startup dispatch share the same `check_danger` function; the detector cares only that *some* arm fired, while the startup dispatch examines *which* arm fired to pick the right action. - -Key abstractions, by responsibility: - -- **`DangerDetector`** ([`recovery/detector.rs`](../../sequencer/src/recovery/detector.rs)): reads danger on a cadence and exits on any non-`Safe` status. It writes nothing and performs no L1 calls. -- **`BatchSubmitter`** ([`l1/submitter/worker.rs`](../../sequencer/src/l1/submitter/worker.rs)): makes L1 progress; the detector owns danger checks. -- **Startup recovery** ([`recovery/mod.rs`](../../sequencer/src/recovery/mod.rs)): a sequential procedure with one exhaustive dispatch shared by repair selection and final admission. Error classification belongs here; command settlement consumes the resulting retry/refuse verdict. -- **Guarded recovery storage** ([`storage/recovery.rs`](../../sequencer/src/storage/recovery.rs)): checks each repair's preconditions and commits its mutation atomically. The cascade checks divergence and the flush-view floor before changing the batch tree. -- **`MempoolFlusher`** ([`recovery/flusher.rs`](../../sequencer/src/recovery/flusher.rs)): consumes unresolved wallet-nonce slots and waits for safe finality. Provider errors leave the attempt; the orchestrator retries. -- **`ProtocolTiming`** ([`sequencer-core/src/protocol.rs`](../../sequencer-core/src/protocol.rs)): shared scheduler timing plus sequencer-local danger and clock policy. - -Procedure tests use the real SQLite inspections and repair transactions, substituting only Sync and Flush at the L1 boundary. - -## The Batch Tree - -Batches form a tree where each node is a batch and edges point from child to parent. Each batch has a single parent: the preceding batch in the valid chain. - -Batches have two identifiers: - -- **Index** (`batch_index`): monotonically increasing, unique, never reused. Creation order. -- **Nonce** (`batch_nonce`): scheduler sequence number. Storage derives it as - `parent.nonce + 1`, or the deployment's anchor nonce for a parentless root. - -In normal operation the tree degenerates into a list -- index and nonce increase in lockstep. Branches appear only after recovery, when a suffix of the chain is invalidated and a new batch forks from the last valid ancestor. - -There is always exactly one **valid path** (root to leaf) that constitutes the current batch chain. The valid path splits into a **prefix** (safe on L1, accepted by the scheduler) and a **suffix** (pending or confirming). - -### Genesis sentinel (nonce-0 edge case) - -Recovery requires at least one Gold ancestor (the cascade invalidates a suffix and forks from the last Gold batch). If the very first batch (nonce 0) goes stale before any batch becomes Gold, there is no ancestor to fork from. - -The TLA+ model handles this with a **genesis sentinel**: the initial state starts with a Gold batch at nonce 0. This is a modeling technique that eliminates the nonce-0 special case, allowing Resolve to use uniform logic (the `fng > 1` guard is always satisfied). Without it, the model would need a separate Resolve action with different arithmetic for the "no Gold ancestor" case. - -The implementation handles the nonce-0 case **structurally**: `open_fresh_tip_in_tx` (`storage/ingress.rs`) roots a nonce-0 batch whenever the valid path is empty (genesis, or a fully-torn cascade) — no sentinel batch is submitted and no recovery branch is special-cased. The model's sentinel and the implementation's structural root play the same role. - -#### Cockroach recovery generalizes the root nonce (the anchor) - -A fresh deployment roots its batch tree at nonce 0. Cockroach recovery roots it -at the scheduler's next nonce after replay. The `batch_tree_anchor` preserves -that starting nonce even if a later cascade removes the whole local branch. -[I16](../invariants.md#i16-the-batch-tree-has-exactly-one-valid-parentless-root-carrying-the-deployments-anchor-nonce) -owns the root invariant and its enforcement. - -## Coloring - -Every batch on the valid path has exactly one color. Dead branches are lead (permanently invalid). - -### Simplified model (three colors) - -| Color | Meaning | Terminal? | -|------------|----------------------------------------------------------------|-----------| -| **Gold** | Safe on L1 and accepted by the scheduler | Yes | -| **Silver** | Valid, optimistically executed, but not yet safe/accepted | No | -| **Lead** | Invalid (has `batches.invalidated_at_ms` set) | Yes | - -Gold batches form a contiguous prefix of the valid path. Silver batches form a contiguous suffix (after the gold prefix up to the open batch). Lead batches hang off gold nodes as dead branches -- the first lead in any cascade always has a gold parent. - -### Extended model (five colors) - -To model the full lifecycle including L1 submission: - -| Color | Meaning | Has `w_nonce`? | -|-------------|--------------------------------------------------------|----------------| -| **Tip** | Open batch, not yet closed | No | -| **Pending** | Closed, may or may not be submitted to mempool | Maybe | -| **Bronze** | Included in an L1 block, block not yet safe | Yes | -| **Silver** | Included, block has reached safe finality | Yes | -| **Gold** | Safe, accepted and executed by the scheduler | Yes | - -The spine ordering invariant: `Gold* Silver* Bronze* Pending* Tip` - -A Pending batch may have a `w_nonce` (submitted to the L1 mempool but not yet included in a block) or not (not yet submitted). The batch submitter assigns `w_nonce`s to all unsubmitted Pending batches at once, in spine-position order. - -## Nonce Poisoning - -The scheduler maintains a single counter: "I expect batch nonce N next." - -When a batch with nonce N arrives stale, the scheduler **skips it entirely** -- no nonce increment, no state change, no report. It is a true noop in nonce-space. - -This poisons the nonce counter. Every subsequent batch (nonce N+1, N+2, ...) is dead on arrival. Not because they are individually stale, but because the scheduler still expects nonce N. The only batch with nonce N was stale and skipped, so the counter will never advance past N. - -Cascade invalidation is therefore **exact, not conservative**. The sequencer's `WHERE batch_index >= stale_batch_index` mirrors precisely what the scheduler will do (refuse). The entire silver suffix is unreachable once any batch in it is stale. - -Recovery is the only way forward: create a new batch with nonce N, giving the scheduler what it needs to resume. - -## Two Staleness References - -The staleness formula is `reference_block - first_frame_safe_block >= MAX_WAIT_BLOCKS`, but the reference block differs by context: - -### Inclusion staleness (scheduler's perspective) - +# Automatic Recovery + +Automatic recovery repairs the sequencer's optimistic history after a liveness +failure. It stops issuing soft confirmations, settles outstanding submissions +when necessary, and replaces the affected suffix before resuming. It uses the +existing SQLite database and assumes the sequencer's own code and accepted +local history are correct. + +For lost or unusable local state, including after a sequencer bug, use +[cockroach recovery](cockroach.md): fix the bug, choose a trusted canonical +application checkpoint, and rebuild from L1 in a fresh data directory. Automatic +recovery does not establish trust in a corrupted application state. + +This page owns the automatic procedure and its rationale. Start with the +lifecycle and dispatch below; read the safety arguments and model boundaries +when changing recovery. The [scheduler contract](../protocol/scheduler-semantics.md) +owns canonical acceptance rules; the [invariant register](../invariants.md) owns +cross-module enforcement. + +## The state being repaired + +The local batch tree has one valid path: an **accepted prefix**, followed by an +**optimistic suffix** ending at the open **Tip**. Recovery invalidates a suffix +and opens a new Tip from the surviving path. Invalidated batch, frame, and +user-op source facts remain available for audit. + +“Accepted” (also called **Gold** in code and models) means the safe-input +projection applied the scheduler's acceptance rules and matched the landed +bytes to a valid local closed batch. It does not mean an independent canonical +machine was observed executing it. A foreign or different accepted payload +records canonical divergence and forbids automatic repair. + +Keep three identities separate: + +| Identity | Meaning during recovery | +|---|---| +| Local `batch_index` | Unique creation identity; never reused. Invalidation targets a local suffix. | +| Scheduler batch nonce | Ordering identity; derived when storage creates the batch. A replacement branch reuses the invalidated suffix's nonces. | +| L1 wallet nonce | Transaction slot; a batch transaction and a flush no-op may compete for it. Covered slots must be consumed at safe depth before post-flush repair. | + +A parentless root uses the deployment's immutable anchor nonce: zero at genesis, +or the scheduler's next nonce after cockroach recovery. Production needs no +accepted ancestor or submitted sentinel to repair a fully invalidated branch +([I16](../invariants.md#i16-the-batch-tree-has-exactly-one-valid-parentless-root-carrying-the-deployments-anchor-nonce)). + +## Lifecycle and startup dispatch + +The recovery cycle crosses a process boundary: + +1. **Detect and stop.** `DangerDetector` polls local `Storage::check_danger`; + any non-`Safe` result stops normal operation. It neither writes the database + nor calls L1. Expected-recovery and retryable exits close intake and drain + workers; diagnosed terminal runtime faults abort immediately. +2. **Respawn.** The orchestrator restarts expected recovery (`10`) and retryable + exits (`20`). Terminal exit `30` or `SIGABRT` requires investigation before a + deliberate restart. Process + ownership and shutdown belong to the [authority-boundary ADR](../plans/2026-08-authority-boundary-adr.md). +3. **Recover under the process lock, with no workers.** Check local terminal + facts before any provider call, attempt initial L1 sync, select at most one + repair, then inspect again after a repair. +4. **Prepare, admit, launch.** Prepare resources without starting tasks. A final + current inspection must still authorize serving. It creates a single-use + `RuntimeAdmission` witness, consumed synchronously by worker launch. + +Startup first refuses a persisted canonical divergence or a missing rollback-safe +checkpoint. After initial sync, one consistent `RecoveryInspection` selects: + +| Local fact | Action | +|---|---| +| `Safe` + open Tip | Ready for runtime preparation. | +| `Safe` + no Tip | `EnsureOpenTip`: create it under a transaction guard. | +| `TipInDanger(N)` | `RecoverTip { N }`: invalidate that Tip and reopen, without flushing. | +| `ClosedBatchInDanger(N)` | Flush → sync → guarded post-flush cascade. | +| `L1ViewStale` | Retry; the persisted view or clock cannot authorize serving. | +| `EstimatedBatchInDanger(N)` | Retry; an estimate alone cannot authorize invalidation. | +| `CanonicalDivergence(N)` or missing recovery checkpoint | Refuse; automatic recovery cannot repair this trust failure. | + +Only an initial-sync **provider failure** may fall back to a still-usable +persisted view. Other failures keep their typed retry/refuse classification. +Post-flush sync has no such fallback: it must establish the view required by +the cascade. + +Every repair must commit an open Tip. A fresh inspection afterward must report +`Safe` with a Tip and no terminal facts; otherwise the boot exits. Startup does +not attempt a second repair in that invocation. Final admission repeats the +same policy after preparation, because preparation can outlive the freshness +of the L1 view. A new repair requirement also exits rather than launching. + +Preparation validates the rollback checkpoint's metadata and performs snapshot +hygiene, but does **not** restore the application. After launch, the inclusion +lane loads a surviving snapshot and completes catch-up before processing new user ops. +Admission authorizes worker launch; application restoration can still fail. + +Startup logs `danger_status`, `danger_batch_index`, and `recovery_decision`, +then any invalidated indexes. Errors retain their retry/refuse classification +and diagnostic cause; the orchestrator owns restart policy and alert routing. + +## Detection and timing + +Canonical staleness and local danger use different reference blocks and thresholds: + +```text +scheduler rejects a batch with at least one frame when: + inclusion_block - first_frame.safe_block >= MAX_WAIT_BLOCKS + +sequencer observes danger when: + current_safe_block - first_frame.safe_block >= danger_threshold + danger_threshold = MAX_WAIT_BLOCKS - preemptive_margin_blocks ``` -inclusion_block - first_frame_safe_block >= MAX_WAIT_BLOCKS -``` - -Used by `populate_safe_accepted_batches` to simulate what the scheduler accepts. Each batch has its own inclusion block (the L1 block where its submission landed). **Not monotonic** across batches -- a promptly submitted old batch can be healthy while a late-submitted newer batch is stale. - -Inclusion staleness determines the **gold frontier**: the set of batches the scheduler has accepted. -### Current staleness (sequencer's detection) - -``` -current_safe_block - first_frame_safe_block >= MAX_WAIT_BLOCKS +An old batch can already have landed while fresh. Its inclusion block decides +acceptance; its age at the current safe head does not undo that acceptance. +The detector therefore examines the first unaccepted closed batch and the Tip, +not accepted history. A wire batch with zero frames is never stale and consumes +its nonce; a normal local batch with zero user ops still has a first frame and +can age. + +### Danger threshold + +The threshold means “stop and recover,” not “this batch cannot land.” The Tip +may still be canonically fresh when invalidated, and closed batches can become +accepted while the flush is running. + +The margin provides headroom before canonical expiry; it is not a grace period +after detection. Startup can repair immediately. Defaults and validation live +in [`TimingArgs`](../../sequencer/src/commands/config.rs): with `MAX_WAIT_BLOCKS` +1200 and margin 300, observed danger starts at age 900 blocks. Neither that +margin nor the fee policy bounds the time required to finish recovery. + +### When safe-head progress stops + +A responsive RPC endpoint can keep returning an old view. The detector uses +both the safe block's timestamp and local time since the last recorded safe-head +advance. [`check_danger_in`](../../sequencer/src/storage/recovery.rs) checks in +this order: + +1. Canonical divergence. +2. Missing or old safe-block timestamp → `L1ViewStale`. +3. Observed closed-batch danger, then observed Tip danger. +4. Clock regression of at least one block-time against either persisted time + baseline → `L1ViewStale`; sub-block skew is tolerated. +5. Estimated missed blocks (`elapsed / seconds_per_block`) reduce the danger + threshold. An unresolved batch crossing it gives `EstimatedBatchInDanger`. +6. Otherwise `Safe`. + +An old view blocks repair selection before observed-age checks. A regressed +clock does not suppress danger already established by observed block numbers; +a remaining clock fault still prevents admission after repair. Estimates stop +new soft confirmations but never decide which work to invalidate. + +## Repairs and their guards + +### Closed batches: flush, sync, cascade + +**Flush the covered wallet slots.** The durable wallet-nonce watermark `W` is an +upper bound on every slot this deployment may have broadcast. Every broadcaster +raises it durably **before** sending at a new nonce. A crash between those steps +may cover a slot that was never used; it must not leave a sent slot uncovered +([I14](../invariants.md#i14-watermark--wallet-nonce-of-every-tx-ever-broadcast)). + +The flusher submits zero-value self-transfers at unresolved slots from the +account's Latest nonce through `max(Pending, W + 1) - 1`. It completes only when: + +```text +Pending <= Safe && Safe >= W + 1 ``` -Used by the danger threshold detector. The reference block (`current_safe_block`) is the same for all batches. **Monotonic within the valid path** -- earlier batches have smaller `first_frame_safe_block`, so larger difference. If the frontier batch is not stale by this measure, no batch is. - -Current staleness triggers **preemptive recovery** (see below). - -## Nonce Uniqueness on the Valid Path - -`batches.nonce` can repeat across the full table -- a recovery batch inherits `parent.nonce + 1` from the last valid ancestor, which is the same nonce the first invalidated suffix batch had. Among **valid batches** (those with `invalidated_at_ms IS NULL`), nonces are unique because the valid path is a strict chain via `parent_batch_index`. - -This matters because L1 works in nonce-space (the scheduler identifies batches by nonce) while the sequencer works in index-space (local `batch_index`). The recovery path needs to translate between them: "which batch indexes should we invalidate?" Nonce uniqueness on the valid path is what makes this mapping unambiguous. - -## The L1 Stream - -L1 processes transactions in `w_nonce` order. At each slot (a given `w_nonce` value), exactly one transaction is included. If multiple transactions compete for the same slot (e.g., a dead batch and a flush no-op), L1 non-deterministically picks one. The loser is discarded. - -This is the interface between the sequencer and the scheduler. The scheduler sees a stream of entries ordered by `w_nonce`, each with a `batch_nonce`, `inclusion_block`, and `safe_block`. It processes them in order, accepting or rejecting based on nonce match and staleness. - -## The Uncertainty Interval - -The core insight behind the recovery design is that **mempool uncertainty is bounded by a time interval**. - -Once a batch's `safe_block` is old enough that `current_safe_block - safe_block >= MAX_WAIT_BLOCKS`, we know it is stale no matter when it lands on L1 (because `inclusion_block >= current_safe_block`). Any batch in the mempool with that `safe_block` is dead-on-arrival. This means mempool uncertainty has a natural expiration: after `MAX_WAIT_BLOCKS`, the L1 outcome doesn't matter. - -This gives us three regimes: - -``` -|---------- safe ----------|-- danger zone --|-- past MAX_WAIT --| - no action flush + recover self-resolved -``` - -- **Before the danger zone**: batches are young. Nothing to do. -- **In the danger zone**: batches might land stale, or might still make it. This is the window of uncertainty. For **closed unresolved batches**, the flush resolves it by forcing every `w_nonce` slot to finalize (batch wins or no-op wins). After the flush, the sequencer reads the scheduler's finalized state and cascades if needed. An **open Tip** has no `w_nonce` slot yet, so it is not part of this uncertainty set. -- **Past MAX_WAIT**: all unresolved batches are guaranteed stale by L1 monotonicity (`inclusion_block >= current_safe_block >= safe_block + MAX_WAIT`). For closed unresolved batches, the L1 outcome no longer matters because every eventual inclusion is stale, but wallet-nonce slots may still need to be flushed (or naturally consumed) before recovery can reconstruct the scheduler frontier. For an aging open Tip, there is no L1-slot uncertainty at all, so startup recovery can invalidate it directly. - -**What TLA+ proves vs external reasoning**: the TLA+ model ([`preemptive.tla`](preemptive.tla)) proves that after all `w_nonce` slots are resolved (however that happens), ZombieSafety holds. It does not model the danger threshold or the passage of time. The claim that "past MAX_WAIT, staleness self-resolves" is an external argument from L1 monotonicity (`inclusion_block >= current_safe_block`), not something TLA+ checks. - -Any recovery design must wait out this uncertainty. The question is how. The preemptive design (implemented here) forces resolution by going offline and flushing. An alternative optimistic design lets the uncertainty resolve naturally but keeps serving soft confirmations -- see [`history/`](history/) for that approach and why we preferred preemptive. - -## Silver-Only for Submitted Batches - -The Silver-only constraint applies to **submitted batches whose L1 slot outcome is still relevant**. This is the zombie path, and it is where the optimistic-design counterexample from [`history/`](history/) still matters. - -A Silver batch's L1 entry is permanent -- no mempool competition can kill it. The scheduler **will** see it, at a `w_nonce` lower than any recovery batch, and be poisoned. This ordering guarantee is what makes nonce poisoning reliable. - -Detecting staleness on Pending or Bronze submitted batches *before wallet-nonce uncertainty is resolved* is unsafe: a recovery batch can take the frontier's L1 slot via wallet-nonce mutual exclusion, preventing the scheduler from ever seeing the stale frontier, and allowing non-frontier dead batches to pass the nonce check. TLA+ model checking found this bug; see [`history/`](history/) for the counterexample. - -The open Tip is different. It has no L1 transaction yet, so there is no `w_nonce` competition and no zombie risk. Once `current_safe_block - first_frame_safe_block >= danger_threshold`, startup recovery can invalidate the aging Tip directly and open a fresh one. Likewise, after a preemptive flush has resolved all competing `w_nonce` slots for closed batches, the atomic recovery transaction can safely use **current staleness** on the oldest unresolved batch (closed or open). - -## Preemptive Recovery Design - -The sequencer uses a preemptive approach: detect danger early, go offline, flush the mempool, then recover on solid ground. This design was preferred over the optimistic alternative because it is simpler to reason about and produces fewer invalidated soft confirmations (the sequencer stops issuing them before the cascade). - -### Step 1: Danger threshold - -Define `DANGER_THRESHOLD = MAX_WAIT_BLOCKS - MARGIN`. When the frontier batch's current staleness (`current_safe_block - safe_block`) reaches `DANGER_THRESHOLD`, **trigger preemptive recovery**. - -The threshold is *only* a trigger. It says "stop running, hand off to recovery." It does **not** say "this batch is doomed." The cascade decision belongs to step 5, which examines the post-flush state and acts on what's actually there. - -#### Why a margin at all (Sorites argument) - -The right value of `MARGIN` is not derived from the recovery procedure's runtime — it falls out of a sharper question: **at what age do we give up on the current batches and start anew?** - -Two endpoints are clear: - -- A batch that's 1 minute behind shouldn't be invalidated. The infra hiccup might pass; pre-confirmations issued against it will likely still land. -- A batch that's 1 minute *before* `MAX_WAIT_BLOCKS` shouldn't be left to die. We've already tried for hours. The last minute won't save us, and pre-confirmations issued in this window are knowingly dishonest — we have strong evidence they won't land. - -Somewhere between those, we want to switch from "keep waiting" to "give up." The exact crossover is a Sorites question with no canonical answer, but two design pressures pin it: - -1. **Stop issuing pre-confirmations on state we reasonably know won't land.** As current staleness approaches `MAX_WAIT_BLOCKS`, the probability that the current batch lands gracefully drops. Pre-confs issued past that point are increasingly dishonest to users. -2. **Give the operator runway to fix infra.** If L1 is misbehaving, network is degraded, mempool is congested — the operator needs hours, not minutes, to diagnose and act before the system commits to recovery and invalidates work. - -The recovery procedure's own runtime (flush submission + L1 safe finality wait of ~13 min on Ethereum + atomic SQLite cascade) is a *floor* on `MARGIN`, not the deciding factor. It must fit, but fitting it is far from the operating point. - -#### Defaults - -With `MAX_WAIT_BLOCKS = 1200` (~4 hours), the default `MARGIN = 300` blocks (~1 hour at 12s/block) gives the operator ~1 hour after danger-zone entry before the system commits to recovery. That's well above the procedure-runtime floor (~15 min) and meaningful runway under the second design pressure. - -Production tunings with a longer `MAX_WAIT_BLOCKS` (e.g. 24h) should keep the margin in the hours range — there's no benefit to a tighter margin once `MARGIN` exceeds the procedure-runtime floor several times over. - -### Step 2: Go offline - -Stop accepting new user operations. From the outside world, the sequencer is temporarily unavailable. This eliminates concurrent batch creation during recovery. - -### Step 3: Flush mempool - -Read the persisted **wallet-nonce watermark** `W` — the highest `w_nonce` this deployment ever broadcast (`wallet_nonce_watermark` singleton; see Implementation Constraint 1). Query the latest confirmed `w_nonce` (N) and the pending `w_nonce` (M). Submit no-op transactions (self-transfers of 0 ETH) at nonces N, N+1, ..., `max(M, W+1) - 1`. These compete with any of our transactions still alive anywhere in the network — including zombies the local node's pool has forgotten. - -Wait until both `pending <= safe` **and** `safe >= W + 1`: every slot this deployment ever used is consumed at safe depth. The second conjunct is the durable anchor — without it the flush trusts the local node's volatile mempool memory, which a dropped-locally-but-alive-elsewhere zombie evades entirely. The flush reports the safe block at which it observed resolution; Step 5 refuses to cascade until the re-synced view reaches at least that block. - -### Step 4: Post-flush state - -Every `w_nonce` slot from N to M-1 is now resolved: - -- **Batch won**: the batch is on L1 and safe (Silver or Gold) -- **No-op won**: the batch is dead forever, its slot consumed - -There are no more mempool entries. All uncertainty is resolved. - -**Flush safety does not depend on eviction; completion depends on L1 progress.** -A rejected no-op surfaces as a hard `FlushError` and the process exits. The -orchestrator respawn re-runs the flush. Inclusion of either the original batch -or a no-op can resolve the slot, but the sequencer remains offline until every -covered slot reaches safe depth. Neither retries nor the danger threshold -establish a recovery deadline. - -No-ops use 3× the fresh fee estimate, followed by a symmetric replacement bump. -This headroom improves their chance of replacing an earlier transaction; it -does not guarantee replacement. Base fees and priority estimates can move in -opposite directions. For example, a poster tx sent at base 10 gwei with cap -22 gwei and tip 2 gwei cannot mine at base 30 gwei. If the current tip estimate -is 0.5 gwei, the no-op offers cap 199.65 gwei and tip 1.65 gwei (plus 1 wei on -each). Geth rejects that replacement because its tip misses the 2.2 gwei -threshold. Both the original and the no-op can therefore fail to make progress. -A previous flush no-op can also block another pass on a flat market. These are -accepted liveness limits of the current [fee policy](../l1-fee-policy.md), not -permission to cascade before the slots resolve. - -### Step 5: Run recovery - -This is an atomic SQLite transaction operating on the best available L1 state. The storage work splits cleanly by whether a flush ran first. - -#### Mental model: "everything past gold is doomed" - -After the flush has resolved every wallet-nonce slot, and `populate_safe_accepted_batches` has been re-synced, the gold spine is at its **maximum extent**: the simulation walked safe-inputs in inclusion order, accepting each one until it hit a barrier (a stale batch, or a missing batch where a no-op consumed the slot). - -Any batch past that gold frontier is **doomed**, in one of three concrete senses: - -| State | What happened | Why doomed | -|---|---|---| -| **Silver-stale** | Original tx landed, scheduler skipped (`inclusion_block - first_frame ≥ MAX_WAIT`) | Scheduler's expected nonce never advances past it; downstream batches are nonce-poisoned | -| **Silver-fresh poisoned** | Original tx landed fresh, but a preceding stale or missing batch poisoned the nonce | Scheduler skipped on nonce mismatch; on-chain row can't be retroactively re-evaluated | -| **Pending (no-op'd)** | Flush no-op consumed the wallet-nonce slot; original tx never landed | The L1 transaction is dead. Re-submission at a fresh slot would land *after* the existing on-chain Silver-poisoned batches; the scheduler sees those at lower `safe_input_index`, advances expected past them on the resub generation, but the per-original-tx work is gone | - -**Why isn't this just "stale"?** Under self-trust (we don't defend against malformed self-submissions), the *first* non-gold closed batch can only be Silver-stale or Pending. Nonce-mismatch is impossible at the frontier — nonces are contiguous on the valid path (`trg_enforce_nonce_contiguity`). But *downstream* batches past that first non-gold are typically Silver-fresh-poisoned: their inclusion-staleness was fine, but they were processed when expected was stuck at the poisoned nonce. - -A **fourth shape** sits outside this taxonomy: a closed batch that was **never submitted** (closed after the submitter's last tick before the detector exit). It has no L1 footprint, no killed tx, and is not literally doomed — it could simply be submitted after recovery. The cascade invalidates it anyway: once committed to recovery, cascading the entire non-gold suffix converges in one cycle and avoids spine-order reasoning about a half-submitted suffix. The cost is real (its soft confirmations are rolled back); this is a deliberate convergence-over-preservation policy choice. - -Cascading from the first non-gold catches all four. **No per-batch age check is needed for the cascade pivot itself** — every closed batch past gold is either doomed by construction or sacrificed by the convergence policy. - -#### Path A — guarded post-flush Cascade - -After step 3 (flush) and step 4 (re-sync), the gold frontier is fresh. Run the atomic recovery transaction: - -1. **Find the cascade pivot.** First try the closed pivot: first valid closed batch with `nonce >= frontier_nonce`. By the contiguity invariant, this batch's nonce is exactly `frontier_nonce`. If one exists, cascade from it. -2. **No closed pivot? Check the Tip.** When all closed batches landed fresh and were accepted (the "everything worked" aftermath), there's no closed pivot — but the Tip can still be in the danger zone. When the lane rotates without a safe-block advance between frames (e.g. immediately after init, both frames share the bootstrap `safe_block`), `S_tip = S_closed`. The closed batch can become gold by inclusion-staleness while the Tip's age — measured against `current_safe_block` after the flush wait — has crossed the danger zone. Pure monotonicity (`S_tip ≥ S_closed`) doesn't rule this out: equality is allowed. So fall through to `find_tip_batch_in_danger(danger_threshold)`. If the Tip's age clears `danger_threshold`, cascade it. -3. **Cascade-invalidate the suffix**: set `invalidated_at_ms` on every valid batch with `batch_index >= pivot.batch_index`. This catches all non-gold batches in cases (2)/(3) above, and the Tip alone in the no-pivot-but-Tip-aging case. The invalidation trigger deletes those batches' canonical `application_inputs` rows, rewinding head `H` to the surviving prefix. Raw L1, batch, frame, and user-op source facts remain available for audit. -4. **Advance external history reality**: iff step 3 invalidated at least one valid batch, increment `RecoveryGeneration` exactly once in this same SQLite transaction. A no-invalidation repair does not bump it. Application-history rewind and generation change are therefore one visible transition. -5. **Open recovery batch**: parent is the last valid ancestor (`MAX(batch_index) FROM valid_batches` after the cascade). Nonce is structurally `parent.nonce + 1`, which equals `frontier_nonce` — the scheduler's `expected_nonce`. Reconcile external directs after the latest surviving frame's `safe_block`, or immutable baseline block `C` if no frame survives. The new application rows reuse offsets beginning at the rewound head under the incremented generation. - -**Threshold = `danger_threshold`, not `MAX_WAIT_BLOCKS`**. We're already committed to recovery; the Tip is past gold; if it's also past the threshold that would have triggered recovery had it been a closed batch, cascade it. Otherwise the next danger detector tick after resume would re-trip on the Tip's eventual close + submission anyway (the closed batch would inherit its first frame's safe_block). - -#### Path B — guarded `RecoverTip` - -The `RecoverTip` action is dispatched when `check_danger` returns `TipInDanger(idx)`: no closed batch is past the gold frontier in the danger zone, but the open Tip's first frame has aged past `danger_threshold`. **No flush ran** — the Tip has no L1 footprint, so there's nothing to flush. - -Closed batches past gold (if any) are still in their natural lifecycle — pending in the mempool, recently included, awaiting safe finality. Cascading them would prematurely abort their progression. We act only on the Tip: - -1. Run `find_tip_batch_in_danger(danger_threshold)`. If `Some(tip_index)`, cascade-invalidate from there (which only touches the Tip — no closed batches have `batch_index >= tip_index`) and increment `RecoveryGeneration` exactly once in that same transaction. -2. Open a fresh recovery batch in the same transaction. -3. If no Tip in danger and no Tip exists at all (torn-state crash recovery), open a Tip anyway. - -The `Safe` decision with no open Tip runs `EnsureOpenTip`. Its transaction rechecks `Safe`, rollback-safe checkpoint presence, and Tip absence, then opens the Tip through `open_fresh_tip_in_tx`. It refuses rather than commit without an open Tip. Startup rechecks danger after this repair; Tip creation never occurs as a worker-construction side effect. - -#### Why `danger_threshold`, not `MAX_WAIT_BLOCKS`, for the Tip threshold - -The Tip threshold is a **policy choice**, not a mathematical staleness bound. A Tip whose first frame is at age `danger_threshold` could in principle still close, submit, and land fresh by inclusion-staleness — `inclusion_block - first_frame` would be roughly `danger_threshold + (rotation + submit latency)`, which (with a reasonable margin) is still below `MAX_WAIT_BLOCKS`. - -We invalidate at `danger_threshold` because: - -1. **Pre-confirmation honesty.** Once the Tip's age crosses the danger zone, the system has decided this generation is operationally suspect. Continuing to issue soft confirmations against it is dishonest to users. -2. **Avoid retrip risk.** The runtime danger detector also fires on `DangerStatus::TipInDanger`. Without invalidating at startup, we'd resume operation, the detector would re-trip on the next tick, and we'd cycle. Cascading at startup converges in one cycle. -3. **Symmetry with the closed-batch trigger.** The closed-batch detector trips at `danger_threshold`. Using the same threshold for the Tip preserves the framing: "danger zone = committed to recovery." - -### Step 6: Resume - -Restart the batch submitter and user-op acceptance. If this recovery invalidated -any valid batch, the generation bump already committed atomically with that -invalidation; otherwise the history version is unchanged. The sequencer is -back online. - -### Why post-flush cascade is unconditional (and not threshold-based) - -An earlier design considered using `MAX_WAIT_BLOCKS` as the cascade threshold even in the post-flush path: only invalidate the frontier if its `current_safe_block - first_frame.safe_block ≥ MAX_WAIT`. The intuition was to preserve soft confirmations when re-submission could still land fresh. - -**This doesn't hold up.** Walk through the boundary case: - -1. Frontier batch has `current_staleness ∈ [danger_threshold, MAX_WAIT)`. Detector trips, flush runs. -2. `recover_post_flush` (with hypothetical threshold) sees age below MAX_WAIT, declines to cascade. Resume. -3. Submitter wakes up, resubmits the Pending frontier (and any non-gold closed batches) at fresh wallet-nonce slots. They enter the mempool. -4. Detector polls again. Frontier age has barely moved or the published safe - head is unchanged; providers may later expose several newly-safe blocks as - one jump, but no cadence assumption makes the frontier clean again. It is - still above `danger_threshold`, so the detector trips again. -5. Recovery 2 starts. Flush submits no-ops at the slots the submitter just used for resubs. Bumped fees on no-ops typically out-bid resubs. Resubs killed. -6. Goto step 2. Loop converges only when `current_staleness` finally crosses `MAX_WAIT_BLOCKS` and the threshold check fires. - -Each loop iteration burns gas (no-ops + doomed resubs), takes ~12 minutes (the flush's safe-finality wait), and the soft confirmations are rolled back at the end anyway. Cascading on first non-gold converges in **one cycle** with predictable cost. - -### Startup behavior summary - -Startup holds the exclusive process lock and launches no workers until recovery and preparation finish. Its first local inspection refuses canonical divergence or a missing rollback-safe checkpoint before any provider call. It then attempts one initial Sync: a provider failure may use a still-fresh persisted view, while other failures retain their typed retry/refuse classification. - -After that attempt, `select_recovery` maps one consistent local inspection as follows: - -| Local fact | Action | Why | -|---|---|---| -| `Safe` + open Tip | Ready for preparation | The local prediction is clean and structurally resumable. | -| `Safe` + no Tip | `EnsureOpenTip` | Open the Tip under its transaction guard, then recheck. | -| `L1ViewStale` | Retry | The persisted view cannot authorize new soft confirmations. | -| `TipInDanger(N)` | `RecoverTip { N }` | The Tip has no L1 footprint; invalidate and reopen directly. | -| `ClosedBatchInDanger(N)` | Flush → Sync → Cascade | Resolve the closed batches' L1 slots before changing their local suffix. | -| `EstimatedBatchInDanger(N)` | Retry | Recovery never mutates from an estimate alone. | -| `CanonicalDivergence(N)` | Refuse | Standard recovery assumes content identity and is forbidden. | - -Closed recovery retains the flush's observed safe block in a local variable. Post-flush Sync must succeed; its provider failure cannot use the initial-sync fallback. The guarded cascade transaction refuses divergence or a missing rollback-safe checkpoint, requires the persisted safe head to reach the flush observation, and then applies the post-flush policy. It runs even if the refreshed danger verdict is `Safe`: a young unresolved suffix is still doomed after flushing. A crash or retry loses the observation, so another invocation must flush again. - -Flush changes only the wallet watermark locally. New divergence can be discovered only by Sync, and the next dispatch or guarded cascade checks it before repair. There is no additional inspection between Flush and Sync. The process lock and task-free startup exclude a competing local writer; revisit this sequencing if startup gains concurrent writers. - -Every repair is followed by a current inspection. A surviving view/clock refusal retries the boot; successful mutation alone does not authorize serving. The guarded Tip operations also recheck their policy at mutation time, because wall-clock aging can change a verdict without a database writer. A repair must commit an open Tip, and startup never starts a second repair in the same invocation. - -After a clean result, runtime preparation launches zero tasks. `admit_runtime` then applies the same dispatch to current facts. Only `Ready` mints `RuntimeAdmission`; any repair requirement or refusal drops the prepared resources and exits. Launch consumes the witness synchronously. This final check is necessary because preparation can outlive the freshness of the persisted L1 view. - -`preemptive.tla` covers slot/batch safety. `admission.tla` covers local terminal dominance, flush/sync prerequisites, loss of observations across retries/crashes, repair postconditions, and final admission soundness. The “everything past gold is doomed” argument remains external to both bounded models. - -### Startup observability - -Startup logs its selected action with `danger_status`, `danger_batch_index`, and `recovery_decision`, and records the invalidated batch indexes after repair. Errors retain the classified retry/refuse verdict and their diagnostic cause. The orchestrator owns restart policy and alert routing. - -### L1 view freshness - -The safety policy does not branch directly on a provider-reachability boolean. Reachability is an execution concern: the initial Sync may fail while a warm persisted view remains usable, whereas a post-flush Sync failure must retry because Cascade requires a newly caught-up view. The decision primitive is the freshness of the L1 view recorded in SQLite plus the post-flush witness floor when one exists. - -The most common real-world trigger for `L1ViewStale` is a stalled RPC gateway: the provider answers, but its safe-head response stops advancing (a degraded upstream node, a load-balancer routing to a lagging replica, or a temporary indexing pause). The sequencer can't distinguish "fresh answer from a stalled view" from "L1 itself is unhealthy" without a second source of truth, so it treats both the same way: refuse to commit to soft confirmations until the recorded safe block is fresh again. - -**At startup**: the sequencer first inspects local terminal facts, then attempts the initial safe-head Sync, then inspects the persisted safe-block and progress timestamps. If the L1 timestamp is missing or older than `l1_read_stale_after_blocks * seconds_per_block`, `check_danger` returns `L1ViewStale` and startup retries. A baseline a full block-time or more ahead of `now` also yields `L1ViewStale`, but only after observed-safe checks have run. If those checks selected a repair, the repair completes and its mandatory next inspection applies the clock refusal. If the view is usable and fresh, observed-safe checks can route to recovery, and the batch-relative wall-clock estimate remains the final retry guard. - -**At runtime**: the `DangerDetector` polls `Storage::check_danger` on its cadence. The input reader records both the observed safe block timestamp and the local time at which the safe head last advanced. If safe-head observations stop advancing, either the global safe block timestamp crosses the read-staleness threshold (`L1ViewStale`) or a specific unresolved batch crosses the batch-relative adjusted threshold (`EstimatedBatchInDanger`). A backward clock step of a full block-time or more against either persisted baseline also produces `L1ViewStale` — evaluated after the observed arms — and saturation must never reinterpret such a regression as zero elapsed time; sub-block steps are quantization noise for the block-granular estimate and are tolerated. The detector then exits with `RecoveryRequired`, the orchestrator respawns, and startup re-runs the same check. The batch submitter never observes danger; this responsibility lives entirely with the detector. - -**Other workers during L1 outages**: the inclusion lane and API are purely local (SQLite) and continue operating. The input reader retries L1 polling with error logging. All L1-dependent workers log errors at the `error` level to alert operators. - -The `seconds_per_block` parameter (default: 12 for Ethereum) is configurable via `CARTESI_SEQUENCER_SECONDS_PER_BLOCK`. The L1 read-staleness threshold is configurable via `CARTESI_SEQUENCER_L1_READ_STALE_AFTER_BLOCKS`; its fixed default is independent of the margin and must remain strictly below the danger threshold (defaults and validation live in `sequencer/src/commands/config.rs`). These estimates are conservative — they may cause earlier detection if blocks are slower than assumed. This is correct: better to crash early than to issue doomed soft confirmations. - -## Dead Batches - -After cascade invalidation, submitted Pending batches (those with `w_nonce` assigned) are **dead batches**. They are still in the L1 mempool, competing with their flush no-op transactions. - -Two outcomes per dead batch, non-deterministic: - -- **Dead batch beats no-op**: lands on L1, scheduler sees it, rejects it (stale by inclusion, or nonce-poisoned by a preceding stale/missing batch) -- **No-op beats dead batch**: dead batch killed forever, scheduler never sees it (the scheduler skips the gap) - -A killed batch acts as **silent nonce poison**: the scheduler never sees it, so `schedulerExpected` stays stuck at its `batch_nonce`. All subsequent batches have wrong nonces. - -Dead batches occupy `w_nonce` slots strictly below `walletNonce`. Recovery batches occupy `w_nonce` slots at or above `walletNonce`. **No overlap.** This is why no mutual exclusion is needed between dead batches and recovery batches -- they live in non-overlapping `w_nonce` ranges. - -## Cockroach recovery (`setup --recovery`) - -Use [cockroach recovery](cockroach.md) when the database is lost or local state -is unusable, including after a sequencer bug. Fix the bug first, then rebuild in -a fresh data directory from a trusted, sufficiently advanced canonical -application checkpoint and L1. The recovery guide owns checkpoint requirements, -the replay procedure, and the resulting resume baseline. - -Plain `setup` also directs the operator to this path when it detects a previous -instance's batches past the checkpoint: it refuses with exit `40` -(`EXIT_SETUP_NEEDS_RECOVERY`). - -## Canonical divergence (terminal, outranks every arm) - -Independent of the staleness machinery, the input reader's acceptance -simulation cross-checks every at/above-anchor **accepted** landing against the -local valid closed batch at that nonce (the content-identity check: -`keccak256` of the landed wire bytes vs the hash stamped at seal). A `Foreign` -(no local batch) or `Mismatch` (different bytes) outcome persists the -`canonical_divergence` marker in the same transaction as the sync that found -it. The freeze, its runtime reaction, the race bound, and the watchdog -boundary are owned by -[I15](../invariants.md#i15-divergence-marker-present--acceptance-frontier-frozen); -the check's completeness scope by [I9](../invariants.md). - -This page owns the recovery side. `check_danger` reports -`CanonicalDivergence` **ahead of every other arm**, so a respawn loop can never -route a known-diverged node into a provider call, batch-tree mutation, or admission. -Startup checks before its initial Sync, after that Sync, inside the post-flush -cascade transaction, and before admission. The guarded repair transactions -reassert the marker's absence and map it to terminal `Refuse`. - -The remedy is **cockroach recovery (wipe + rebuild from L1), never the -standard recovery on this page**: the cascade reconciles the batch tree's -*shape* under the assumption that accepted nonce N is our batch N — a content -mismatch means canonical state contains executed effects with no reliable -local source, so rebuild-from-L1 is the only honest repair. - -### Restore points and admission - -Every admitted database retains a rollback-safe application artifact: the -baseline before any local batch is accepted, or an accepted batch snapshot. -Startup may load a newer surviving optimistic snapshot to reduce replay, but -that snapshot alone cannot justify admission because recovery may remove it. -Once an accepted artifact exists, GC may retire baseline bytes while preserving -immutable baseline metadata and active leases. `admission.tla` calls this -`hasRecoveryCheckpoint`; artifact creation and GC remain outside that model. - -## Implementation Constraints - -These constraints were discovered during TLA+ model checking and are required for correctness: - -1. **`walletNonce` must NOT be reset during recovery.** Recovery batches must use `w_nonces` strictly past all dead batch slots. The flush consumes dead batch slots by advancing `nextL1Slot` up to `walletNonce`. Recovery starts fresh from there. - **Mechanism:** `walletNonce` is realized durably as the `wallet_nonce_watermark` singleton — the highest wallet nonce ever broadcast. Every broadcaster (the batch poster and the flusher's no-ops alike) commits `watermark = max(watermark, n)` power-loss-durably (`synchronous=FULL`) **before** sending at nonce `n` (write-before-broadcast; a crash between commit and send only over-covers — one wasted no-op). The flush's completion condition is `pending <= safe && safe >= watermark + 1`, so it cannot declare victory while any slot we ever used is unresolved — restoring this constraint against the local pool's volatile memory. The watermark is never reset and never lowered. - -2. **`SubmitBatch` must use `max(walletNonce, nextL1Slot)`.** Prevents assigning `w_nonce` values for slots L1 has already consumed. - -3. **`SubmitBatch` must assign ALL pending batches at once, in spine-position order.** If batches are submitted individually, a flush-win can bump one batch's `w_nonce` past a later batch's, violating the spine ordering invariant. - -4. **Wall-clock freshness when the L1 view stops advancing.** The input reader records the L1 safe block timestamp and the local last-safe-head-progress time. `Storage::check_danger` first refuses on an old or missing safe block timestamp; a clock a full block-time or more out of step with either persisted baseline also refuses, but only after the observed-safe checks (sub-block skew is tolerated as quantization noise). Only a usable clock reaches the unresolved-batch estimate (`elapsed / seconds_per_block`). Without these checks, an L1 outage or a large backward clock step can silently push batches past the danger zone while the DB-based safe-block number remains frozen. - -5. **The accepted-frontier cache persists acceptances, not scan progress.** `safe_accepted_batches` stores the scheduler-accepted prefix and resumes from the latest accepted safe input. Rejected batch-submitter inputs after that frontier can be rescanned on later safe-head syncs until a later batch is accepted. This is a performance tradeoff, not a correctness bug: recovery batches can reuse a scheduler nonce after earlier rejected rows, so a separate persistent scan cursor would need careful nonce-reuse tests before being introduced. +Here Latest, Pending, and Safe are account transaction counts, not block +numbers; absent `W` contributes a lower bound of zero. An original batch or a +no-op may win each slot. Completion means every covered slot is consumed at +safe depth, even if the local node forgot an original transaction. It does not +require erasing that transaction's bytes from every mempool. + +The flusher returns the **safe block number** at which it observed completion. +Startup keeps it only in the current call; a crash or retry loses that +observation and the next attempt flushes again. Flush changes no local recovery +facts except the wallet watermark. + +**Sync through that observation.** Sync ingests safe InputBox events and updates +the local scheduler-acceptance projection. It does not query a canonical +application machine. A provider failure here retries the boot; there is no +fallback to the pre-flush view. + +**Cascade under an immediate SQLite transaction.** Its guard requires no +canonical divergence, a rollback-safe checkpoint, and a persisted safe head at +least as high as the flush observation. Then choose the pivot: + +- First valid closed batch beyond the accepted frontier, regardless of age. +- If none remains, the Tip only if it has reached `danger_threshold`. +- Otherwise invalidate nothing, retaining a fresh Tip or opening a missing one. + +The cascade deliberately runs even if refreshed danger is now `Safe`. The +closed-suffix policy follows from having completed flush and sync; it does not +repeat the trigger test. There is no extra inspection between flush and sync: +the process lock and absence of workers exclude competing local writers, and +sync is the step that can discover new divergence. Revisit this sequence if +startup gains concurrent writers. + +Flush completion depends on L1 progress. Replacement attempts can be rejected +or remain uncompetitive, and provider failures can interrupt the attempt. +Retries preserve safety but establish no recovery deadline. Pricing and its +accepted liveness limits belong to the [L1 fee policy](../l1-fee-policy.md). + +### Open Tip: repair without flushing + +An open Tip has never been submitted, so invalidating it creates no L1-slot +race. `RecoverTip { N }` rechecks divergence, checkpoint availability, and +**exactly** `TipInDanger(N)` inside its transaction, then invalidates that Tip +and opens a fresh one. It does not invalidate closed batches or fall back to +repairing a changed decision. + +The threshold is a policy choice: retaining an aging Tip would make the +runtime detector stop service again. Waiting for `MAX_WAIT_BLOCKS` would keep +the same suspected prediction alive without solving that cycle. + +`EnsureOpenTip` is a separate action. Its transaction requires `Safe`, no Tip, +no divergence, and a rollback-safe checkpoint. It opens the Tip without +invalidating history. Guarded writes matter even without concurrent writers: +wall-clock aging alone can change the decision after inspection. + +### Atomic history change and replay + +Invalidation, history rewind, generation change, and Tip creation share one +transaction. Invalidation removes the suffix's `application_inputs` projection; +raw source facts remain. `RecoveryGeneration` increments once iff at least one +valid batch was invalidated. Failed reopening rolls all of this back. + +The new Tip follows the latest surviving batch, or uses the immutable root +anchor if none survives. It attributes direct inputs after the surviving +frame's drain boundary, with the era baseline block as a floor. Storage records +those application entries; the launched lane executes them during catch-up. +This prevents a restored prefix's directs from being executed twice. + +Recovery requires the latest accepted batch snapshot, or the era baseline +before any local batch is accepted. A newer surviving optimistic snapshot can +reduce replay, but cannot replace that rollback guarantee. Snapshot publication, +artifact validation, leases, and GC are owned by the +[snapshot lifecycle](../snapshots/lifecycle.md); history coordinates and +subscription behavior by the [API contract](../../README.md). + +## Why the closed-suffix policy is safe + +**Settling slots removes the zombie race.** Before the flush, an old batch may +still win an L1 slot after local invalidation. Reusing its scheduler nonce too +early can let later old batches execute against the replacement branch. The +[historical counterexample](history/README.md) demonstrates this failure. +Detecting danger before settlement is necessary; mutating the closed suffix +before settlement is the unsafe step. + +After completion, the original transactions cannot newly win those consumed +slots on descendants of the observed safe chain. Sync through the observation +accounts for the originals that did win. Replacements use later wallet slots +and reuse only the scheduler nonces beyond the accepted prefix. This relies on +the trusted, consistent L1 view and dedicated submitter key in the +[threat model](../threat-model/README.md), and on every broadcaster preserving +the watermark. + +**A skipped batch does not advance the scheduler nonce.** When nonce `N` arrives +stale, its frames are skipped and later `N+1`, `N+2`, … envelopes encounter a +nonce mismatch. The overdue-direct backstop still runs before envelope +classification, so “skipped batch” does not mean the whole input has no state +effect. A missing batch whose slot was consumed by a no-op also leaves the +expected nonce unchanged. Later input cannot retroactively make already +rejected envelopes execute. + +**Discarding the entire remaining closed suffix is a convergence policy.** It +can include rejected landings, no-op-replaced transactions, and batches never +submitted at all. Some of that work could theoretically be resubmitted fresh; +“everything past Gold is doomed” is not a general impossibility proof. Recovery +sacrifices it to avoid preserving a partly submitted suffix and restarting into +the same danger/flush cycle. The cost is invalidated soft confirmations. + +If every closed batch became accepted, an aging Tip can still need repair: +its first frame may share the preceding batch's safe block, while its age is +measured at the later post-flush head. A fresh Tip survives without a generation +change. Thus flush alone does not imply invalidation. + +**Content identity is a prerequisite.** The input reader checks at/above-anchor +accepted wire bytes against the local valid closed batch. A foreign or different +payload persists divergence and freezes the acceptance frontier. Startup checks +that marker before L1 access, after sync, inside repair transactions, and before +admission. Repairing the tree's shape cannot recover missing canonical effects; +investigate the fault and use [cockroach recovery](cockroach.md). Check scope +and enforcement are owned by [I9 and I15](../invariants.md). ## Formal Verification -The recovery design is verified with two complementary bounded TLA+ models. [`preemptive.tla`](preemptive.tla) owns slot/batch safety; [`admission.tla`](admission.tla) owns startup reduction and runtime admission. An alternative optimistic batch design is preserved in [`history/optimistic.tla`](history/optimistic.tla). - -**Scope and limitations**: these are bounded safety models. They exhaustively check all reachable states within the configured bounds but do not prove liveness or model concrete timing margins. The admission model includes abstract owner loss/crash with fresh-attempt restart (there is no admission state machine to model); the slot model does not model crash/restart and relies on SQLite atomicity for its implementation mapping. +Two bounded TLA+ models check complementary safety obligations. They are not +a refinement proof of the Rust implementation, a proof of their composition, +or a liveness guarantee. Read both before changing recovery code. -### `preemptive.tla` -- Slot-level safety under adversarial flush +### `preemptive.tla`: batches and wallet slots -Models the core slot-level mechanics of preemptive recovery. At every `w_nonce` slot, L1 non-deterministically includes the spine batch OR a flush no-op (killing the batch). This covers the case where the frontier batch itself is killed during flush. The model also treats the open Tip's `safe_block` as meaningful, so it can explicitly recover an aging Tip that has no L1 footprint yet. +[`preemptive.tla`](preemptive.tla) models safe-block advancement, wallet-slot +competition between batches and no-ops, scheduler acceptance, and branch +invalidation. Its `Inv` checks `ZombieSafety` at every reachable state: +`schedulerExpected = CountGold(spine)`. It also checks batch-nonce contiguity, +invalid-branch ancestry, wallet-slot uniqueness, and L1/scheduler cursor bounds. -The model is a **safety over-approximation for the actions it shares with the implementation**: it allows `AdvanceTip` and `SubmitBatch` to interleave freely with recovery, which the real protocol prevents (the sequencer goes offline). This makes the proof stronger -- if `ZombieSafety` holds under more interleavings, it holds under fewer. However, the over-approximation claim does **not** hold action-for-action — two implementation actions sit *outside* the model's transition set: (1) the model discards an aging Tip only at `MAX_WAIT_BLOCKS`, while the implementation invalidates at `danger_threshold` (= `MAX_WAIT − MARGIN`); (2) the model's `Resolve` has no case for a killed-Pending frontier (it relies on resubmission until the frontier is Silver), while guarded post-flush Cascade invalidates killed Pendings unconditionally. Their safety rests on the external arguments above. Sequential startup ordering is intentionally delegated to `admission.tla` rather than cross-producting this already-large slot model. +Several details must not be read as literal production behavior: -**Verified**: 157M states, 0 violations. +| Model | Production mapping or limit | +|---|---| +| A Gold genesis sentinel at nonce zero | Production opens a parentless root at its stored anchor, without a submitted sentinel. Root/anchor cases are tested in Rust. | +| `SubmitBatch` assigns the pending suffix with `max(walletNonce, nextL1Slot)` | The poster derives the suffix and Latest account nonce, and raises the durable watermark before sending. The model expression is not a Rust nonce-allocation recipe. | +| Tip advancement and submission can interleave with recovery; dead batches can race after model invalidation | Production stops workers and settles covered slots before the closed cascade. These additional modeled interleavings do not establish coverage of different production actions. | +| `Resolve` handles a stale Silver frontier or a Tip at `MAX_WAIT_BLOCKS` | Production also invalidates a killed/unsubmitted closed suffix after flush and repairs a Tip at the earlier danger threshold. Those actions need the arguments above and Rust tests. | -| Invariant | Meaning | -|-----------|---------| -| ZombieSafety | `schedulerExpected = CountGold(spine)` -- scheduler accepts exactly the Gold prefix | -| BatchNoncesContiguous | Batch nonces are 0..N-1 for non-Tip spine | -| InvalidOnlyOnGold | Dead branches only hang off Gold nodes | -| L1WNonceUnique | No two L1 entries share a `w_nonce` | -| L1BeforeCursor | All L1 entries have `w_nonce < nextL1Slot` | -| SchedulerBehindL1 | Scheduler cursor doesn't pass L1 cursor | -| DeadNotYetIncluded | Dead batches have `w_nonce >= nextL1Slot` | +The model's `Gold`, `Silver`, `Bronze`, `Pending`, and `Tip` colors describe +stages of inclusion and acceptance. `Gold* Silver* Bronze* Pending* Tip` is +**not** an invariant: flushing can leave a killed Pending before a surviving +Silver. Do not build implementation assumptions on that ordering. -### `admission.tla` -- Sequential startup and admission +The configured finite bounds are in [`preemptive.cfg`](preemptive.cfg). +The model has neither crash/restart nor the wall-clock freshness policy. -Models the local terminal gate, initial Sync with warm-provider fallback, repair selection, guarded Tip repair, Flush → Sync → Cascade with an ephemeral observation, Sync-discovered divergence, post-repair checking, task-free preparation, and final current admission. Retry, refusal, and crash return to a fresh attempt over surviving durable facts; terminal-fault telemetry does not gate the next boot. +### `admission.tla`: startup and permission to launch -**Verified**: 554 generated states, 155 distinct states, depth 11, 0 violations. +[`admission.tla`](admission.tla) models the local terminal gate, initial-sync +fallback, at most one repair, flush observation and mandatory sync, guarded +cascade, post-repair inspection, task-free preparation, and final admission. +Retry, refusal, or owner loss starts a fresh attempt over surviving durable +facts; the flush observation and admission witness do not survive. -The invariants cover runtime admission soundness, terminal dominance, repair preconditions, the caught-up post-flush view, and observation scope across attempts. They express these safety obligations independently of the number of inspections. Concrete SQLite transaction atomicity and guards are tested in Rust; the batch spine remains in `preemptive.tla`. +Its invariants cover terminal dominance, repair preconditions, a caught-up +post-flush view, and admission soundness. It abstracts successful repair as +producing a Tip; concrete transactions and rollback are checked by Rust tests. +Neither model covers external era/generation metadata, the application-input +projection, or snapshot artifact/lease/GC durability. Those obligations remain +in storage constraints, tests, and the [snapshot lifecycle](../snapshots/lifecycle.md). -### Running the spec +Run the configured checks with: ```bash tlc -workers auto -deadlock docs/recovery/admission.tla -tlc -workers auto -deadlock docs/recovery/preemptive.tla # ~90s +tlc -workers auto -deadlock docs/recovery/preemptive.tla just -f docs/recovery/justfile check-all ``` -Bounds are in `admission.cfg` and `preemptive.cfg`. The `MaxWalletNonce` bound keeps the slot model finite (kill/resubmit cycles generate new `w_nonce` values). Increase bounds for higher confidence at the cost of longer runtime. +## Implementation and test map + +| Concern | Owner and useful tests | +|---|---| +| Startup dispatch, error classification, final admission | [`recovery/mod.rs`](../../sequencer/src/recovery/mod.rs); procedure tests substitute only L1 sync/flush, keeping real SQLite inspections and repairs. | +| Detection, mutation guards, pivot and atomic cascade | [`storage/recovery.rs`](../../sequencer/src/storage/recovery.rs), [`recovery_tests.rs`](../../sequencer/src/storage/recovery_tests.rs); exact Tip guard, safe-view floor, unconditional post-flush policy, generation rollback, root nonce, and direct replay. | +| Observed danger versus estimates, accepted frontier and content identity | [`storage/l1_submission.rs` tests](../../sequencer/src/storage/l1_submission.rs), [`safe_accepted_batches.rs`](../../sequencer/src/storage/safe_accepted_batches.rs); stale-view precedence, clock faults, reused nonces, divergence freeze. | +| Slot settlement and broadcast coverage | [`recovery/flusher.rs`](../../sequencer/src/recovery/flusher.rs), [`l1/watermark.rs`](../../sequencer/src/l1/watermark.rs), [`submitter/poster.rs`](../../sequencer/src/l1/submitter/poster.rs). | +| Task-free preparation, launch, restore and catch-up | [`commands/run/`](../../sequencer/src/commands/run/), [`inclusion_lane/mod.rs`](../../sequencer/src/ingress/inclusion_lane/mod.rs). | + +The accepted-frontier cache stores acceptances, not scan progress. Rejected +inputs after the frontier may be rescanned on later syncs; a separate persistent +cursor would need nonce-reuse reasoning and tests. This is a storage performance +tradeoff, not an extra recovery phase or a modeled TLA+ invariant. diff --git a/docs/recovery/history/README.md b/docs/recovery/history/README.md index 74e173e..c4ea14e 100644 --- a/docs/recovery/history/README.md +++ b/docs/recovery/history/README.md @@ -1,56 +1,77 @@ # Recovery Design History -This directory preserves the optimistic recovery design -- an alternative to the preemptive approach documented in the parent [`README.md`](../README.md). Both designs are sound. We preferred preemptive for its operational properties. +This directory preserves the **historical optimistic recovery design** and its +counterexample. It is not the production recovery procedure. The +[current recovery guide](../README.md) owns automatic recovery; the +[cockroach guide](../cockroach.md) owns manual rebuilding. -## The Optimistic Design +## The optimistic alternative -In the optimistic design, the sequencer keeps accepting user operations and building batches while recovery plays out in the background. If a batch goes stale, the system detects it when the batch becomes Silver (safe on L1), cascade-invalidates, and submits recovery batches -- all while the sequencer continues serving soft confirmations. +The sequencer would keep accepting user operations and building batches while +recovery ran concurrently. The retained [model](optimistic.tla) permits a +cascade only when the first unresolved batch is **Silver** (included in a safe +L1 block) and stale by its **inclusion block**. Recovery replaces the suffix and +resets the next wallet nonce to the next unconsumed L1 slot. Submitted batches +from the invalidated suffix may remain in the network as zombies competing with +new recovery batches. -The TLA+ spec [`optimistic.tla`](optimistic.tla) models this design with a scheduler, wallet nonces, zombie batches (invalidated batches still in the L1 mempool), and adversarial L1 inclusion. At each `w_nonce` slot where a zombie and a recovery batch compete, L1 non-deterministically picks one (wallet-nonce mutual exclusion). +The recorded bounded check reported 194M states with no invariant violations +after the Silver-only fix. This is model evidence, not a proof of production +recovery or its completion time. Bounds are in [`optimistic.cfg`](optimistic.cfg). -**Verified**: 194M states, 0 violations (after the Silver-only fix below). +## The counterexample: invalidating before slot resolution -## The Silver-Only Constraint +The rejected variant allowed an unresolved frontier to be invalidated based on +its current age, before its L1 outcome was settled. The danger was **cascading +and reusing wallet-nonce slots**, not detecting danger early. -Both designs share a critical constraint: **recovery must wait for the frontier batch to be Silver before cascade-invalidating.** - -This constraint was discovered through the optimistic model. The original design allowed staleness detection on Pending or Bronze batches (a "short-circuit" for faster recovery). TLA+ found a counterexample: - -Three batches with `MAX_WAIT_BLOCKS = 2`: +Take `MAX_WAIT_BLOCKS = 2` and three original batches: +```text +batch nonce 0 1 2 +safe_block 0 0 1 +wallet nonce 0 1 2 ``` -batch bn=0 bn=1 bn=2 -sb 0 0 1 -wn 0 1 2 -``` - -With `currentSafeBlock = 2`, `bn=1` is stale by current block, `bn=2` is fresh. If we cascade from `bn=1`, both become zombies. Recovery creates a new `bn=1` at `wn=1`. -At L1 slot 1, zombie `bn=1` and recovery `bn=1` compete (same `w_nonce`): +Assume batch 0 is already accepted. At `currentSafeBlock = 2`, batch 1 is old +enough to be stale if included now, while batch 2 is still fresh. If recovery +invalidates batches 1 and 2 while they are pending, it can submit a fresh +replacement batch 1 at wallet nonce 1. -- **Zombie wins**: scheduler sees it, stale, skip. Nonce poisoned. Safe. -- **Recovery wins**: zombie `bn=1` dies (never reaches L1). Recovery accepted. `schedulerExpected` advances to 2. Zombie `bn=2(wn=2)` is fresh (`inclusion_block - safe_block = 1 < 2`), matches expected nonce -> **accepted**. The scheduler executes invalidated batch data. +At L1 slot 1, the original and replacement compete: -The two protection layers (wallet-nonce mutual exclusion and nonce poisoning) undercut each other: mutual exclusion kills the batch that nonce poisoning needs. +- **Original wins:** the scheduler sees the stale batch and leaves its expected + batch nonce at 1. The original batch 2 then fails the nonce check. +- **Replacement wins:** the original batch 1 cannot land. The fresh replacement + advances the scheduler's expected nonce to 2. If the original batch 2 lands + in block 2, its age is `2 - 1 < 2` and its nonce matches: the scheduler accepts + data the sequencer already invalidated. -The fix: only detect staleness when the frontier is Silver (safe on L1, immutable). The scheduler is guaranteed to see it before any recovery batch. +Wallet-nonce mutual exclusion removed the stale batch that the nonce-poisoning +argument depended on. The retained optimistic model's `Resolve` therefore +requires a Silver frontier that is stale by inclusion: that original batch is +already on safe L1 and cannot be displaced by a replacement. -## Why We Chose Preemptive +## Why production uses preemptive recovery -Both designs are sound once Silver-only detection is enforced. The difference is operational: +Production closes intake and performs recovery offline. For closed-batch +recovery, the flush consumes every covered wallet-nonce slot at safe depth, +**whether the original transaction or a no-op wins**, then re-syncs the accepted +prefix before cascading. It does not require the original frontier batch to +become Silver. An unsubmitted open Tip has no wallet slot to settle. -**Both designs wait.** Any recovery design must wait for the frontier to become Silver before cascading. In the optimistic design, the sequencer keeps issuing soft confirmations during this wait -- confirmations that will be invalidated when the cascade fires. In the preemptive design, the sequencer goes offline before the cascade, so no doomed soft confirmations are issued. +This gives recovery a sequential procedure and stops new soft confirmations +while submission uncertainty is being resolved. The optimistic alternative +keeps serving through that interval and may add confirmations that a later +cascade revokes. -**Preemptive is simpler to reason about.** The optimistic design has concurrent actors: the batch submitter, the inclusion lane, L1 mempool competition, and recovery all interleave. The preemptive design is sequential: stop, flush, recover, resume. Each step has clear preconditions and postconditions. +The tradeoff is downtime. Flush completion requires L1 progress; fee headroom +does not guarantee replacement or establish a deadline. The +[current recovery guide](../README.md) owns the safety conditions, and the +[L1 fee policy](../../l1-fee-policy.md) owns the accepted liveness limits. -**Preemptive eliminates mempool races.** The flush resolves all `w_nonce` slot uncertainty before recovery runs. Recovery operates on fully-finalized L1 state. No zombie mutual exclusion needed. - -**The cost is downtime.** Preemptive recovery takes the sequencer offline for the duration of the flush + safe finality wait (~15-20 minutes on Ethereum). For a rare event (a batch approaching the 4-hour staleness deadline), this is acceptable. - -## Running the Spec +## Running the historical model ```bash -tlc -workers auto -deadlock docs/recovery/history/optimistic.tla # ~3min +tlc -workers auto -deadlock docs/recovery/history/optimistic.tla ``` - -Bounds are in `optimistic.cfg`. diff --git a/docs/snapshots/lifecycle.md b/docs/snapshots/lifecycle.md index f99ad0d..0e9cd3f 100644 --- a/docs/snapshots/lifecycle.md +++ b/docs/snapshots/lifecycle.md @@ -127,6 +127,9 @@ the enclosing dump directories recursively. This filesystem operation disposes of all checkpoint resources. Filesystem deletion failure leaves a harmless orphan for the startup sweep. The reverse ordering would leave a durable row pointing at missing state and is forbidden. Startup clears leases left by the dead process, -validates a rollback checkpoint, collects obsolete rows, and sweeps orphan -directories before workers start. Missing or corrupt referenced artifacts fail -loud; operational filesystem errors retain their normal error classification. +checks the rollback checkpoint's `info.toml` and format version, collects obsolete +rows, and sweeps orphan directories before workers start. Application restoration +runs afterward in the launched inclusion lane, before processing new user ops; +the metadata check does not validate the application bytes. Missing or corrupt +referenced artifacts fail loud when read or restored; operational filesystem +errors retain their normal error classification. diff --git a/sequencer/src/commands/config.rs b/sequencer/src/commands/config.rs index 3702bdf..33837c7 100644 --- a/sequencer/src/commands/config.rs +++ b/sequencer/src/commands/config.rs @@ -43,9 +43,9 @@ pub struct TimingArgs { /// The danger threshold is MAX_WAIT_BLOCKS minus this margin. /// Must be less than MAX_WAIT_BLOCKS (validated at startup). /// - /// Default 300 (~1h at 12s/block) is sized to give operators meaningful - /// runway to investigate before the system gives up on the current - /// batches — see `docs/recovery/README.md` "Step 1: Danger threshold". + /// Default 300 gives ~1h of headroom before canonical expiry at 12s/block. + /// Detection starts recovery immediately; this is neither an operator + /// grace period nor a completion deadline. See `docs/recovery/README.md`. #[arg( long, env = "CARTESI_SEQUENCER_PREEMPTIVE_MARGIN_BLOCKS", diff --git a/sequencer/src/commands/run/startup_hygiene.rs b/sequencer/src/commands/run/startup_hygiene.rs index 409acaa..e6005f4 100644 --- a/sequencer/src/commands/run/startup_hygiene.rs +++ b/sequencer/src/commands/run/startup_hygiene.rs @@ -1,7 +1,7 @@ // (c) Cartesi and individual authors (see AUTHORS) // SPDX-License-Identifier: Apache-2.0 (see LICENSE) -//! Startup clears stale leases, validates the rollback checkpoint, then collects +//! Startup clears stale leases, checks rollback checkpoint metadata, then collects //! obsolete snapshots and orphan directories before workers are admitted. use crate::commands::error::CommandError; diff --git a/sequencer/src/commands/run/workers.rs b/sequencer/src/commands/run/workers.rs index 7ce2257..9fde839 100644 --- a/sequencer/src/commands/run/workers.rs +++ b/sequencer/src/commands/run/workers.rs @@ -184,7 +184,7 @@ impl PreparedRuntime { let dumps_dir = std::path::Path::new(&run_config.data_dir).join("dumps"); std::fs::create_dir_all(&dumps_dir)?; - // Validate the rollback artifact and collect obsolete snapshots before admission. + // Validate rollback checkpoint metadata and collect obsolete snapshots before admission. super::startup_hygiene::run_snapshot_hygiene(&mut storage, &dumps_dir)?; // Prepare every remaining fallible or awaited dependency before the diff --git a/sequencer/src/ingress/inclusion_lane/catch_up.rs b/sequencer/src/ingress/inclusion_lane/catch_up.rs index cfd7ce8..1571437 100644 --- a/sequencer/src/ingress/inclusion_lane/catch_up.rs +++ b/sequencer/src/ingress/inclusion_lane/catch_up.rs @@ -1,7 +1,7 @@ // (c) Cartesi and individual authors (see AUTHORS) // SPDX-License-Identifier: Apache-2.0 (see LICENSE) -//! Restore the application's committed history suffix before runtime admission. +//! Replay committed application history after launch, before processing new user ops. use std::path::PathBuf; diff --git a/sequencer/src/recovery/flusher.rs b/sequencer/src/recovery/flusher.rs index bf9cc6a..ee4242f 100644 --- a/sequencer/src/recovery/flusher.rs +++ b/sequencer/src/recovery/flusher.rs @@ -157,10 +157,10 @@ impl MempoolFlusher { /// Flush the mempool by submitting no-op transactions for unresolved /// nonce slots, then waiting until every slot we ever used is safe. /// - /// `watermark` is the persisted wallet-nonce watermark — the highest - /// nonce this deployment ever broadcast, or `None` if nothing was ever - /// broadcast (or no DB survives, the cockroach-recovery best-effort - /// case). The loop runs until + /// `watermark` durably covers every wallet nonce this deployment may + /// have broadcast. Write-before-send can cover an unused slot. `None` + /// means no covered broadcasts, or a lost DB in cockroach recovery's + /// best-effort flush. The loop runs until /// /// ```text /// pending <= safe && safe >= watermark + 1 diff --git a/sequencer/src/storage/ingress.rs b/sequencer/src/storage/ingress.rs index 095f1fa..2d7c090 100644 --- a/sequencer/src/storage/ingress.rs +++ b/sequencer/src/storage/ingress.rs @@ -97,7 +97,7 @@ impl Storage { /// warm resume and recovery batches use), so there is no cold-start drain. /// /// This unguarded form exists for test harnesses only. Production uses - /// `ensure_open_tip_for_recovery`, which reasserts the reducer facts in its + /// `ensure_open_tip_for_recovery`, which reasserts the startup facts in its /// write transaction. The lane only ever *loads* a Tip (fail-loud if /// absent); Cascade owns its own atomic reopen using the same mechanism. /// @@ -471,7 +471,8 @@ pub(super) fn open_fresh_tip_in_tx(tx: &Transaction<'_>) -> Result<()> { } /// Capture the unaccounted range before creating its new frame, then attribute -/// its external directs. Catch-up executes these rows before runtime admission. +/// its external directs. After launch, lane catch-up executes these rows before +/// processing queued user operations. fn insert_draining_tip_with_executions( tx: &Transaction<'_>, batch_index: Option, diff --git a/sequencer/src/storage/mutations.rs b/sequencer/src/storage/mutations.rs index e62cf6f..8495b2a 100644 --- a/sequencer/src/storage/mutations.rs +++ b/sequencer/src/storage/mutations.rs @@ -16,8 +16,8 @@ use super::l1_inputs::query_deployment_identity; use super::{DirectInputExecution, SafeInputRange}; /// Insert a new batch. Nonce is derived from `parent_batch_index`: -/// `parent.nonce + 1`, or 0 if `parent_batch_index` is None (genesis or -/// post-cascade torn-state new Tip). +/// `parent.nonce + 1`, or the deployment's anchor for a parentless root +/// (genesis, cockroach recovery, or a fully invalidated branch). /// /// If `batch_index_opt` is None, SQLite auto-assigns (highest existing +1). /// The explicit form is used only by `initialize_open_state` to pin the @@ -64,9 +64,7 @@ pub(super) fn insert_new_batch( fn compute_next_nonce(tx: &Transaction<'_>, parent_batch_index: Option) -> Result { match parent_batch_index { // A parentless root carries the deployment's batch-tree anchor nonce: - // 0 for a genesis deployment, N' for a cockroach-recovered one. This - // generalizes the old hard-coded 0; the `batch_tree_anchor` row defaults - // to 0, so genesis and post-cascade re-roots are unchanged. Mirrored by + // 0 for a genesis deployment, N' for a cockroach-recovered one. Mirrored by // `trg_enforce_nonce_contiguity`'s parentless arm. None => batch_tree_anchor_in(tx), Some(parent_bi) => { diff --git a/sequencer/src/storage/recovery.rs b/sequencer/src/storage/recovery.rs index 6b7dbd2..f09e5eb 100644 --- a/sequencer/src/storage/recovery.rs +++ b/sequencer/src/storage/recovery.rs @@ -1,11 +1,10 @@ // (c) Cartesi and individual authors (see AUTHORS) // SPDX-License-Identifier: Apache-2.0 (see LICENSE) -//! Recovery writer: cascade-invalidates stale batches, opens recovery batches, -//! and composes the startup-recovery transaction. +//! Recovery storage: danger inspection, guarded suffix invalidation, and Tip creation. //! -//! See `docs/recovery/README.md` for the full design (batch tree, coloring, -//! nonce poisoning, TLA+ proof). This file's job is to enforce that design +//! See `docs/recovery/README.md` for the procedure, safety arguments, and +//! bounded model coverage. This file's job is to enforce that design //! locally — read the design first if you're touching this code. //! //! Free functions here are shared with the batch submitter @@ -40,8 +39,7 @@ use super::snapshot_dumps::has_rollback_safe_snapshot_in; /// Each variant maps to a distinct response in the startup recovery procedure: /// /// - `L1ViewStale` → retry boot. The L1 safe block is too old or unknown. -/// - `ClosedBatchInDanger(closed_idx)` → enter the phase-granular -/// Flush/Sync/Cascade sequence. +/// - `ClosedBatchInDanger(closed_idx)` → Flush → Sync → Cascade. /// - `TipInDanger(tip_idx)` → direct Tip recovery, no flush. The Tip has no L1 /// footprint, so we can invalidate it and open a fresh one without /// any L1 round-trip. @@ -88,7 +86,7 @@ pub enum DangerStatus { /// One transactionally consistent local view consumed by the startup /// recovery procedure. /// -/// Keeping these facts together is load-bearing: admission and recovery-phase +/// Keeping these facts together is load-bearing: admission and repair /// selection must not combine a danger verdict from one SQLite snapshot with /// Tip/snapshot/head facts from another. #[derive(Debug, Clone, Copy, PartialEq, Eq)] @@ -100,7 +98,7 @@ pub(crate) struct RecoveryInspection { } /// A recovery mutation was refused because the transaction no longer -/// satisfies the phase selected by the reducer. +/// satisfies the selected startup action's preconditions. #[derive(Debug, thiserror::Error)] pub(crate) enum RecoveryMutationError { #[error(transparent)] @@ -114,15 +112,13 @@ pub(crate) enum RecoveryMutationError { }, #[error("cannot open the Tip without a recovery checkpoint")] MissingRecoveryCheckpoint, - /// The `EnsureOpenTip` phase found a valid open Tip already present. A + /// `EnsureOpenTip` found a valid open Tip already present. A /// stale no-Tip decision, not a danger change; unreachable under the /// process lock, and retryable if it ever fires. #[error("the Tip was already open when the EnsureOpenTip phase ran")] TipAlreadyOpen, - /// The `EnsureOpenTip` phase's own transaction left no valid open Tip - /// after opening one. Impossible by construction; refused rather than - /// committed, so the reducer's one cycle (Repaired → EnsureOpenTip → - /// Repaired) cannot spin on it. + /// `EnsureOpenTip` left no valid open Tip after opening one. This broken + /// postcondition must roll back and refuse startup. #[error("the EnsureOpenTip phase left no valid open Tip in its own transaction")] TipMissingAfterOpen, #[error( @@ -247,7 +243,7 @@ impl Storage { self.read(|tx| inspect_recovery_in(tx, protocol, now_ms)) } - /// Execute the reducer's `EnsureOpenTip` phase only if its local decision + /// Execute `EnsureOpenTip` only if its local decision /// still holds in the write transaction. pub(crate) fn ensure_open_tip_for_recovery( &mut self, @@ -272,11 +268,8 @@ impl Storage { return Err(RecoveryMutationError::TipAlreadyOpen); } open_fresh_tip_in_tx(&tx)?; - // Postcondition, enforced where it can be violated: this phase is the - // only edge back into `Repaired` without a Tip, so it must never - // commit without one. A violation is a typed refuse (exit 30), never - // a retry that would spin the reducer, and never a `debug_assert` - // that compiles out. + // A broken postcondition must roll back this transaction and refuse + // startup, including in release builds. if !has_valid_open_batch(&tx)? { return Err(RecoveryMutationError::TipMissingAfterOpen); } @@ -284,7 +277,7 @@ impl Storage { Ok(()) } - /// Execute the reducer's `RecoverTip` phase only while the same Tip is + /// Execute `RecoverTip` only while the same Tip is /// still the observed-danger arm in the write transaction. pub(crate) fn recover_aging_tip_for_recovery( &mut self, @@ -312,7 +305,7 @@ impl Storage { Ok(invalidated) } - /// Execute the reducer's `Cascade` phase. The ephemeral flush witness is + /// Execute post-flush Cascade. The boot-local flush observation is /// represented by its observed safe-block floor; this transaction /// reasserts both I15 and the post-flush resync coherence check /// immediately before changing the batch tree. @@ -345,7 +338,8 @@ impl Storage { } /// Mark a single batch as invalid. Test-only seeder — production code goes - /// through [`Storage::recover_post_flush`] or [`Storage::recover_aging_tip`]. + /// through [`Storage::recover_post_flush_for_recovery`] or + /// [`Storage::recover_aging_tip_for_recovery`]. /// Idempotent: leaves already-invalid rows alone. #[cfg(test)] pub(crate) fn insert_invalid_batch(&mut self, batch_index: u64) -> Result<()> { @@ -368,7 +362,7 @@ impl Storage { /// Test-only unguarded primitive; production calls /// [`Storage::recover_aging_tip_for_recovery`], which transactionally - /// reasserts the exact reducer decision. Design rationale on + /// reasserts the exact startup decision. Design rationale on /// [`recover_aging_tip_inner`], the shared body. #[cfg(test)] pub fn recover_aging_tip(&mut self, danger_threshold: u64) -> Result> { @@ -441,126 +435,32 @@ fn refuse_divergence(danger: DangerStatus) -> std::result::Result<(), RecoveryMu // ── Free functions used by both recovery and the batch submitter ────────── -/// Cascade the non-gold suffix and open a fresh recovery batch (the shared -/// body behind [`Storage::recover_post_flush_for_recovery`], which the -/// reducer reaches after carrying a Flush witness through a caught-up -/// Sync). Homed here, not on the test wrapper, so rustdoc builds it and a -/// wrapper cleanup cannot delete the design record. +/// Discard the entire non-accepted closed suffix after flush and caught-up Sync. +/// This is a convergence policy: replaced or never-submitted work could be +/// submitted fresh, but preserving it can re-enter the same danger/recovery cycle. +/// The caller must settle wallet slots and refresh acceptance through the flush +/// observation; the production guard checks that the local view caught up. /// -/// # The "everything past gold is doomed" rule -/// -/// At this point the gold frontier is at its maximum extent: every -/// submitted batch has either been accepted (gold) or rejected by the -/// scheduler simulation (Silver-stale, since nonce-mismatch is impossible -/// at the frontier under self-trust), or its tx was killed by a flush -/// no-op (Pending, no `safe_input`). All three non-gold states are doomed: -/// -/// - **Silver-stale:** scheduler skipped it; downstream batches are -/// nonce-poisoned. -/// - **Pending:** the original L1 tx is dead. Re-submission could in -/// principle land fresh, but the *next* recovery cycle's flush would -/// compete with the resub at its new wallet-nonce slot and the bumped -/// no-op typically wins. The system would loop until current staleness -/// crossed `MAX_WAIT_BLOCKS`. Cascading now converges in one cycle. -/// -/// So once we've committed to recovery (the danger detector tripped, the -/// flush ran), the right move is to cascade the entire non-gold suffix -/// and open a fresh recovery batch. -/// -/// Three aftermath shapes: -/// -/// 1. **Everything worked:** all in-flight batches landed fresh and were -/// accepted. Gold extends to the last submitted batch; no first -/// non-gold closed. (See "Tip handling" below for the subtle subcase.) -/// 2. **Mixed:** some landed (stale or poisoned), some replaced. First -/// non-gold closed is either Silver-stale or Pending. Cascade from -/// there; the `batch_index >= N` rule catches the rest of the suffix -/// including the open Tip. -/// 3. **All replaced:** flush no-ops won every race. Gold doesn't -/// advance; first non-gold closed is the very first non-accepted batch. -/// -/// # Tip handling -/// -/// In cases (2)/(3) the cascade catches the Tip via `batch_index >= N`. -/// In case (1), there's no closed pivot — but the Tip can still be in -/// the danger zone: -/// -/// When the lane rotates a batch without a safe-block advance between -/// frames (e.g. immediately after init, when both share the bootstrap -/// `safe_block`), the Tip's `first_frame.safe_block` equals the closed -/// batch's. The closed batch can become gold by inclusion-staleness -/// (`inclusion_block - first_frame < MAX_WAIT`) while the Tip's age, -/// computed against `current_safe_block` after the flush wait, has -/// crossed `danger_threshold`. Pure monotonicity (`S_tip ≥ S_closed`) doesn't -/// rule this out — equality is allowed. -/// -/// So in the no-pivot branch we additionally check the Tip against -/// `danger_threshold` (the same threshold that would have triggered -/// recovery had the Tip been a closed batch). We're already committed -/// to recovery; the Tip is past gold; if it's also in the danger zone, -/// cascade it and open a fresh one. -/// -/// # Atomicity -/// -/// Runs as a single SQLite write transaction. On crash mid-way, the -/// txn rolls back; on commit, the cascade and the recovery batch open -/// land together. Idempotent on re-run because `valid_*` views filter -/// out already-invalidated rows. -/// -/// # Precondition -/// -/// The caller MUST have just synced L1 state via -/// [`Storage::append_safe_inputs`]; the gold frontier in -/// `safe_accepted_batches` must reflect the latest safe head. Otherwise -/// the cascade may invalidate batches that haven't yet had a chance to -/// be processed by the scheduler simulation. +/// If no closed pivot remains, only an aging Tip is invalidated. A closed batch +/// can have landed fresh while the Tip, even with the same first-frame clock, +/// has since crossed the danger threshold. Cascade and reopening share `tx`. /// /// Returns the newly-invalidated batch indices (empty if none). fn recover_post_flush_inner(tx: &Transaction<'_>, danger_threshold: u64) -> Result> { // Path 1: any closed batch past gold cascades unconditionally. let pivot = match first_non_gold_closed_batch(tx)? { Some(batch_index) => Some(batch_index), - // Path 2 (corner case): all closed are gold, but the Tip might be - // in the danger zone — see `recover_post_flush` doc on Tip handling. + // All closed batches are accepted; the Tip can still have aged. None => find_tip_batch_in_danger(tx, danger_threshold)?, }; cascade_and_reopen(tx, pivot) } -/// Cascade the open Tip if its first frame has aged past -/// `danger_threshold` (the shared body behind -/// [`Storage::recover_aging_tip_for_recovery`]). Homed here, not on the -/// test wrapper, so rustdoc builds it and a wrapper cleanup cannot delete -/// the design record. -/// -/// # Why a threshold here, but no closed-frontier check -/// -/// Outside a flush path, closed batches past the gold -/// frontier (if any) might still be in their natural lifecycle — -/// pending in the mempool, recently included, awaiting safe finality. -/// Cascading them would prematurely abort their progression. -/// -/// The Tip is different: it has no L1 footprint at all (no `w_nonce`, -/// no `safe_input`), so there's no L1 outcome to wait on. Once its -/// first frame has aged into the danger zone, the rule "everything -/// past gold is bad once we're committed to recovery" applies, and in -/// the `RecoverTip` path startup is already committed. -/// -/// # Threshold = danger_threshold, not MAX_WAIT -/// -/// We use `danger_threshold` (= `MAX_WAIT_BLOCKS - margin`) rather than -/// `MAX_WAIT_BLOCKS`. The Tip threshold is the same one that would -/// trigger the recovery cycle had the Tip been a closed batch. If the -/// Tip is past that threshold, the next danger detector tick after -/// resume would re-trip on the Tip's eventual first close + submission -/// anyway (the closed batch would inherit its first frame's safe_block). -/// Cascading now saves the cycle. -/// -/// # Precondition -/// -/// As with [`Storage::recover_post_flush`], the caller must have synced -/// L1 state. (Threshold comparison reads `current_safe_block` from -/// `l1_safe_head`.) +/// Discard only the aging Tip. It has no L1 footprint, so no flush is required. +/// Using `danger_threshold` avoids restarting with the same age that triggered +/// recovery; it is a policy threshold, not proof of canonical staleness. +/// The production caller rechecks the exact `TipInDanger` decision against the +/// current local inspection before entering this shared body. /// /// Returns the newly-invalidated batch indices (empty if Tip is fresh, /// `[tip_index]` when the Tip was cascaded). @@ -604,15 +504,12 @@ fn cascade_and_reopen(tx: &Transaction<'_>, pivot: Option) -> Result=`, not `>`: `frontier_nonce` is the *next-expected* nonce -/// (`latest_accepted.nonce + 1`), so the actual cascade-pivot batch carries -/// `nonce == frontier_nonce`. Using `>` would skip it. +/// (`latest_accepted.nonce + 1`, or the anchor before any acceptance), so the +/// actual cascade-pivot batch carries `nonce == frontier_nonce`. Using `>` +/// would skip it. /// -/// On the valid path, batch nonces are contiguous (enforced by the -/// `trg_enforce_nonce_contiguity` trigger), so the first match always has -/// `nonce == frontier_nonce`. We don't double-check that invariant here — -/// the trigger is the source of truth (see AGENTS.md "Self-trust": no -/// defense-in-depth checks against the sequencer's own bugs). Returns -/// `None` if all closed batches are gold. +/// Valid-path nonce contiguity (I16) makes the first match exactly +/// `frontier_nonce`. Returns `None` if all closed batches are accepted. fn first_non_gold_closed_batch(conn: &Connection) -> Result> { let frontier = frontier_nonce(conn)?; let batch_index: Option = conn @@ -629,13 +526,10 @@ fn first_non_gold_closed_batch(conn: &Connection) -> Result> { /// Either the closed-frontier batch or the Tip, whichever (if either) has /// aged past `threshold` against `current_safe_block`. Used by /// [`Storage::check_danger`]'s wall-clock-adjusted arm, where the dispatch -/// is the same (`Refuse`) regardless of which one fired. +/// is the same (`Retry`) regardless of which one fired. /// /// Closed-frontier wins: frame `safe_block`s are non-decreasing along the -/// spine, so the closed frontier is at least as *old* as the Tip — whenever -/// the Tip is in danger, the closed frontier is too, and cascading from the -/// closed batch covers the Tip via `batch_index >= N`. (This ordering is -/// also determines which batch snapshots remain valid.) +/// spine, so an existing closed frontier is at least as old as the Tip. /// /// Reads `safe_accepted_batches`, which is maintained atomically with each /// [`Storage::append_safe_inputs`] call. @@ -650,14 +544,9 @@ pub(super) fn find_first_batch_in_danger(conn: &Connection, threshold: u64) -> R /// than `current_safe_block - threshold`. Returns `None` if no such batch /// exists. /// -/// Why look only at the frontier batch, not "every batch past gold"? -/// `safe_accepted_batches` is updated atomically with each safe-head advance -/// (see [`super::safe_accepted_batches`]) and walks the spine until it hits -/// a barrier — a stale batch, or a missing slot the scheduler can't bridge. -/// So the first batch past the frontier IS the barrier; downstream batches -/// are nonce-poisoned by definition (a stale frontier ⇒ scheduler skips ⇒ -/// every later batch arrives at an unexpected nonce). Looking further is -/// redundant. +/// First-frame clocks are non-decreasing along the valid path (I3), so the +/// earliest non-accepted closed batch is at least as old as its successors. +/// Checking younger batches cannot reveal danger that this check missed. /// /// Does NOT consider the Tip — the Tip has no L1 transaction, so it's not /// part of the closed-frontier-staleness category. From 3ef56ec30e685792b5d869d25a20baa3d69b3548 Mon Sep 17 00:00:00 2001 From: gcdepaula Date: Thu, 17 Sep 2026 15:33:18 -0300 Subject: [PATCH 3/4] docs: align history snapshots and watchdog contracts --- AGENTS.md | 2 +- README.md | 38 ++- docs/plans/2026-07-coordination-tracks.md | 66 ++-- .../2026-07-track3-feed-replay-design.md | 150 ++-------- docs/plans/2026-07-track6-dump-api-design.md | 2 +- docs/plans/2026-08-authority-boundary-adr.md | 4 +- docs/plans/application-history.md | 92 ------ docs/protocol/application-contract.md | 45 ++- docs/protocol/application-history.md | 150 ++++++++++ docs/protocol/scheduler-semantics.md | 2 +- docs/recovery/cockroach.md | 2 +- .../2026-09-09-application-lane-dex-review.md | 2 +- docs/review/register.md | 6 +- docs/snapshots/README.md | 20 +- docs/snapshots/format.md | 135 ++------- docs/watchdog/README.md | 283 +++++++++--------- docs/watchdog/design-notes.md | 6 +- docs/watchdog/getting-started.md | 34 ++- docs/watchdog/operator-deployment.md | 106 ++++--- 19 files changed, 525 insertions(+), 620 deletions(-) delete mode 100644 docs/plans/application-history.md create mode 100644 docs/protocol/application-history.md diff --git a/AGENTS.md b/AGENTS.md index ba2494d..a64aae4 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -445,7 +445,7 @@ work reaches another boundary. | Application implementation, execution, or native integration | [Application contract](docs/protocol/application-contract.md) — determinism, progress, failure, capacity, and checkpoints; [C binding](docs/protocol/c-application-binding.md) for native engines. | | Automatic recovery or danger detection | [Automatic recovery](docs/recovery/README.md), then [preemptive.tla](docs/recovery/preemptive.tla) and [admission.tla](docs/recovery/admission.tla) — repair ordering and the models' bounded guarantees. | | Manual rebuild after lost state or a sequencer bug | [Cockroach recovery](docs/recovery/cockroach.md) — trusted checkpoint, fixed input boundary, and fresh baseline. | -| API, subscriber replay, or application-history coordinates | [README API contract](README.md#api) — wire behavior; [application history](docs/plans/application-history.md) — era, generation, offsets, and recovery boundaries. | +| API, subscriber replay, or application-history coordinates | [README API contract](README.md#api) — wire behavior; [application history](docs/protocol/application-history.md) — era, generation, offsets, and recovery boundaries. | | Snapshots, restart, export, retention, or watchdog checkpoints | [Snapshot lifecycle](docs/snapshots/lifecycle.md) — durable publication, accepted comparison points, leases, and GC; [wallet format](docs/snapshots/format.md) when changing wallet bytes. | | Trust boundaries, provider behavior, or hostile L1 input | [Threat model](docs/threat-model/README.md) — actor assumptions, supported failures, and residual risks. | | Submission fees or oracle pricing | [L1 fee policy](docs/l1-fee-policy.md) — estimation and replacement limits; [threat-model actor table](docs/threat-model/README.md#actors-and-trust) — oracle source and outage assumptions. | diff --git a/README.md b/README.md index 68c1a0a..8dfc7cf 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # Sequencer -A sequencer for Cartesi app-specific rollups. Provides low-latency soft confirmations for user operations, posts them to L1 in batches, and maintains a deterministic replay feed that matches the application's final execution order. +A sequencer for Cartesi app-specific rollups. Provides low-latency soft confirmations for user operations, posts them to L1 in batches, and exposes its current application execution order for replica replay. **Security-critical infrastructure.** Handle every change with the care financial systems demand. @@ -86,7 +86,7 @@ The sequencer is designed to handle: - **L1 provider outages** — workers retry with exponential backoff. The inclusion lane and API continue operating locally. A wall-clock fallback detects when an outage pushes batches into the danger zone. - **Undiagnosed interruptions (OOM, SIGKILL, reboot)** — restart can recover automatically: every boot derives any required recovery from SQLite and L1 safe state through startup recovery, never assuming the previous exit was clean. Terminal errors returned through a command bracket best-effort record their cause in `terminal_faults`; terminal runtime aborts leave only process diagnostics. - **Extended downtime** — startup syncs to the current L1 safe head, flushes if needed, and recovers before admission. A terminal exit requires operator investigation; rebuilding untrustworthy state follows the cockroach recovery procedure above. -- **Adversarial L1 mempool** — block builders and private mempools are treated as adversarial. The recovery flusher consumes every pending nonce slot with a no-op so delayed "zombie" submissions cannot land later. +- **Adversarial L1 mempool** — block builders and private mempools are treated as adversarial. Recovery waits until every covered wallet-nonce slot is consumed at safe depth, whether the original transaction or a flush no-op wins, so delayed "zombie" submissions cannot land later. ## Interfaces @@ -96,7 +96,13 @@ Users submit signed operations via `POST /tx` (JSON). Operations are signed with ### Sequenced Transaction Feed -Subscribers connect via `GET /ws/subscribe?era_id=&recovery_generation=&next_input=` (WebSocket). The feed delivers all sequenced transactions (user ops + direct inputs) in deterministic order, matching the on-chain execution order. This is the primary interface for downstream consumers (frontends, indexers). The endpoint is designed for a small number of indexer subscribers, which serve users directly. +Subscribers restore an HTTP snapshot, then use one WebSocket stream to replay +application inputs and follow the optimistic tip. Recovery can replace that +history; the snapshot's era, generation, and input count bind a resume request +to the state the consumer actually holds. The endpoint serves a small number of +infrastructure subscribers, which serve users directly. See the +[bootstrap workflow](docs/protocol/application-history.md#replica-bootstrap-and-resume) +and [wire contract](#api). ### Batch Submission @@ -187,6 +193,16 @@ Notes: - queue capacity is an internal runtime constant tuned alongside inclusion-lane chunking to absorb short bursts; if this starts triggering persistently, it is a signal to revisit runtime sizing or throughput rather than add another admission layer. - Browser wallets can call `POST /tx` and `GET /fee` from any origin with any request headers; preflight permits GET and POST and is cached for one hour. CORS is applied only to ingress. Egress routes remain operator-only and require network access controls. +Success response after inclusion: + +```json +{ + "ok": true, + "sender": "0x...", + "nonce": 0 +} +``` + ### `GET /fee` Fee quote for setting signed user-op `max_fee` before `POST /tx`. All three fields are log-space exponents (base 129/128), the same encoding as `max_fee`. Inclusion rejects any op with `max_fee` below the open-frame `fee`. @@ -205,7 +221,7 @@ Notes: ### `GET /ws/subscribe?era_id=&recovery_generation=&next_input=` -WebSocket stream of canonical application inputs, replaying from the inclusive +WebSocket stream of the current application history, replaying from the inclusive `next_input` offset and then following the optimistic tip. Fetch and restore `/latest_snapshot` first; its headers supply the complete subscription claim. After each successfully applied input at offset `X`, persist the claim with @@ -235,16 +251,6 @@ Message shapes: { "kind": "direct_input", "offset": 11, "sender": "0x...", "block_number": 123, "block_timestamp": 1700000000, "transaction_hash": "0x...", "payload": "0x...", "input_index": 42, "batch_nonce": 4 } ``` -Success response: - -```json -{ - "ok": true, - "sender": "0x...", - "nonce": 0 -} -``` - ### Operator snapshot endpoints (internal only) These serve application state to the operator's watchdog and indexers. @@ -265,7 +271,7 @@ api split lands). adding a coherent `checkpoint.toml` receipt with its L1 inclusion block and next batch nonce for trusted recovery. -All state/archive responses include `X-History-Era`, `X-Recovery-Generation`, +Successful state/archive downloads include `X-History-Era`, `X-Recovery-Generation`, and `X-Executed-Input-Count`, selected atomically with the artifact lease. Streaming holds the lease until the response ends or the client disconnects. The accepted endpoints return `404` until a comparable checkpoint exists: @@ -341,6 +347,8 @@ validation for a change; some tests require Anvil or libslirp. - [`docs/threat-model/README.md`](docs/threat-model/README.md) — trust boundaries, in-scope and out-of-scope threats. - [`docs/recovery/README.md`](docs/recovery/README.md) — automatic recovery, TLA+ formal verification, design history. - [`docs/recovery/cockroach.md`](docs/recovery/cockroach.md) — manual rebuild after lost state or a sequencer bug. +- [Application history and replay](docs/protocol/application-history.md) — progress, history identity, and replica bootstrap/resume. +- [Snapshots](docs/snapshots/README.md) — engine checkpoints, lifecycle, accepted comparison, and wallet encoding. - [`docs/watchdog/getting-started.md`](docs/watchdog/getting-started.md) — step-by-step: run the watchdog with a local sequencer. - [`docs/watchdog/operator-deployment.md`](docs/watchdog/operator-deployment.md) — watchdog on live L1 (Sepolia staging, mainnet production). - [`docs/watchdog/README.md`](docs/watchdog/README.md) — watchdog architecture, modules, and test commands. diff --git a/docs/plans/2026-07-coordination-tracks.md b/docs/plans/2026-07-coordination-tracks.md index 8522267..b4781d7 100644 --- a/docs/plans/2026-07-coordination-tracks.md +++ b/docs/plans/2026-07-coordination-tracks.md @@ -14,17 +14,17 @@ freely at this stage — no backward-compatibility constraints. |---|-------|-------|--------| | 1 | WS context fields + L1 provenance (PR #26) | Stephen | **done** — merged to main | | 2 | Restore `docs/review/` ledger + this plan | us | **done** | -| 3 | Feed & replay protocol redesign | us (design) → us/Stephen (impl) | **implemented** — canonical application history, snapshot restore archives, mandatory WS claims, typed refusals, and SDK cutover; [remaining integration gates](2026-07-track3-feed-replay-design.md#5-acceptance-evidence-and-remaining-work) | +| 3 | Feed & replay protocol redesign | us (design) → us/Stephen (impl) | **implemented** — canonical application history, snapshot restore archives, mandatory WS claims, typed refusals, and SDK cutover; [remaining integration gates](2026-07-track3-feed-replay-design.md#remaining-integration-gates) | | 4 | Storage decode policy | us | **done** — fail-loud for contract-impossible values; the named `saturating_query_bound` only where clamping preserves the predicate (policy lives in `storage/convert.rs` + the invariants check policy) | | 5 | Fee exponentiation LUT | us | **deferred** — decided exact-floor if built (the table *is* the spec, algorithm-free; replay continuity across the upgrade explicitly not preserved); a separate pending design decision may make log-space fees defunct — revisit after syncing with Bart | -| 6 | Dump / `Application` API redesign | us + Bart | **revised interface implemented** — [Application contract](../protocol/application-contract.md); native bridge conformance is a separate integration branch | +| 6 | Dump / `Application` API redesign | us + Bart | **interface and reference C binding implemented** — [Application contract](../protocol/application-contract.md); native-engine integration gates remain | | 7 | LLM context-engineering review | us | **done** — skills/agents/settings homed in-tree; the docs-practice rules live in AGENTS.md | | 8 | Runtime ownership and terminal stop | us | **done** — owned by the [authority-boundary ADR](2026-08-authority-boundary-adr.md) | **Current campaign order:** -1. Validate Track 6 against the reference C bridge, then the private DEX engine when shared. -2. Exercise native-engine snapshot bootstrap and remeasure feed latency in the representative environment. +1. Validate snapshot-to-live replica bootstrap through the reference C bridge, then the private DEX engine when shared. +2. Remeasure feed latency in the representative environment. 3. Track 5 (fee LUT) only after the log-space-fees decision. Full restore archives now support file and directory application prefixes. @@ -32,25 +32,15 @@ Additional snapshot retention or transport mechanisms require a measured consume ## Track 3 — Feed & replay protocol redesign -Infrastructure subscribers download an application-defined snapshot over HTTP, -restore their application, and use one WS stream for both canonical backlog and -live inputs. The [design](2026-07-track3-feed-replay-design.md) owns the history -claims, typed refusals, resource bounds, and fresh-snapshot recovery workflow. -Raw `/inputs` and separate HTTP transaction replay are outside this feature. -The watchdog retains its independent trusted-state/L1 comparison workflow. +The current [history contract](../protocol/application-history.md) owns replica +bootstrap, history identity, replay, and recovery boundaries. The +[API contract](../../README.md#api) owns wire behavior. The wallet's cold replica +and canonical recovery/watchdog gates have a +[validation record](../review/2026-09-16-track3-validation.md). -The implemented path uses one current `application_inputs` projection for catch-up -and egress. Snapshot headers identify the same leased artifact being downloaded; -WS claims name an era, generation, and inclusive next-input count. A valid -available backlog is replayable without a total catch-up cap, with bounded pages, -queues, and subscribers. Recovery refuses old claims before delivering inputs. - -The former physical replay cursor and sparse attribution design are superseded -by the [application-history design](application-history.md). The wallet's cold -replica and canonical recovery/watchdog gates have a -[validation record](../review/2026-09-16-track3-validation.md). Native-engine -bootstrap and representative latency measurements remain integration gates; -no additional protocol layer is assumed for them. +Remaining work is native-engine integration and representative deployment +latency, tracked in the [integration plan](2026-07-track3-feed-replay-design.md). +Additional transport or retention mechanisms require a measured consumer need. ## Track 5 — Fee exponentiation LUT (deferred) @@ -70,23 +60,15 @@ until the pending log-space-fees decision lands (with Bart). ## Track 6 — Dump / `Application` API redesign -The accepted boundary keeps checkpoint creation, restore, disposal, and a pure -path to canonical comparison bytes. Creation takes `&mut self`, allowing an -adapter to flush or replace backing mappings while preserving logical state. -Checkpoints are durable before SQLite references them, immutable afterward, -and independently restorable even after source deletion. The application -prefix may be a file or directory. - -The engine owns count/clock progress and reports it by value. Successful apply -hooks advance it; the shared boundary verifies the exact successor. Keep -`Send`, remove unused `Clone + Sync`, and place canonical inspection on its -actual consumer. See the [Application contract](../protocol/application-contract.md) -for migration and the [review ledger](../review/2026-09-09-application-lane-dex-review.md) -for the accepted simplifications. - -The [July proposal](2026-07-track6-dump-api-design.md) is superseded. CoW, -flush/reopen sequencing, and working-image management belong inside an engine -adapter. Additional public primitives or asynchronous checkpoint scheduling -need a measured requirement. The DEX's private scheduler and bridge have not -been shared; conformance of the reference C bridge cannot establish theirs. -A watchdog comparison against the canonical DEX state drive is separate work. +The [Application contract](../protocol/application-contract.md) owns execution, +engine progress, and checkpoint semantics. The [C binding guide](../protocol/c-application-binding.md) +maps that contract to native engines; its reference conformance suite is +implemented. End-to-end native snapshot-to-live bootstrap remains an integration +gate, alongside the private DEX engine when available. Reference bridge +conformance cannot establish private-engine correctness. + +The [July proposal](2026-07-track6-dump-api-design.md) is historical; the +[September review](../review/2026-09-09-application-lane-dex-review.md) records the +accepted simplifications. Additional public checkpoint primitives or asynchronous +scheduling need a measured requirement. Watchdog extraction from the DEX's +canonical state drive remains separate work. diff --git a/docs/plans/2026-07-track3-feed-replay-design.md b/docs/plans/2026-07-track3-feed-replay-design.md index f4546f4..6a2700d 100644 --- a/docs/plans/2026-07-track3-feed-replay-design.md +++ b/docs/plans/2026-07-track3-feed-replay-design.md @@ -1,135 +1,33 @@ -# Feed and Replay Protocol (Track 3) +# Feed and replay integration (Track 3) -**Status: implemented in the current application-history redesign.** -The former physical-rowid feed and sparse execution mapping are superseded. -The [README](../../README.md) owns the wire contract; the -[application-history design](application-history.md) owns storage and recovery -boundaries. This document records the consumer workflow and remaining gates. +The application-history protocol is implemented. Its current contracts live in: -## 1. Consumer workflow +- [Application history and replay](../protocol/application-history.md): coordinates, + storage/recovery boundaries, snapshot bootstrap, and consumer resume. +- [README API](../../README.md#api): routes, wire messages, refusal codes, and limits. +- [Snapshot lifecycle](../snapshots/lifecycle.md): durable artifacts, accepted + comparisons, recovery exports, and leases. -1. Download `GET /latest_snapshot` using the SDK. Its tar body contains the - complete immutable restore artifact (`info.toml` and the opaque `state` - file or directory). -2. Restore the application and verify its executed-input count against - `X-Executed-Input-Count`. Keep the matching `X-History-Era` and - `X-Recovery-Generation` headers with those bytes and the restored state. -3. Subscribe with that `HistoryClaim`: mandatory `era_id`, - `recovery_generation`, and `next_input` query fields. -4. Apply each entry whose `offset` equals the application's current count. - Successful execution advances the count by one. Persist identity with the - replicated state before using it for a later resume. -5. After an ordinary disconnect, reconnect with the saved identity and actual - next-input count. On an era or generation refusal, discard the incompatible - replica and bootstrap from a current snapshot. +## Remaining integration gates -A fresh identity lookup cannot authorize old state. The SDK requires an explicit -claim on every subscription; it does not silently change identity on reconnect. -There is no separate history-version endpoint. +1. Validate the native reference adapter's snapshot-to-live replica workflow; + repeat against the private DEX bridge when available. Reference-engine + conformance does not establish private-engine correctness. +2. Measure submit-to-matching-WS-event latency in the representative deployment, + including checkpoint creation and L1 reconciliation under the supported load. -Snapshot selection, its count, history version, and GC lease share one storage -transaction. The lease lasts through response completion or disconnect. A -recovery between snapshot acquisition and subscription is handled by refusing -its old claim, without blocking history advancement during transfer. - -## 2. Coordinates and storage - -- `safe_inputs.safe_input_index` names a source L1 InputBox event. It includes - scheduler batch envelopes and direct application inputs. -- `application_inputs.offset` is an `ExecutedInputCount`: an application at - count `N` consumes entry `N` next. Every row has an offset, owner frame, and - exactly one user-op or source-L1 reference. Batch envelopes never appear. -- The immutable era baseline supplies the unavailable application prefix - `K` and accounted L1 block. Current entries occupy `[K, H)`, where `H` is - the next application count. -- Standard recovery deletes an invalidated projection suffix and advances - its generation atomically. Replacement inputs reuse those canonical - offsets. Raw L1 inputs, batches, frames, and user ops retain their source - evidence; the invalidated flattened sequence is not separately retained. -- Rebuild creates a new UUIDv4 era and a complete baseline. It does not insert - padding inputs or preserve a physical replay cursor. - -Catch-up and egress use the same named entry and coherent canonical-page -reader. Bounds, identity, and rows are read in one SQLite transaction. Missing -interior rows and invalid payload context fail loudly. Empty requests at the -head do not convert the exclusive boundary back into a SQL row coordinate. - -The latest valid frame's `safe_block`, bounded below by the era's L1 baseline, -accounts for the complete L1 prefix. No separate mutable processed-input cursor -is needed. Snapshot application count and L1 accounting are different facts; -see the application-history design for recovery's terminal drain and sparse -checkpoint availability. - -## 3. Subscription admission - -Validate history identity before position, using one coherent `(era, -generation, K, H)` read: - -| Condition | HTTP 409 policy code | Consumer action | -|---|---|---| -| Era differs | `ERA_CHANGED` | Bootstrap from a current snapshot. | -| Generation differs | `STALE_GENERATION` | Bootstrap from a current snapshot. | -| `N < K` | `HISTORY_UNAVAILABLE`, with `available_from` | Bootstrap from an available snapshot. | -| `N > H` | `AHEAD_OF_HEAD`, with `head` | Correct the invalid claim. | -| `K <= N <= H` | Upgrade to WebSocket | Replay inclusively from `N`, then follow the tip. | - -Refusals precede the upgrade and all input delivery. The JSON body is also -carried in `X-History-Error`: WebSocket libraries may stop reading a refused -handshake at its headers before the body arrives. Missing or malformed required -query fields receive HTTP 400. - -A successful stream carries the existing tagged user-op/direct-input messages -with canonical offsets and persisted context. The mandatory admission claim -binds the session identity; there is no hello frame or per-event generation. -Recovery changes history only across a process boundary, after existing -subscriptions have ended. No generation bus or farewell guarantee is needed. - -`N == H` waits normally. Every valid available backlog is replayable: there is -no total 50,000-event cap. Page size, send queue, subscriber count, and inbound -message limits remain bounded independently of backlog depth. The same durable -query handles backlog and live delivery, avoiding a separate handoff cursor. - -## 4. Snapshot and watchdog boundaries - -`/latest_snapshot` is a replica restore archive. `/finalized_state` remains the -watchdog's application comparison bytes, with its inclusion-block metadata -route. `/finalized_snapshot` exports an accepted recovery artifact and a derived -`checkpoint.toml` receipt. These are operator-infrastructure routes. - -The watchdog starts from trusted state and independently consumes L1; snapshot -bootstrap for a tip replica does not replace that trust boundary. Finalized -comparison/export is available only at a supported accepted checkpoint, not at -an invented intra-frame or arbitrary execution position. - -## 5. Acceptance evidence and remaining work - -The implementation tests cover inclusive pages and source context, exclusion -of envelopes, nonzero rebuild bases, actual suffix invalidation and replacement, -coherent SQLite snapshots during a second writer's recovery, counts beyond the -largest SQL row, and loud interior-gap detection. Feed/API/SDK tests cover -mandatory claims, typed refusals, exact-head waiting and live delivery, -50,001-entry history with bounded pages, ordinary resume, subscriber limits, -terminal storage faults, and cancelled preparation retaining process ownership. -Snapshot integration tests own artifact/header association and restore proof. -The Anvil recovery/WS gate also exercises process restart, generation refusal, -and re-drained direct replay at a reused offset. - -The cold-replica E2E restores a nonempty HTTP archive, checks it against a -genesis-fed replica, and holds catch-up behind a barrier while new writes commit. -It checks whole-state, count, and clock agreement through live direct inputs -and user operations, then exercises real stale recovery, claim refusal, and -fresh bootstrap. Canonical-machine gates cover genesis, ordinary execution, -stale recovery, and database reconstruction from an exported checkpoint. The [validation record](../review/2026-09-16-track3-validation.md) records the -pinned environment and local latency measurements. +wallet's nonempty cold bootstrap, concurrent replay/live delivery, recovery and +rebootstrap, canonical-machine gates, and local latency measurements. Those +results do not replace the consumer/environment gates above. -Remaining integration gates are concrete consumers and environments: +## Revisit only with a consumer need -- Validate the native reference adapter and, when available, the private DEX - bridge against the application contract and this bootstrap workflow. -- Remeasure submit-to-matching-WS-event latency on the representative deployment. +- Resumable snapshot transfer: when artifact size makes interrupted downloads costly. +- Retained client checkpoints: when full rebootstrap cost matters. +- Archival HTTP replay or raw L1 feeds: for an identified consumer. +- Session fencing: if history can mutate within an admitted process or multiple + local writers become supported. -Revisit resumable snapshot transfer only when artifact size requires it; -retained client checkpoints only when full rebootstrap cost matters; archival -HTTP replay only for an identified consumer. Revisit session fencing if history -can mutate within an admitted process or multiple writers become supported. +These are triggers for design work, not promised APIs. The +[coordination plan](2026-07-coordination-tracks.md) owns cross-track priorities. diff --git a/docs/plans/2026-07-track6-dump-api-design.md b/docs/plans/2026-07-track6-dump-api-design.md index 93c25c1..528ce0a 100644 --- a/docs/plans/2026-07-track6-dump-api-design.md +++ b/docs/plans/2026-07-track6-dump-api-design.md @@ -1,4 +1,4 @@ -# Dump / `Application` API — Design Draft (Track 6) +# Historical dump / `Application` proposal (Track 6) **Status: superseded by the 2026-09-09 design decision.** Preserved as the original proposal; its public clone/flush machinery was not adopted. The diff --git a/docs/plans/2026-08-authority-boundary-adr.md b/docs/plans/2026-08-authority-boundary-adr.md index 72b7896..305e73f 100644 --- a/docs/plans/2026-08-authority-boundary-adr.md +++ b/docs/plans/2026-08-authority-boundary-adr.md @@ -170,8 +170,8 @@ standard-recovery transaction iff it invalidates at least one valid batch; a clean restart changes neither. The pair is an equality/discontinuity token, not an ordered counter. Snapshot headers and mandatory WS claims expose these coordinates. Every application row has its pre-execution count; recovery replaces only the current -suffix. See the [history contract](application-history.md) and -[Track 3 handoff](2026-07-track3-feed-replay-design.md). +suffix. See the [history contract](../protocol/application-history.md) and +[remaining integration gates](2026-07-track3-feed-replay-design.md). ## Performance posture diff --git a/docs/plans/application-history.md b/docs/plans/application-history.md deleted file mode 100644 index e7b935f..0000000 --- a/docs/plans/application-history.md +++ /dev/null @@ -1,92 +0,0 @@ -# Application history and checkpoints - -The sequencer keeps L1 observations, application ordering, and batch acceptance -as separate durable facts. L1 inputs include batch envelopes; application -history contains only included user operations and external direct inputs. -Source references provide provenance without defining a mapping between the -two timelines. - -## History - -`application_inputs` is the current application sequence. Its primary key is -the input's pre-execution `ExecutedInputCount`; each row belongs to a local -batch/frame and references either its user operation or its source L1 input. -Every row executes. Payloads remain in their source tables. - -The latest surviving frame's `safe_block` records complete L1 accounting. -Reconciliation covers the whole newly safe interval before committing its new -frame and application inputs. Full-block ingestion and indivisible range -reconciliation make a separate mutable processing cursor unnecessary. Empty -intervals and intervals containing only batch envelopes advance this boundary -without adding application inputs. - -Recovery invalidates a batch suffix, removes its current application rows, -advances the history generation, and opens the replacement tip atomically. -Replacement inputs reuse suffix offsets under the new generation. Original -L1, batch, frame, and user-operation records remain available for diagnostics. - -## Era baseline - -Setup registers a complete baseline after its artifact is durable: application -count `K`, accounted L1 stop block `C`, starting batch nonce, and history identity. -The recovered L1 prefix through `C` is opaque to ordinary operation. Both direct -ordering and accepted-batch scanning begin after it. The baseline metadata -survives root invalidation and artifact garbage collection. - -The recovery fold drains queued directs through `C`, including young directs -that the canonical scheduler has not executed yet. Its output is a restart -baseline, without a claim that its bytes equal canonical state at block `C`. -Genesis supplies the trusted block-zero comparison state. - -## Snapshots and acceptance - -The lane creates a durable snapshot at every batch close. Snapshot registration -and batch sealing commit together. Snapshots reference immutable local batch -identities; a nonce can be reused by recovery. The baseline is a separate -snapshot origin. - -Acceptance is derived from complete safe L1 observations, the scheduler's -acceptance rules, and byte identity with the local sealed batch. An accepted -batch confirms existing application history and adds no replay entry. - -Checkpoint selection uses these facts directly: - -- Restart and replica bootstrap use the newest surviving batch snapshot, or - the baseline. -- Recovery requires a retained accepted snapshot, or the baseline before the - first post-baseline acceptance. -- The watchdog compares an accepted checkpoint at the end of its L1 inclusion - block. Per-batch snapshots make the latest accepted batch's artifact available. - -Select the required accepted batch before loading its snapshot: a missing -required artifact is an invariant violation, never permission to choose an -older checkpoint. Divergence blocks publication of a newly derived comparison. - -There is no snapshot promotion mutation. Retention keeps the newest accepted -snapshot (or baseline), all valid snapshots beyond the accepted frontier, and -leased artifacts. The baseline bytes can be retired once an accepted artifact -provides the recovery fallback. Artifact creation precedes DB publication; -DB retirement precedes filesystem deletion. - -A portable accepted checkpoint includes the application artifact and coherent -sequencer metadata identifying its canonical comparison point and resume nonce. -Acceptance metadata is derived at export; application artifacts stay immutable. -Sparse snapshot creation and intra-block watchdog checkpoints are separate work. - -## Replay and egress - -Restart and egress share application-only pages beginning at an inclusive input -count. Snapshot bootstrap uses HTTP; one WS stream replays available history -then follows the tip. Subscription claims include era, generation, and next -input count. Wrong identity or unavailable history requires bootstrap; a claim -at the head waits and one beyond it fails. Pages and queues are bounded, while -total replay has no arbitrary catch-up cap. - -## Validation boundaries - -Exercise prefix exclusion for previously rejected future-nonce batches; -baseline-only restart and repeated root invalidation; envelopes-only frame -advancement; acceptance observed during downtime; empty accepted batches; -snapshot retirement with active leases; atomic suffix replacement; and cold -replica restore followed by canonical replay. The recovery models constrain -admission and batch safety, not the concrete snapshot/GC implementation. diff --git a/docs/protocol/application-contract.md b/docs/protocol/application-contract.md index 40f0c8f..536a69a 100644 --- a/docs/protocol/application-contract.md +++ b/docs/protocol/application-contract.md @@ -34,6 +34,8 @@ Before an apply hook, the boundary computes the checked expected successor. After `Ok`, it asserts that the engine reports exactly that successor and returns the input's pre-execution count as its history offset. An engine may use `ApplicationProgress::advance` or implement the same transition natively. +Adapters report the engine-owned progress rather than maintaining a separate +count or clock mirror. Overflow fails before the hook runs. An error defines no successor: callers terminate the execution path and discard the instance, without attempting to roll back or inspect partially updated state. @@ -56,6 +58,9 @@ live execution, canonical execution, and replay. State changes happen through apply hooks; dump creation may change backing resources but preserves logical state. +Wrapping an existing engine must preserve its transaction encoding, +rejection/inclusion semantics, and canonical state bytes. + `Application: Send + Sized` permits moving the engine to the lane's blocking worker. It requires neither `Sync` nor `Clone`: the lane owns one mutable engine, and an independent state fork is a fallible checkpoint/restore @@ -117,7 +122,8 @@ execution offsets, checked during catch-up. HTTP snapshot metadata and mandatory WS claims carry the history version and this count. Both restart and subscriber replay read the same current application -sequence; see the [API contract](../../README.md). +sequence. The [history guide](application-history.md) owns coordinates and +replica bootstrap; the [API contract](../../README.md) owns wire shapes. ### 5. Operational capacity for L1 reconciliation @@ -138,8 +144,18 @@ or measured checkpoint latency demonstrates the need. A **recovery checkpoint** contains everything needed to resume the engine. A **canonical comparison file** contains the deterministic state the watchdog compares against the canonical application. They may be the same file; a -machine checkpoint may instead contain a separate app-state projection. The -[format contract](../snapshots/format.md) describes three relevant layouts. +machine checkpoint may instead contain a separate app-state projection. + +| Engine | Recovery checkpoint | Canonical comparison file | +|---|---|---| +| Wallet | SSZ wallet state | The same SSZ file, also returned by canonical inspect | +| Cartesi Machine wrapper | Full multi-file machine state | Deterministic app-state projection stored alongside it | +| Native DEX design | Fixed-memory state `M` plus required resumable metadata | Canonical `M`, matching the designated drive in the canonical machine | + +The DEX row describes an integration requirement, not a verified private +implementation. The [watchdog guide](../watchdog/README.md) owns comparison +transport and support; the [wallet format](../snapshots/format.md) owns its SSZ +representation. - `create_dump(&mut self, prefix)` creates a checkpoint at an absent path, which may become a file or directory. On `Ok`, all files and directory @@ -177,26 +193,3 @@ implementation. `CanonicalState::canonical_snapshot_bytes` is a separate inspection trait required by the shared Rust scheduler's inspection method and canonical harness, not by the native sequencer. Human-readable debugging state also stays on the concrete application. - -## Adapter migration - -1. Remove the capability parameters and mutable progress accessor. Return the - native count/clock pair from `progress()` without a Rust-side mirror. -2. Advance both fields inside each successful native apply transition, - including no-ops. Keep validation pure and map its fatal failures to - `AppError`, with expected rejection as `Ok(ValidationOutcome::Reject(...))`. -3. Change checkpoint creation to `&mut self` and establish durable, immutable - checkpoints with independent restores. Preserve existing canonical bytes. -4. Implement `CanonicalState` only where canonical inspection needs it. - Remove any `Clone` or `Sync` added solely to satisfy the old host bounds; - justify `Send` against the native engine's ownership contract. -5. Return the stable payload bound from `max_method_payload_bytes()` rather - than an associated constant. Keep checkpoint artifacts self-contained so - recursive filesystem deletion disposes of them without an application hook. - -Changing these Rust interfaces preserves transaction encoding, expected -rejection semantics, snapshot bytes, scheduler ordering, and the database -schema. Fatal `AppError` propagation is an intentional exception: validation -and execution failures discard the engine rather than becoming a rejection -or an included no-op. The host's terminal-versus-retryable classification -still applies. diff --git a/docs/protocol/application-history.md b/docs/protocol/application-history.md new file mode 100644 index 0000000..2023241 --- /dev/null +++ b/docs/protocol/application-history.md @@ -0,0 +1,150 @@ +# Application history and replay + +Application history is the sequencer's current execution order: included user +operations and external direct inputs. It contains an optimistic suffix that +recovery may replace. A stable offset identifies an input only together with +its history version; receiving it does not establish L1 acceptance. + +This document owns history coordinates, recovery boundaries, and replica +bootstrap. The [Application contract](application-contract.md) owns execution +and progress; the [snapshot lifecycle](../snapshots/lifecycle.md) owns artifact +publication, selection, and retention; the [README API](../../README.md#api) +owns routes, wire fields, and refusal codes. + +## Progress, ordering, and acceptance + +These facts answer different questions: + +| Fact | Meaning | +|---|---| +| `ExecutedInputCount` | Number of application inputs already executed. At count `N`, entry `N` executes next. | +| `ApplicationProgress.last_executed_safe_block` | Maximum clock carried by an executed input: frame safe block for user ops, inclusion block for directs. | +| Latest surviving frame's `safe_block` | Complete L1 interval accounted for by local ordering, bounded below by the era baseline. | +| `safe_accepted_batches` | Safe L1 landings accepted by the scheduler rules and matched to local sealed bytes. | + +A frame can account for an empty interval or only batch envelopes, advancing +L1 accounting without executing an application input. An accepted batch can +confirm existing execution without adding an input. Empty batches can share +an application count. Neither the count nor the app clock substitutes for the +L1 accounting boundary or acceptance facts. + +`safe_inputs` retains all InputBox observations, including batch envelopes. +`application_inputs` contains only the current ordered application sequence; +each row has a mandatory pre-execution offset, an owning batch/frame, and +exactly one reference to a user op or external direct input. Payloads remain +in those source tables. Included business failures and malformed-direct no-ops +advance the count; validation rejections and envelopes do not. + +Restart and egress read this same sequence. Replay executes stored valid +user ops with their recorded fee and frame clock, without revalidation; +directs use their original inclusion block. Timestamps and transaction hashes +are provenance, not extra application-transition inputs. The +[execution contract](application-contract.md#the-execution-methods) defines the +shared execution boundary. + +## Identity and available history + +A `HistoryClaim` combines: + +- **Era**: a UUIDv4 created by setup/rebuild, identifying one local history. +- **Recovery generation**: a revision within that era, advanced atomically + whenever automatic recovery invalidates at least one valid batch. +- **Next input**: the application's actual executed-input count. + +If the era baseline count is `K` and the current head is `H`, available entries +occupy `[K, H)`. A claim at `H` waits for future entries; a claim below `K` or +above `H` is refused. Identity is checked before position. Equal counts cannot +authorize resuming a different era or generation, even if the consumer believes +its state precedes the replaced suffix. + +Automatic recovery invalidates a batch suffix, removes its current application +rows, advances the generation, and opens the replacement Tip in one transaction. +Replacement inputs reuse suffix offsets under the new generation. Original L1, +batch, frame, and user-op source records remain; the invalidated flattened +sequence is not separately retained. A repair that invalidates nothing leaves +the generation unchanged. The [recovery guide](../recovery/README.md) owns +repair selection and guards. + +### Era baseline + +Setup publishes a complete baseline only after its artifact is durable: +application count `K`, accounted L1 stop block `C`, starting batch nonce, and +history identity. Ordinary direct ordering and accepted-batch scanning begin +after `C`. The baseline's metadata survives root invalidation and artifact GC; +no padding inputs or separate mutable processing cursor represent its prefix. + +Cockroach recovery drains queued directs through `C`, including young directs +that the canonical scheduler has not executed yet. Its output is a stable +restart baseline; its bytes need not equal canonical state at block `C`. +A later accepted batch snapshot supplies the first comparison checkpoint in that era. +Genesis supplies the trusted block-zero comparison state. The +[rebuild guide](../recovery/cockroach.md) owns checkpoint requirements and the +fixed stopping boundary. + +## Three consumers of checkpoints + +| Consumer | Starting point and continuation | +|---|---| +| Native restart | Load the newest surviving batch snapshot, or baseline, check the engine's count against its row, then replay current application inputs. | +| Sequencer replica | Download `/latest_snapshot`, restore its application state, and subscribe using the matching history claim. This follows optimistic execution. | +| Watchdog | Start from independently trusted canonical machine state and replay L1. Compare at the sequencer's accepted checkpoint; the replica feed does not establish independent trust. | + +A batch-close snapshot is identified by its local batch identity, not just its +count or nonce. Recovery can reuse a nonce and empty batches can repeat a count. +Acceptance derives the comparison point without modifying the artifact. +Selection must first choose the required accepted batch and then require its +snapshot; a missing one cannot justify falling back to an older comparison. +The [snapshot lifecycle](../snapshots/lifecycle.md#acceptance-and-comparison) +explains block-boundary comparison and rollback retention. The +[watchdog guide](../watchdog/README.md) explains independent verification. + +## Replica bootstrap and resume + +1. Download `GET /latest_snapshot`. The tar archive contains `info.toml` and + the complete opaque application `state` file or directory. +2. Retain its `X-History-Era`, `X-Recovery-Generation`, and + `X-Executed-Input-Count` headers with those bytes. Restore the application and + verify its count against that header. The archive alone does not carry the + complete subscription claim. +3. Subscribe with the matching era, generation, and next-input count. Snapshot + selection, headers, and lease share one transaction; recovery during the + download can still invalidate the claim before subscription. Rebootstrap + if the server refuses that old identity. +4. Require each entry's offset to equal the application's current count, then + execute it through the shared execution boundary. Successful application + advances the count by one. Persist the history identity with the replica's + state so that a later resume cannot combine different histories. +5. After an ordinary disconnect, reconnect with that saved identity and the + actual count. A history mismatch or unavailable prefix requires a current + snapshot. A count ahead of the server's head is an invalid claim to correct. + +The [Rust SDK](../../sdk/rust-client/src/lib.rs) returns a `HistoryClaim` with +its snapshot response and requires an explicit claim for subscriptions. The +consumer owns restore, persistence, and reconnect. Fetching fresh identity +metadata cannot authorize old application state. + +One durable page reader handles both backlog and live delivery, so there is no +separate cursor to switch at the live boundary. Each page reads identity, +bounds, and rows in one SQLite transaction. Interior gaps and invalid source +context fail loudly. Page size and send queues bound memory; available backlog +has no total replay cap. The [API contract](../../README.md#api) owns exact +resource limits and handshake errors. + +History replacement happens across a process boundary, after existing +subscriptions end. A session's mandatory claim therefore binds all its events; +there is no per-event generation or guaranteed farewell message. Revisit this +assumption if history can change within an admitted process or multiple local +writers become supported. + +## Code and validation map + +| Boundary | Code and tests | +|---|---| +| Coordinates and claim ordering | [`sequencer-core/src/history.rs`](../../sequencer-core/src/history.rs) — identity before position, inclusive head, checked counts. | +| Coherent application pages | [`storage/egress/canonical.rs`](../../sequencer/src/storage/egress/canonical.rs) and its tests — source context, nonzero baselines, replacement offsets, concurrent recovery, gaps and SQL limits. | +| Replay followed by live delivery | [`l2_tx_feed`](../../sequencer/src/egress/l2_tx_feed/) — bounded deep replay, identity refusals, shutdown and persistent faults; [`catch_up.rs`](../../sequencer/src/ingress/inclusion_lane/catch_up.rs) for native replay. | +| Artifact and claim association | [`snapshot_endpoints.rs`](../../sequencer/src/integration_tests/snapshot_endpoints.rs) — headers, restore, archive contents, and lease lifetime. | + +The [integration validation record](../review/2026-09-16-track3-validation.md) +records wallet replica and canonical-machine evidence. Remaining consumer and +deployment gates belong to the [Track 3 plan](../plans/2026-07-track3-feed-replay-design.md). diff --git a/docs/protocol/scheduler-semantics.md b/docs/protocol/scheduler-semantics.md index 4d1550a..8b5cb81 100644 --- a/docs/protocol/scheduler-semantics.md +++ b/docs/protocol/scheduler-semantics.md @@ -181,7 +181,7 @@ history is no longer locally available. The durable `application_inputs` sequence uses these same offsets. HTTP snapshots carry the history version and application count; WS subscriptions -must claim both before inclusive replay. See the [history contract](../plans/application-history.md). +must claim both before inclusive replay. See the [history contract](application-history.md). --- diff --git a/docs/recovery/cockroach.md b/docs/recovery/cockroach.md index 0016e26..ff588f7 100644 --- a/docs/recovery/cockroach.md +++ b/docs/recovery/cockroach.md @@ -123,7 +123,7 @@ orphan artifact. Identity pinning and raw L1 ingestion may survive an incomplete attempt, but the lock and setup admission prevent serving a partial baseline. `C` remains the fallback reconciliation boundary if standard recovery invalidates -the root. The [history contract](../plans/application-history.md#era-baseline) +the root. The [history contract](../protocol/application-history.md#era-baseline) owns these immutable coordinates; [snapshot lifecycle](../snapshots/lifecycle.md) owns restore selection, rollback-safe retention, and eventual baseline disposal. diff --git a/docs/review/2026-09-09-application-lane-dex-review.md b/docs/review/2026-09-09-application-lane-dex-review.md index 9009cd3..a4bfa34 100644 --- a/docs/review/2026-09-09-application-lane-dex-review.md +++ b/docs/review/2026-09-09-application-lane-dex-review.md @@ -27,7 +27,7 @@ cherry-picked into the main implementation. The private DEX scheduler and native engine are still unavailable. Reference bridge tests verify the proposed seam, not private engine conformance. DEX -conformance and the [Track 3 history API](../plans/2026-07-track3-feed-replay-design.md#7-ordered-implementation-handoff) +conformance and the [Track 3 history API](../plans/2026-07-track3-feed-replay-design.md) remain follow-ups. This review establishes reference integration coverage; it does not establish that the Application surface is production-proven. diff --git a/docs/review/register.md b/docs/review/register.md index fadd389..8942aa8 100644 --- a/docs/review/register.md +++ b/docs/review/register.md @@ -53,7 +53,7 @@ remaining dated ledgers stay valid. 7. **Closed** (2026-09-16): mandatory era/generation/application-count claims reject resume across recovery; snapshot headers provide cold-bootstrap coordinates. Current application suffix replacement is atomic with the - generation bump. See the [Track 3 contract](../plans/2026-07-track3-feed-replay-design.md). + generation bump. See the [history contract](../protocol/application-history.md). 8. **Fee-determinism contract under-specified** — the LSB-first floor-after-each-multiply order is implemented but not stated as contract (`sequencer-core/src/fee.rs`). Load-bearing for the C++ scheduler port; @@ -291,7 +291,7 @@ Each entry: the decision, its reason, and where the reasoning now lives. versioned claims replace the mixed replay log and sparse mapping. Acceptance facts select immutable per-batch snapshots without promotion or restamping; per-batch cadence and end-of-block watchdog comparison remain. The complete - model lives in [application history](../plans/application-history.md), I5–I11, + model lives in [application history](../protocol/application-history.md), I5–I11, I18/I20, and the snapshot lifecycle. - **No architectural restructure** (2026-06-10): one file per writer role, @@ -705,5 +705,5 @@ for `2026-06-10-correctness-review.md`, `2026-06-10-simplification.md`, | 2026-09-07 | PR #28 premise review and maintainer-approved simplification | Ordered recovery replaces the phase driver; diagnosed terminal runtime faults abort immediately; ordinary shutdown and snapshot leases remain; reader drain race and duplicate offset check fixed | Current ADR and recovery design; finding 33 and settled decisions above. Validation: 692 host tests, seven targeted restart/outage E2Es, workspace check, strict Clippy, formatting, and admission TLC passed. The broader stale-batch recovery E2E reached its watchdog comparison but was blocked by the host Lua emulator 0.21 loading the pinned 0.20 image (archive version mismatch); no protocol pin was changed. | | 2026-09-09 | Application, inclusion lane, and public DEX integration branch | Native progress ownership, typed validation failures, mutable independent checkpoints, optional canonical inspection, and lane bookkeeping simplified; ingress CORS and Lua 5.4 parity restored. Reference C bridge port kept separate. | [Application/lane review](2026-09-09-application-lane-dex-review.md); current Application and snapshot contracts. Workspace check, strict Clippy, 697 host tests, and 62 watchdog tests passed; private DEX conformance remains unverified. | | 2026-09-11 | Reference C bridge port and review boundary | Keep the current Application contract, runtime payload bound, paired progress, filesystem-owned checkpoint disposal, and host failure fixes together. Engine-dependent API refinements and external-engine conformance remain follow-ups. | Settled decisions and owed tests above; [C binding guide](../protocol/c-application-binding.md), Application contract, and snapshot lifecycle. | -| 2026-09-16 | Application-history and Track 3 implementation, with independent storage/recovery/snapshot review | Replaced mixed replay and sparse attribution with current application inputs; complete atomic baselines; acceptance-derived immutable snapshots; HTTP restore/recovery archives and mandatory WS claims. Review narrowed equal-block recovery to empty genesis to avoid losing same-block pending directs. | [History design](../plans/application-history.md), current invariants/API/snapshot/recovery docs. Validation: 693 workspace tests, strict Clippy, formatting, 62 watchdog tests, admission TLC (155 distinct states), and the Anvil recovery/old-claim refusal gate passed. Full canonical watchdog comparison remains blocked by the local emulator 0.21 vs repository 0.20 environment; no pin changed. | +| 2026-09-16 | Application-history and Track 3 implementation, with independent storage/recovery/snapshot review | Replaced mixed replay and sparse attribution with current application inputs; complete atomic baselines; acceptance-derived immutable snapshots; HTTP restore/recovery archives and mandatory WS claims. Review narrowed equal-block recovery to empty genesis to avoid losing same-block pending directs. | [History design](../protocol/application-history.md), current invariants/API/snapshot/recovery docs. Validation: 693 workspace tests, strict Clippy, formatting, 62 watchdog tests, admission TLC (155 distinct states), and the Anvil recovery/old-claim refusal gate passed. Full canonical watchdog comparison remains blocked by the local emulator 0.21 vs repository 0.20 environment; no pin changed. | | 2026-09-16 | Track 3 integration validation | Nonempty HTTP cold replica, concurrent backlog/live consumption, stale recovery/rebootstrap, and four real canonical-machine gates pass under emulator 0.20. Tooling fixes preserve Lua paths and make benchmark fee defaults admissible. | [Validation and latency evidence](2026-09-16-track3-validation.md); native bridge/DEX and representative deployment latency remain separate gates. | diff --git a/docs/snapshots/README.md b/docs/snapshots/README.md index 5b983d3..63eb0de 100644 --- a/docs/snapshots/README.md +++ b/docs/snapshots/README.md @@ -5,18 +5,24 @@ known execution boundary. They let the inclusion lane resume with load and replay. A snapshot may contain optimistic state; L1 acceptance determines which artifact can back a canonical comparison or recovery export. -Two documents, split by concern: +Keep three boundaries separate: the engine embeds its executed-input count and +safe-block clock; SQLite associates an artifact with a local batch or era +baseline; safe L1 acceptance selects a comparison checkpoint. Counts and clocks +alone do not establish acceptance. The +[history guide](../protocol/application-history.md) explains these coordinates +and snapshot-plus-replay bootstrap. -- **[`format.md`](format.md)** — the on-disk *format*: the `Application` dump - trait (`from_dump` / `create_dump` / `state_file_in_dump`) and - the toy wallet's SSZ wire encoding. What a dump *is*. - -- **[`lifecycle.md`](lifecycle.md)** — creation at batch close, restart selection, +- [Application contract](../protocol/application-contract.md#6-checkpoint-lifecycle) + — engine dump methods, durability, immutable artifacts, independent restore, + and the canonical comparison file. +- [Wallet format](format.md) — the wallet's layout, deterministic SSZ encoding, + and decode rules. +- [Lifecycle](lifecycle.md) — creation at batch close, restart selection, acceptance-derived comparison checkpoints, recovery exports, retention, download leases, and crash safety. Acceptance is a separate durable fact; artifacts are never promoted or rewritten. -For automatic startup repair, see [standard recovery](../recovery/README.md). +For automatic startup repair, see [automatic recovery](../recovery/README.md). For rebuilding after database loss or a sequencer bug, see [cockroach recovery](../recovery/cockroach.md). The root [README](../../README.md) owns endpoint shapes. diff --git a/docs/snapshots/format.md b/docs/snapshots/format.md index 963a532..68f409f 100644 --- a/docs/snapshots/format.md +++ b/docs/snapshots/format.md @@ -1,81 +1,11 @@ -# App Snapshot Format (Wallet Toy App) +# Wallet snapshot format -This document defines the on-disk snapshot format produced by the toy wallet -app in `examples/app-core` via the `Application` trait's dump methods. - -## Scope - -This document covers two things: - -1. The trait shape that any `Application` implementation must satisfy to - participate in snapshot lifecycle (`from_dump`, `create_dump`, - `state_file_in_dump`). -2. The wire format the toy wallet uses to encode its canonical state into - the dump's state file. Checkpoint ownership and durability are defined by -the [Application contract](../protocol/application-contract.md#6-checkpoint-lifecycle). - -It does NOT define when snapshots are triggered, how the inclusion lane -records and selects them, how the HTTP layer serves them, or recovery -interactions. Those are layered above the trait and live in their own -modules. - -## Trait Surface - -```rust -trait Application: Send + Sized { - // ... other methods ... - - fn from_dump(prefix: &Path) -> Result; - fn create_dump(&mut self, prefix: &Path) -> Result<(), AppError>; - fn state_file_in_dump(prefix: &Path) -> PathBuf; -} -``` - -Contract: - -- `prefix` is an opaque app-owned path, which may be a file or directory. - All checkpoint-owned artifacts reside at or below it. The sequencer owns the - enclosing dump directory and its `info.toml`, and disposes of the checkpoint - by removing that directory recursively. No other resource cleanup is needed. -- `create_dump` creates the absent `prefix` and makes the complete checkpoint - durable before returning. It may replace backing resources but preserves - logical state. Later execution cannot alter a checkpoint. Restored engines - are independent of one another and remain usable after source deletion. -- `state_file_in_dump` is a pure function of `prefix`: callers may - compute it without loading the dump or instantiating the Application. - Each impl pins its own layout convention. -- The bytes at `state_file_in_dump(prefix)` are the canonical state — - the bytes a watchdog running an independent canonical machine would - produce for the same logical state through inspect or a designated state - drive. - They must be deterministic: identical logical state must produce - byte-identical files across runs, hosts, and toolchains. -- For implementations whose persistence representation IS the canonical - state (the toy wallet), `create_dump` writes the - same bytes that `state_file_in_dump` names — a single file with no - duplication. For implementations whose persistence is richer than the - canonical state (e.g. a Cartesi Machine wrapping app), `create_dump` - writes the full machine state alongside a separate canonical-state file - under the same prefix. - -The recovery checkpoint and canonical comparison representation differ by app: - -| Engine | Recovery checkpoint | Canonical comparison file | -|---|---|---| -| Toy wallet | SSZ wallet state | The same SSZ file, also returned by canonical inspect | -| Cartesi Machine wrapper | Full multi-file machine state | Deterministic app-state projection stored alongside it | -| Native DEX design | Fixed-memory state `M` plus any required resumable metadata | Canonical `M`, matching the designated drive in the canonical machine | - -The DEX row describes the integration requirement, not a verified private -implementation. The current watchdog reads canonical inspect output; direct -comparison against a canonical drive remains separate watchdog work. A native -adapter does not need to implement the Rust canonical inspection trait to serve -its checkpoint's comparison file. - -CoW is an implementation choice within checkpoint creation and restore. Shared -physical extents are compatible with the contract; shared mutable bytes are -not. The sequencer does not prescribe a filesystem clone primitive or expose -engine flush/reopen steps. +This document owns the wallet's application-state bytes, implemented by +[`wallet_snapshot.rs`](../../examples/app-core/src/wallet_snapshot.rs). +The [Application contract](../protocol/application-contract.md#6-checkpoint-lifecycle) +owns the dump methods, durability, and independent restoration; +[lifecycle.md](lifecycle.md) owns the enclosing artifact, metadata, selection, +and retention. ## Toy Wallet Layout @@ -88,6 +18,13 @@ and its canonical state coincide; one write per `create_dump`. state SSZ-encoded WalletSnapshot bytes ``` +Here `prefix` is the app-owned `state` directory inside the sequencer's dump +directory. Thus the wallet file is `dumps//state/state`. The surrounding +`info.toml` and exported `checkpoint.toml` use the sequencer's metadata format +version; that version does not identify the wallet's SSZ schema. Application +progress is embedded in the wallet bytes. History identity and acceptance +metadata come from the sequencer; see [application history](../protocol/application-history.md). + ## Toy Wallet Wire Format - **Encoding**: SSZ @@ -114,9 +51,6 @@ and its canonical state coincide; one write per `create_dump`. reflects, so it must live in the canonical state bytes (both the bare-metal and canonical-machine sides advance it identically). -`last_executed_safe_block` was added before any environment was deployed; there -is a single, unversioned schema today (see [Versioning](#versioning)). - ### Determinism `WalletApp` stores balances and nonces in `HashMap`s, so iteration order @@ -137,21 +71,16 @@ The decoder rejects: - Malformed SSZ bytes (any decode error from the SSZ library). - A snapshot containing two entries in `balances` with the same address. - A snapshot containing two entries in `nonces` with the same address. +- Zero `executed_input_count` with a nonzero `last_executed_safe_block`. -The duplicate-address checks exist to keep the encoded bytes canonical: -without them, multiple distinct byte sequences could decode to the same -logical state (the second entry would silently overwrite the first), -breaking the property that watchdog-side and sequencer-side bytes are -comparable. +Duplicate checks prevent an entry from silently overwriting another during +restore. The decoder accepts unique entries in any order; encoding the restored +state sorts them. Deterministic emitted bytes do not imply that the decoder +accepts only that ordering. ## Versioning -There is a single, unversioned schema: `WalletSnapshot`. The encoded bytes carry -no leading version tag, and — because there is no backward-compatibility -requirement yet (no long-lived deployment whose dumps a newer binary must read) — -the struct name carries no version suffix either. An earlier draft distinguished -a `V1`/`V2` pair (the `last_executed_safe_block` field was added before any -environment existed); that split was collapsed since no `V1` dumps ever survived. +There is one SSZ schema, `WalletSnapshot`, with no leading version tag. If a future change ever needs to break the wire format against live dumps: @@ -165,23 +94,7 @@ Until then, do not reorder, repurpose, or reinterpret existing fields in place. ## Trust Model -The dump file is part of the sequencer's persistent data directory and -shares its trust boundary. An attacker with write access to the data -directory has already won; no integrity tag, checksum, or HMAC is -included on the snapshot bytes for this reason. Consumers that obtain -the bytes via a less trusted channel (e.g. a future peer-to-peer -distribution mechanism) would need to add an outer integrity layer; the -format itself does not provide one. - -## Out of Scope - -This document deliberately does not define: - -- When the inclusion lane decides to take a snapshot. -- How dumps are registered, selected by acceptance, or - garbage-collected. -- The on-the-wire archive format for streaming a dump over HTTP. -- Inspect-state procedures on other implementations (Cartesi Machine, - bare-metal DEX). -- Cross-implementation determinism test vectors (will land when a - second implementation of the wallet exists to validate against). +The file shares the persistent data directory's +[trust boundary](../threat-model/README.md). Its bytes contain no integrity tag, +checksum, or HMAC. Distribution through a less trusted channel would require +an outer integrity mechanism. diff --git a/docs/watchdog/README.md b/docs/watchdog/README.md index 23f9036..2e34627 100644 --- a/docs/watchdog/README.md +++ b/docs/watchdog/README.md @@ -1,8 +1,15 @@ # Watchdog -The watchdog is an off-chain safety process that compares the sequencer's -**finalized SSZ state dump** against state produced by the canonical Cartesi -Machine at the same L1 inclusion block. +The watchdog independently replays L1 inputs in the canonical Cartesi Machine +and compares its application-state bytes with the sequencer's accepted +checkpoint at the same L1 block boundary. The wallet's comparison format is SSZ; +the watchdog itself only compares bytes. + +The `/finalized_state` name refers to the sequencer's latest **safe, accepted +batch checkpoint**, not Ethereum's `finalized` tag or a state the watchdog has +already verified. [Snapshot lifecycle](../snapshots/lifecycle.md#acceptance-and-comparison) +owns checkpoint selection and why that application state is comparable at a +whole L1 block boundary. ## Documentation @@ -12,6 +19,7 @@ Machine at the same L1 inclusion block. | **[`getting-started.md`](getting-started.md)** | **Local dev only** — Anvil + `sequencer-devnet`, harness smoke, two-terminal flow | | This file | Architecture, modules, runtime contract, checkpoints, test commands | | [`staging-drills.md`](staging-drills.md) | Webhook smoke, synthetic alarms, staging compare daemon | +| [`design-notes.md`](design-notes.md) | Detection boundaries and checkpoint crash model | | [`sepolia.md`](sepolia.md) | Redirect → [`operator-deployment.md`](operator-deployment.md) | ### Quick start (pick your environment) @@ -45,6 +53,143 @@ overlapping ticks Details: **[`getting-started.md`](getting-started.md)**. +## Runtime Contract + +The watchdog consumes two operator-internal routes: + +- `GET /finalized_state/inclusion_block` — cheap JSON `{ inclusion_block, executed_input_count }` polled every compare tick. +- `GET /finalized_state` — streams the comparison file (`application/octet-stream`); the watchdog reads its `X-Inclusion-Block` and `X-Executed-Input-Count` headers. + +The sequencer's genesis baseline is immediately comparable. A rebuilt baseline +is a restore artifact; these routes return 404 until a new accepted batch +provides a comparison checkpoint. The [snapshot lifecycle](../snapshots/lifecycle.md) +owns availability, response metadata, and artifact leases. + +**Positioning is by L1 block.** If `inclusion_block` equals the watchdog +checkpoint's `safe_block`, the tick exits idle: no state download, L1 fetch, or +CM work. A lower block is a terminal `inclusion_block_regressed` event. For a +higher block, the watchdog replays every InputBox input from `safe_block + 1` +through that block, then downloads the comparison file. If its block header differs from the +polled target, the tick retries rather than comparing different boundaries. + +The client parses `executed_input_count` but does not use it as a replay cursor +or compare it between responses. It does not consume history era/generation +headers or WebSocket events. Its independently replayed L1 checkpoint is +separate from the snapshot-plus-feed protocol described in +[Application history](../protocol/application-history.md). + +The CM's `state` inspect query must return exactly one report. The watchdog +compares those bytes directly with the downloaded file, without decoding or +canonicalizing either side. + +For the toy wallet app, SSZ encoding lives in `examples/app-core/src/wallet_snapshot.rs` +and is shared by `WalletApp::create_dump`, `CanonicalState::canonical_snapshot_bytes`, +and the canonical scheduler's `Inspect` handler (`examples/canonical-app`). + +## Checkpoints + +V1 persists the whole Cartesi Machine checkpoint, including scheduler state; +it does not persist fetched L1 inputs. This is different from the sequencer's +app-owned restore archives (`/latest_snapshot` and `/finalized_snapshot`) and +from its comparison file (`/finalized_state`). The watchdog downloads neither +archive. + +`manifest.json` records `safe_block` (the L1 block through which the CM has +consumed all inputs), timestamp, and optionally the CM image hash. A new +checkpoint directory is written first, then `head.json` is atomically replaced +to point at it. [Design notes](design-notes.md#checkpoint-crash-model) own the +state layout, best-effort pruning, and crash guarantees. + +`init` stores a trusted operator-provided CM snapshot and its declared block +into this layout. That block may precede the sequencer's current comparison +target: `tick` replays the intervening inputs. `init` does not compare against +the sequencer, and a first tick at the same block exits idle; successful init +or idle is not evidence of a state comparison. See the +[accepted detection boundary](design-notes.md#watchdog-state). + +`tick` requires both `config.json` and `head.json`; it never bootstraps from env. +`CARTESI_WATCHDOG_BLOCKCHAIN_HTTP_ENDPOINT` is not persisted in `config.json`, so +operators can rotate RPC endpoints without rewriting watchdog state. It is +required at `tick` for L1 reads, and optionally present at `init` when +auto-detecting `CARTESI_WATCHDOG_BLOCKCHAIN_ID` via `eth_chainId` (prefer setting +the chain id explicitly). + +Bootstrap inputs read by `init`: + +- `CARTESI_WATCHDOG_CM_SNAPSHOT_DIR` +- `CARTESI_WATCHDOG_CM_SNAPSHOT_SAFE_BLOCK` + +## How it runs + +The watchdog has two subcommands: + +```bash +sequencer-watchdog init # setup: writes config.json + head.json (idempotent if complete) +sequencer-watchdog tick # one compare cycle; schedule this +``` + +`tick` does one cycle per process, then exits — infra schedules re-runs +(systemd timer / k8s CronJob) and reacts to the exit code. There is no daemon +loop. `sequencer-watchdog` takes a non-blocking `flock` for `init`/`tick`; +host scheduling should provide the same non-overlap guarantee. A tick follows +the [runtime contract](#runtime-contract), writes a checkpoint only after a +successful comparison, and emits `watchdog_event` on mismatch or regression. +It atomically writes `status.prom` before exit. + +Runtime knobs: + +- `CARTESI_WATCHDOG_BLOCKCHAIN_HTTP_ENDPOINT`: current L1 JSON-RPC endpoint for tick (and optional at `init` for chain-id auto-detect). +- `CARTESI_WATCHDOG_SEQUENCER_URL`: optional tick-time override of the URL persisted at `init` (useful when ephemeral ports change). +- `CARTESI_WATCHDOG_BLOCKCHAIN_ID`: optional chain id label persisted at `init` for `status.prom` (prefer explicit; tick never queries `eth_chainId`). +- `CARTESI_WATCHDOG_METRICS_FILE`: optional override for the Prometheus textfile path (default `$CARTESI_WATCHDOG_STATE_DIR/status.prom`). +- `CARTESI_WATCHDOG_RETRY_ATTEMPTS`: bounded retry attempts per run, default `3`. +- `CARTESI_WATCHDOG_RETRY_DELAY_SEC`: delay between retry attempts, default `5`. + +## Metrics (`status.prom`) + +Each `tick` writes a [Prometheus textfile](https://github.com/prometheus/node_exporter#textfile-collector) +before exiting. Operators scrape or push it from their side — the watchdog does +not run an HTTP server. + +| Exit code | `state` label | Meaning | +|-----------|---------------|---------| +| `0` | `ok` | Compare passed, or idle (finalized unchanged) | +| `1` | `warning` | Retryable failure after retries, or an operator/configuration error | +| `2` | `failed` | State mismatch or inclusion-block regression | + +Gauges (labels `chain`, `app_address` on every series): + +- `cartesi_watchdog_status{state="ok|warning|failed"}` — exactly one series is `1` +- `cartesi_watchdog_divergence_info{kind}` — only on exit `2` + +Exit codes map to `state` only (`0→ok`, `1→warning`, `2→failed`); we do not +export a separate exit-code or last-tick gauge — Prometheus scrape/push already +carries a sample timestamp. + +Set `CARTESI_WATCHDOG_BLOCKCHAIN_ID` at `init` for the `chain` label. If unset, +`init` queries `eth_chainId` from `CARTESI_WATCHDOG_BLOCKCHAIN_HTTP_ENDPOINT` and +persists the result. At `tick`, the env var takes precedence over the persisted value; +the exit path never blocks on RPC (defaults to `unknown` only when neither source +is set). Golden fixtures: [`tests/fixtures/watchdog_status_ok.prom`](../../tests/fixtures/watchdog_status_ok.prom), +[`tests/fixtures/watchdog_status_failed.prom`](../../tests/fixtures/watchdog_status_failed.prom). + +Example after a clean tick: + +```prometheus +cartesi_watchdog_status{app_address="0x4ce...",chain="11155111",state="ok"} 1 +cartesi_watchdog_status{app_address="0x4ce...",chain="11155111",state="warning"} 0 +cartesi_watchdog_status{app_address="0x4ce...",chain="11155111",state="failed"} 0 +``` + +Example Prometheus alert (pull or push gateway — operator choice): + +```promql +cartesi_watchdog_status{state="failed"} == 1 +``` + +Divergence playbook: **notify only**; manual intervention (see +[`operator-deployment.md`](operator-deployment.md)). + ## Host dependencies (`watchdog-lua-deps`) The watchdog cycle and any test that hits HTTP need a native **`lcurl.so`** built into `.deps/lua/`. JSON is pure Lua (no compile step). @@ -112,7 +257,7 @@ Lua modules: - `metrics.lua`: Prometheus textfile (`status.prom`) built and written each tick. - `retry.lua`: bounded retry helper used by the runtime. - `runner.lua`: one compare cycle — cheap `/finalized_state/inclusion_block` - poll, then (when finalized advanced) L1 fetch, CM replay, SSZ compare, + poll, then (when the accepted checkpoint advances) L1 fetch, CM replay, byte comparison, checkpoint write. - `main.lua`: dispatches `init` and `tick`; `tick` exits `0`/`1`/`2` and writes `status.prom`. @@ -135,136 +280,6 @@ this: its scan floor is the operator-supplied checkpoint and it performs no version witness. Do not copy the app-deployment floor into the Lua side without also porting the version witness that makes it sound. -## Runtime Contract - -The sequencer exposes operator-internal snapshot routes (see `sequencer/src/egress/api/snapshot.rs`): - -- `GET /finalized_state/inclusion_block` — cheap JSON `{ inclusion_block, executed_input_count }` polled every compare tick. -- `GET /finalized_state` — streams the finalized SSZ state file (`application/octet-stream`) with `X-Inclusion-Block` and `X-Executed-Input-Count` headers. - -**Idle optimization:** when `inclusion_block` has not advanced past the watchdog -checkpoint's `safe_block`, the tick returns -immediately — no `/finalized_state` download, no L1 `eth_getLogs`, no CM load/advance/inspect. - -The watchdog compares the finalized SSZ bytes with the bytes returned by CM -inspect. It must not canonicalize either side before deciding pass/fail. - -For the toy wallet app, SSZ encoding lives in `examples/app-core/src/wallet_snapshot.rs` -and is shared by `WalletApp::create_dump`, `CanonicalState::canonical_snapshot_bytes`, -and the canonical scheduler's `Inspect` handler (`examples/canonical-app`). - -## Checkpoints - -V1 persists only the resulting Cartesi Machine checkpoint, not the fetched L1 -inputs. - -```text -state_dir/ - config.json - head.json - status.prom # Prometheus textfile from the last tick (see Metrics below) - run.lock # advisory lock handle; file existence is not lock state - checkpoints/ - 00000000000001234567/ - snapshot/ - manifest.json -``` - -`manifest.json` records `safe_block` (the L1 reference block the CM snapshot -covers — the finalized `inclusion_block`), timestamp, -and optionally the CM image hash. A new checkpoint directory is written first, -then `head.json` is atomically replaced to point at it. - -`init` stores the operator-provided bootstrap CM snapshot into this layout. `tick` -requires both `config.json` and `head.json`; it never bootstraps from env. -`CARTESI_WATCHDOG_BLOCKCHAIN_HTTP_ENDPOINT` is not persisted in `config.json`, so -operators can rotate RPC endpoints without rewriting watchdog state. It is -required at `tick` for L1 reads, and optionally present at `init` when -auto-detecting `CARTESI_WATCHDOG_BLOCKCHAIN_ID` via `eth_chainId` (prefer setting -the chain id explicitly). - -- `CARTESI_WATCHDOG_CM_SNAPSHOT_DIR` -- `CARTESI_WATCHDOG_CM_SNAPSHOT_SAFE_BLOCK` - -## How it runs - -The watchdog has two subcommands: - -```bash -sequencer-watchdog init # setup: writes config.json + head.json (idempotent if complete) -sequencer-watchdog tick # one compare cycle; schedule this -``` - -`tick` does one cycle per process, then exits — infra schedules re-runs -(systemd timer / k8s CronJob) and reacts to the exit code. There is no daemon -loop. `sequencer-watchdog` takes a non-blocking `flock` for `init`/`tick`; -host scheduling should provide the same non-overlap guarantee. Each tick: - -1. Loads the watchdog checkpoint from `head.json`. -2. Polls `/finalized_state/inclusion_block`. If it has not advanced past a - watchdog checkpoint, exits `0` (idle). Otherwise: -3. Streams and decodes `InputAdded` logs for the new block range. -4. Replays each successful L1 partition into the in-process Cartesi Machine, - then inspects with query `state`. -5. Byte-compares the SSZ report against `GET /finalized_state`; on match writes a - new checkpoint, on mismatch emits a `watchdog_event` and exits `2`. -6. Atomically writes `$CARTESI_WATCHDOG_STATE_DIR/status.prom` (or - `CARTESI_WATCHDOG_METRICS_FILE`) before exit. - -Runtime knobs: - -- `CARTESI_WATCHDOG_BLOCKCHAIN_HTTP_ENDPOINT`: current L1 JSON-RPC endpoint for tick (and optional at `init` for chain-id auto-detect). -- `CARTESI_WATCHDOG_SEQUENCER_URL`: optional tick-time override of the URL persisted at `init` (useful when ephemeral ports change). -- `CARTESI_WATCHDOG_BLOCKCHAIN_ID`: optional chain id label persisted at `init` for `status.prom` (prefer explicit; tick never queries `eth_chainId`). -- `CARTESI_WATCHDOG_METRICS_FILE`: optional override for the Prometheus textfile path (default `$CARTESI_WATCHDOG_STATE_DIR/status.prom`). -- `CARTESI_WATCHDOG_RETRY_ATTEMPTS`: bounded retry attempts per run, default `3`. -- `CARTESI_WATCHDOG_RETRY_DELAY_SEC`: delay between retry attempts, default `5`. - -## Metrics (`status.prom`) - -Each `tick` writes a [Prometheus textfile](https://github.com/prometheus/node_exporter#textfile-collector) -before exiting. Operators scrape or push it from their side — the watchdog does -not run an HTTP server. - -| Exit code | `state` label | Meaning | -|-----------|---------------|---------| -| `0` | `ok` | Compare passed, or idle (finalized unchanged) | -| `1` | `warning` | Transient failure after retries | -| `2` | `failed` | Deterministic divergence | - -Gauges (labels `chain`, `app_address` on every series): - -- `cartesi_watchdog_status{state="ok|warning|failed"}` — exactly one series is `1` -- `cartesi_watchdog_divergence_info{kind}` — only on exit `2` - -Exit codes map to `state` only (`0→ok`, `1→warning`, `2→failed`); we do not -export a separate exit-code or last-tick gauge — Prometheus scrape/push already -carries a sample timestamp. - -Set `CARTESI_WATCHDOG_BLOCKCHAIN_ID` at `init` for the `chain` label. If unset, -`init` queries `eth_chainId` from `CARTESI_WATCHDOG_BLOCKCHAIN_HTTP_ENDPOINT` and -persists the result. At `tick`, the env var overrides a missing persisted value; -the exit path never blocks on RPC (defaults to `unknown` only when neither source -is set). Golden fixtures: [`tests/fixtures/watchdog_status_ok.prom`](../../tests/fixtures/watchdog_status_ok.prom), -[`tests/fixtures/watchdog_status_failed.prom`](../../tests/fixtures/watchdog_status_failed.prom). - -Example after a clean tick: - -```prometheus -cartesi_watchdog_status{app_address="0x4ce...",chain="11155111",state="ok"} 1 -cartesi_watchdog_status{app_address="0x4ce...",chain="11155111",state="warning"} 0 -cartesi_watchdog_status{app_address="0x4ce...",chain="11155111",state="failed"} 0 -``` - -Example Prometheus alert (pull or push gateway — operator choice): - -```promql -cartesi_watchdog_status{state="failed"} == 1 -``` - -Divergence playbook: **notify only**; manual intervention (see -[`operator-deployment.md`](operator-deployment.md)). - ## Local Tests | Command | What it exercises | diff --git a/docs/watchdog/design-notes.md b/docs/watchdog/design-notes.md index 91fe6eb..5c2e9ec 100644 --- a/docs/watchdog/design-notes.md +++ b/docs/watchdog/design-notes.md @@ -1,7 +1,7 @@ # Watchdog Design Notes The watchdog is an independent off-chain safety monitor. It advances a -canonical Cartesi Machine from L1 inputs, inspects the resulting SSZ snapshot, +canonical Cartesi Machine from L1 inputs, inspects its application-state bytes, and byte-compares it with the sequencer's `GET /finalized_state` response at the same finalized `inclusion_block`. @@ -30,7 +30,7 @@ Each tick: 3. Exits cheaply if the finalized block is unchanged. 4. Fetches L1 `InputAdded` logs for the open block range. 5. Advances the CM, inspects state, fetches `GET /finalized_state`, and compares - raw SSZ bytes. + raw bytes (SSZ for the wallet). 6. Writes a new checkpoint only after a successful compare. There is no advance-only mode. Advancing the CM is just an implementation step @@ -48,7 +48,7 @@ safe-input sync. For every at/above-anchor landing the mirrored scheduler accepts, it requires a byte-identical valid local sealed batch at that nonce. A foreign or mismatched landing persists `canonical_divergence`, which freezes the accepted frontier -and finalized-snapshot promotion +and the selection of newer accepted comparison checkpoints ([I15](../invariants.md#i15-divergence-marker-present--acceptance-frontier-frozen)). The offending landing therefore normally never produces a newer `/finalized_state/inclusion_block` for the watchdog to compare. Under the diff --git a/docs/watchdog/getting-started.md b/docs/watchdog/getting-started.md index 17d8505..ab74e9e 100644 --- a/docs/watchdog/getting-started.md +++ b/docs/watchdog/getting-started.md @@ -27,7 +27,7 @@ Step-by-step guide for running the watchdog alongside a **local** `sequencer-dev | Process | Role | |---------|------| | **Anvil** | Local L1 with Cartesi rollups contracts pre-deployed (`just setup`) | -| **sequencer-devnet** | Off-chain sequencer (wallet app, batches, snapshot promotion) | +| **sequencer-devnet** | Off-chain sequencer (wallet app, batches, accepted comparison checkpoints) | | **watchdog** | Polls `/finalized_state/inclusion_block`, replays L1 inputs in CM, compares SSZ to `/finalized_state` | The sequencer exposes (operator-internal, same HTTP listener today): @@ -109,9 +109,13 @@ A brand-new Anvil history still needs a **fresh** `$CARTESI_WATCHDOG_STATE_DIR` (e.g. `rm -rf /tmp/watchdog-state-devnet`) and a new `init` — old checkpoints won't match. -### Wait for finalized snapshot +### Check comparison availability -The watchdog needs a **finalized** SSZ dump. Right after boot, the cheap endpoint may return **404** until the sequencer has promoted a snapshot. +The watchdog needs a comparable checkpoint from `/finalized_state`. A fresh +genesis setup already provides one at block zero. A rebuilt baseline is not a +comparison checkpoint: after cockroach recovery these routes return **404** +until a new batch is accepted. See +[snapshot selection](../snapshots/lifecycle.md#acceptance-and-comparison). In another shell (use the printed `CARTESI_WATCHDOG_SEQUENCER_URL`): @@ -119,7 +123,11 @@ In another shell (use the printed `CARTESI_WATCHDOG_SEQUENCER_URL`): curl -s "$CARTESI_WATCHDOG_SEQUENCER_URL/finalized_state/inclusion_block" ``` -When you see JSON like `{"inclusion_block":0,"executed_input_count":0}` (numbers may differ), the watchdog can compare. If it stays 404 for a long time, check sequencer logs in `tests/e2e/results/` and that L1 is mining (devnet Anvil auto-mines by default). +JSON such as `{"inclusion_block":0,"executed_input_count":0}` confirms that the +endpoint has a comparison checkpoint. A tick compares only after the reported +block advances beyond its own CM checkpoint; an equal block exits idle. +Unexpected 404 on a fresh devnet warrants checking the URL and sequencer logs +in `tests/e2e/results/`. Optional — inspect SSZ size: @@ -141,11 +149,17 @@ export CARTESI_WATCHDOG_LUA_DEPS=.deps/lua ./watchdog/sequencer-watchdog tick ``` -Success: exit **0**. If finalized has advanced, stderr ends in `compare pass complete`; if it has not, the tick exits idle after the cheap poll. +Success: exit **0**. If the comparison block has advanced, stderr ends in +`compare pass complete`; if it is unchanged, the tick exits idle after the +cheap poll. `init` and an idle tick do not verify the bootstrap state. -Exit codes from `sequencer-watchdog tick`: **0** clean (or idle — finalized unchanged), **1** transient failure (RPC/CM/network after retries), **2** deterministic divergence (`watchdog_event` emitted on stderr before exit). Each tick writes `$CARTESI_WATCHDOG_STATE_DIR/status.prom` — see [`README.md` — Metrics](README.md#metrics-statusprom). +Exit codes from `sequencer-watchdog tick`: **0** comparison passed or idle, +**1** retries exhausted or operator/configuration error, **2** state mismatch +or inclusion-block regression (`watchdog_event` emitted on stderr). Each tick +writes `$CARTESI_WATCHDOG_STATE_DIR/status.prom` — see +[`README.md` — Metrics](README.md#metrics-statusprom). -The watchdog tick runs **one cycle per process and exits** — re-run it on a timer/cron for continuous monitoring. When `inclusion_block` has not advanced since the watchdog checkpoint, the cycle **skips** L1/CM work (idle-cheap) and exits 0. +The watchdog tick runs **one cycle per process and exits** — re-run it on a timer/cron for continuous monitoring. `sequencer-watchdog` takes a non-blocking `flock`; production schedulers should also prevent overlapping ticks with systemd or Kubernetes CronJob `concurrencyPolicy: Forbid`. @@ -161,7 +175,7 @@ Local paths A–B do **not** apply to public L1. There is no `just devnet-for-wa | You spawn Anvil + `sequencer-devnet` | Sequencer already run by ops | | `canonical-machine-image` (devnet guest) | `canonical-machine-image-sepolia` (today); mainnet guest when released | | Snapshot HTTP on localhost | **Internal** operator network only | -| Genesis bootstrap (`safe_block=0`) usual | Bootstrap must match **current** finalized `inclusion_block` | +| Genesis bootstrap (`safe_block=0`) usual | Trusted CM checkpoint at or before the current comparison block; tick replays the gap | **Sepolia is the dress rehearsal for mainnet** — same checklist, alarms, checkpoint volume, and firewall rules; only chain IDs, RPC URLs, and contract addresses change. @@ -200,9 +214,9 @@ See `watchdog/config.lua` for the full list. | `cartesi Lua module is required` | Install Cartesi Machine; use nix/direnv shell; ensure `cartesi-machine` on `PATH` | | `inspect endpoint not implemented` | Rebuild CM image: `just canonical-build-machine-image` | | CM inspect ~27 bytes / JSON in error | Stale image (old JSON inspect); rebuild: `just canonical-build-machine-image` | -| HTTP 404 on `/finalized_state/inclusion_block` | Sequencer not promoted yet; wait or drive L1 + batches | +| HTTP 404 on `/finalized_state/inclusion_block` | Wrong URL, or no comparable checkpoint after a rebuild; a new accepted batch makes the rebuilt history comparable | | `state_mismatch` at genesis | Wrong `CARTESI_WATCHDOG_CM_SNAPSHOT_*` or stale CM image vs sequencer build | -| `inclusion_block_regressed` | Watchdog state ahead of sequencer (reset state dir or fix bootstrap block) | +| `inclusion_block_regressed` | Watchdog checkpoint is ahead of the advertised comparison block; inspect deployment identity, bootstrap block, and sequencer recovery history before reinitializing | | `flock` lock conflict | Another tick is still running or the scheduler allows overlap. With the container `flock`, a leftover `run.lock` path alone is harmless. | | `could not determine which binary to run` | Use `just test-watchdog-compare-harness` (not bare `cargo run -p rollups-e2e`) | | Harness `87 vs 76` or `27 vs 76` byte mismatch | Stale CM image and/or wrong fixture; see [harness troubleshooting](README.md#troubleshooting-just-test-watchdog-compare-harness) | diff --git a/docs/watchdog/operator-deployment.md b/docs/watchdog/operator-deployment.md index 1202813..f9b31d8 100644 --- a/docs/watchdog/operator-deployment.md +++ b/docs/watchdog/operator-deployment.md @@ -25,7 +25,10 @@ For **local development only** (Anvil + `sequencer-devnet`, CI smoke tests), use └─────────────┘ └────────────────┘ ``` -The watchdog never substitutes for the sequencer. It reads **finalized SSZ** the sequencer already committed and independently replays L1 through the canonical CM. +The watchdog independently replays L1 through the canonical CM and compares +application-state bytes with the sequencer's accepted checkpoint. The wallet +uses SSZ. The [runtime contract](README.md#runtime-contract) explains block +positioning and how the `/finalized_state` name relates to safe acceptance. --- @@ -43,9 +46,13 @@ Verify snapshot API before CM bootstrap: ```bash curl -sS -o /dev/null -w "%{http_code}\n" "$CARTESI_WATCHDOG_SEQUENCER_URL/finalized_state/inclusion_block" -# expect 200 when a finalized snapshot exists (404 = not promoted yet or wrong host) +# expect 200 when a comparable checkpoint exists ``` +A genesis baseline is comparable immediately. A rebuilt baseline returns 404 +until a new batch is accepted; also check for a wrong host or tier. See +[snapshot selection](../snapshots/lifecycle.md#acceptance-and-comparison). + ### 2. Watchdog runtime (release image or local build) **Production (recommended):** pull the **release container image** for tag `vX` — same @@ -156,7 +163,7 @@ Today `WalletApp::default()` / `WalletConfig::sepolia()` align with Sepolia stag | `CARTESI_WATCHDOG_CONTRACTS_INPUT_BOX_ADDRESS` | InputBox on that L1 ([Cartesi deployed contracts](https://docs.cartesi.io/cartesi-rollups/2.0/deployment/self-hosted.md); same lowercase normalization) | | `CARTESI_WATCHDOG_STATE_DIR` | Persistent volume on watchdog host. If the path embeds an address, use **lowercase** — Linux paths are case-sensitive and EIP-55 vs lowercase create sibling dirs | | `CARTESI_WATCHDOG_CM_SNAPSHOT_DIR` | Bootstrap CM snapshot (`init` only) | -| `CARTESI_WATCHDOG_CM_SNAPSHOT_SAFE_BLOCK` | L1 block that bootstrap snapshot represents (= finalized `inclusion_block` at bootstrap) | +| `CARTESI_WATCHDOG_CM_SNAPSHOT_SAFE_BLOCK` | L1 block through which the trusted bootstrap CM has consumed all inputs; at or before the current comparison target | | `CARTESI_WATCHDOG_BLOCKCHAIN_ID` | Chain id label for `status.prom` metrics (prefer set at `init`; optional auto-detect via `eth_chainId` when L1 endpoint is present at `init`) | | `CARTESI_WATCHDOG_METRICS_FILE` | Override path for the Prometheus textfile written by each `tick` | | `CARTESI_WATCHDOG_LUA_DEPS` | `.deps/lua` | @@ -166,21 +173,31 @@ The sequencer discovers and pins `input_box_address` at startup; use the same va ### 5. Initialize watchdog state (first run on a live chain) -On a long-lived deployment, **`CARTESI_WATCHDOG_CM_SNAPSHOT_SAFE_BLOCK=0` is usually wrong** unless finalized state is still at genesis. +The bootstrap must be a trusted **whole CM checkpoint** after all inputs +through its declared L1 block. It need not match the sequencer's current +comparison block: tick replays the gap. `SAFE_BLOCK=0` is valid for a genesis +image even on an old deployment, but replaying the full history can be costly. Pick one: -1. **Ops hands off** a CM snapshot directory + block number matching current finalized `inclusion_block`, or +1. **Ops hands off** a trusted CM snapshot directory + its covered block number, or 2. **Watchdog reuses** `CARTESI_WATCHDOG_STATE_DIR` from a prior run on this deployment, or -3. **Replay from genesis** (only for new rollups / low block height — slow). +3. **Replay from genesis** (potentially slow). + +The sequencer's `/finalized_state` comparison file and app-owned snapshot +archives are different artifacts; neither is automatically a CM bootstrap. +`init` trusts the supplied checkpoint and block without comparing them against +the sequencer. If the first tick sees that same block, it exits idle. See +[the detection boundary](design-notes.md#watchdog-state). Run `init` once to store the bootstrap CM snapshot into the watchdog state layout. Re-running `init` on a **complete** already-initialized state directory is a no-op success (exit `0`), matching `sequencer setup` — safe for process -supervisors that always invoke init before tick. If `head.json` exists but -`config.json` or the selected snapshot is missing/corrupt, `init` fails (exit -`1`) and asks you to wipe `state_dir` and re-run — it will not certify an -unusable state. The L1 RPC URL is not persisted — each `tick` reads +supervisors that always invoke init before tick. It validates the saved +metadata and that the selected snapshot directory is nonempty; it does not +reload an existing CM checkpoint to prove it usable. Missing or malformed +metadata and missing/empty snapshots fail with exit `1`. The L1 RPC URL is not +persisted — each `tick` reads `CARTESI_WATCHDOG_BLOCKCHAIN_HTTP_ENDPOINT` so it can rotate without editing state. If `CARTESI_WATCHDOG_BLOCKCHAIN_ID` is unset at `init`, auto-detect also needs that endpoint present then (prefer setting the chain id explicitly): @@ -202,8 +219,8 @@ budget. `checkpoints//`. 2. If `head.json` is missing: run `sequencer-watchdog init` with `CARTESI_WATCHDOG_CM_SNAPSHOT_DIR` and `CARTESI_WATCHDOG_CM_SNAPSHOT_SAFE_BLOCK` - set to a CM snapshot whose safe block equals the sequencer's current - finalized `inclusion_block`, then run `tick`. + set to a trusted CM checkpoint and its covered block, at or before the + current comparison target, then run `tick`. 3. Schedule `sequencer-watchdog init && sequencer-watchdog tick` (init is a no-op when state is already complete). 4. If `head.json` exists but `config.json` or the selected snapshot is @@ -224,7 +241,7 @@ last-tick gauges — Prom already timestamps samples). Set `CARTESI_WATCHDOG_BLOCKCHAIN_ID` at `init` so `chain` is labeled. If unset, `init` queries `eth_chainId` from `CARTESI_WATCHDOG_BLOCKCHAIN_HTTP_ENDPOINT` and -persists the result. At `tick`, the env var overrides a missing persisted value; +persists the result. At `tick`, the env var takes precedence over the persisted value; the exit path never blocks on RPC (falls back to `unknown` only when neither source is set). @@ -262,7 +279,7 @@ Prometheus push/pull) or on the process exit code. If the process is killed mid-tick, `status.prom` keeps the last completed value until the next run. ```bash -sequencer-watchdog tick # exit 0 = clean/idle, 1 = transient, 2 = divergence +sequencer-watchdog tick # 0 = passed/idle, 1 = retry or operator error, 2 = mismatch/regression ``` `sequencer-watchdog` wraps `init` and `tick` with a non-blocking `flock` on @@ -271,7 +288,8 @@ dies. Use the scheduler's non-overlap primitive as well (for example systemd or Kubernetes CronJob `concurrencyPolicy: Forbid`). A leftover `run.lock` path is only a lock handle; by itself it does not mean a lock is held. -When `inclusion_block` ≤ the watchdog checkpoint, the runner only hits `/finalized_state/inclusion_block` and skips L1/CM work. +An unchanged `inclusion_block` exits idle; a lower block reports +`inclusion_block_regressed` and exits `2`. Both skip L1/CM work. --- @@ -299,7 +317,7 @@ export CARTESI_WATCHDOG_APP_ADDRESS="0x..." export CARTESI_WATCHDOG_CONTRACTS_INPUT_BOX_ADDRESS="0x..." export CARTESI_WATCHDOG_STATE_DIR="/var/lib/watchdog/state-sepolia" export CARTESI_WATCHDOG_CM_SNAPSHOT_DIR="/path/to/canonical-machine-image-sepolia" -export CARTESI_WATCHDOG_CM_SNAPSHOT_SAFE_BLOCK="" +export CARTESI_WATCHDOG_CM_SNAPSHOT_SAFE_BLOCK="" export CARTESI_WATCHDOG_LUA_DEPS="/path/to/sequencer/.deps/lua" ``` @@ -308,7 +326,7 @@ export CARTESI_WATCHDOG_LUA_DEPS="/path/to/sequencer/.deps/lua" If your team runs the sequencer on Sepolia (not only the public endpoint): 1. `sequencer` / release binary with Sepolia `CARTESI_SEQUENCER_*` (chain id, app address, batch submitter key, L1 RPC). -2. Inclusion lane promotes finalized snapshots when L1 safe advances — required for `/finalized_state` 200. +2. Safe accepted batches select comparison checkpoints; genesis is also comparable. A rebuilt baseline needs a new accepted batch before `/finalized_state` returns 200. 3. Snapshot routes on an **internal** bind / port reachable by the watchdog host. 4. Sequencer binary built with **`WalletApp::new(WalletConfig::sepolia())`** (see `sequencer-devnet` vs production binary choice in your release pipeline). @@ -323,7 +341,7 @@ When the rollup runs on Ethereum mainnet, **reuse the same operator checklist ab | L1 RPC | Production-grade archive provider; rate limits matter for wide `getLogs` ranges | | Contracts | Mainnet InputBox, application, portals from production deployment manifest | | CM image | Build from production app/scheduler artifacts (mainnet wallet constants when defined in app-core) | -| Schedule cadence | A cron/timer interval of 300s+ is fine; finalized promotion follows mainnet safe head | +| Schedule cadence | Choose the alert delay you can tolerate; new comparison checkpoints follow safe batch acceptance | | Security | Stricter firewall between public ingress and internal snapshot tier; secrets management for RPC credentials | | Bootstrap | Almost always ops-provided CM snapshot or continued state dir — not genesis replay | @@ -333,35 +351,35 @@ There is no `just devnet-for-watchdog` or automated harness on mainnet; treat Se ## Compare Cycle Behavior (All Live Chains) -Same on Sepolia and mainnet: - -1. Load watchdog checkpoint from `head.json`. -2. `GET /finalized_state/inclusion_block` — if unchanged, **stop** (cheap). -3. If advanced: `eth_getLogs` on InputBox for `(last_block+1)..inclusion_block`. -4. Advance CM incrementally; `inspect` → SSZ bytes. -5. `GET /finalized_state` → SSZ bytes. -6. Raw compare; emit `watchdog_event` + non-zero exit on mismatch. -7. Write new CM checkpoint on success. - -Details: [`README.md`](README.md), [`docs/snapshots/lifecycle.md`](../snapshots/lifecycle.md). +The [runtime contract](README.md#runtime-contract) is the same on Sepolia and +mainnet: poll the comparison block, replay new inputs, compare bytes at the +same block, then save a checkpoint. It defines idle, regression, and retry +behavior when the comparison target moves during a tick. --- ## Checkpoint disk usage and backups -Each successful promotion stores a full CM snapshot under -`$CARTESI_WATCHDOG_STATE_DIR/checkpoints//`, and the watchdog **keeps only -the selected one** — after the atomic `head.json` flip it deletes the -checkpoint it superseded (crash-safe: `head.json` always names a complete -checkpoint). Local disk therefore stays bounded at a single snapshot; no -operator cleanup is required. - -For backups / rollback history, schedule the watchdog tick (it runs one cycle and -exits) and **after it exits** `aws s3 sync $CARTESI_WATCHDOG_STATE_DIR/checkpoints/ -s3://…` (without `--delete`). Because the process has exited there is no race -with its store or prune, and omitting `--delete` **accumulates a per-block -history in S3** while local disk stays at one snapshot. Restore feeds a chosen -snapshot back through the watchdog/sequencer recovery workflow. +Each successful comparison stores a full CM snapshot under +`$CARTESI_WATCHDOG_STATE_DIR/checkpoints//`, then atomically replaces +`head.json` and attempts to delete its predecessor. Pruning is best effort: +failed or interrupted writes/prunes can leave extra directories, so disk use +is not strictly bounded to one snapshot. The +[checkpoint crash model](design-notes.md#checkpoint-crash-model) owns the +pointer-swap sequence and its current lack of file/directory fsync. + +Run tick and backup sequentially in the **same nonoverlapping scheduled job**, +so the next tick cannot store or prune while backup is reading. For a complete +watchdog-state backup, retain `config.json`, `head.json`, and the selected +checkpoint together. Alternatively, syncing `checkpoints/` to object storage +without deletion accumulates a per-block CM history; restore a chosen snapshot +with its manifest's block through watchdog `init`. + +Sequencer recovery takes a native application archive with an acceptance +receipt, exported by `/finalized_snapshot`. Do not treat a watchdog CM +checkpoint as that archive. The +[snapshot backup workflow](../snapshots/lifecycle.md#http-and-recovery-exports) +owns this separate restore path. ## Sequencer restart policy @@ -410,8 +428,8 @@ unclassified restart-with-backoff. Operational notes: |---------|----------------| | `/finalized_state` missing on public URL | Wrong tier — use internal `CARTESI_WATCHDOG_SEQUENCER_URL` | | `failed to load watchdog head` / missing `head.json` | Uninitialized or wiped `STATE_DIR` — see [Missing or corrupt head.json](#missing-or-corrupt-headjson-tick-exit-1) | -| `state_mismatch` | CM image / wallet constants ≠ sequencer build; or wrong bootstrap block | -| `inclusion_block_regressed` | Stale watchdog state vs sequencer finalized head | +| `state_mismatch` | Sequencer/canonical-state disagreement; also verify CM image, wallet constants, and trusted bootstrap block | +| `inclusion_block_regressed` | Watchdog checkpoint is ahead of the advertised comparison block; inspect deployment identity, bootstrap block, and sequencer recovery history before reinitializing | | Slow or failing `getLogs` | RPC range limits — watchdog uses same partition strategy as sequencer | | Transient `L1 RPC latest head lags target block` | Fallback RPC is behind the sequencer's finalized inclusion block; watchdog retries until the node has indexed through the target (avoids truncated `eth_getLogs` false mismatches) | | `inspect endpoint not implemented` | Rebuild CM image for the correct chain target | From 35697691d7a5ba5a2c868f51b3a45c3dd5b6ee44 Mon Sep 17 00:00:00 2001 From: gcdepaula Date: Thu, 17 Sep 2026 15:52:14 -0300 Subject: [PATCH 4/4] docs: retire stale reviews and define review lifecycle --- AGENTS.md | 15 +- README.md | 3 + docs/invariants.md | 4 +- docs/plans/2026-07-coordination-tracks.md | 31 +- docs/plans/2026-07-track6-dump-api-design.md | 218 ----- docs/plans/2026-08-authority-boundary-adr.md | 69 +- docs/protocol/c-application-binding.md | 10 +- .../2026-08-18-over-engineering-review.md | 46 - .../2026-08-22-lifecycle-simplification.md | 60 -- docs/review/2026-09-03-branch-stocktake.md | 847 ------------------ .../2026-09-09-application-lane-dex-review.md | 389 -------- docs/review/2026-09-16-track3-validation.md | 33 +- docs/review/README.md | 60 ++ docs/review/register.md | 832 +++-------------- sequencer-core/src/fee.rs | 46 +- 15 files changed, 295 insertions(+), 2368 deletions(-) delete mode 100644 docs/plans/2026-07-track6-dump-api-design.md delete mode 100644 docs/review/2026-08-18-over-engineering-review.md delete mode 100644 docs/review/2026-08-22-lifecycle-simplification.md delete mode 100644 docs/review/2026-09-03-branch-stocktake.md delete mode 100644 docs/review/2026-09-09-application-lane-dex-review.md create mode 100644 docs/review/README.md diff --git a/AGENTS.md b/AGENTS.md index a64aae4..92b5cb7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -341,12 +341,11 @@ Keep three kinds of material distinct: must distinguish proposed behavior from implemented contracts. On completion, put the durable design in its owner and reduce the plan to its remaining work and links. -- **Historical evidence** lives in review ledgers, explicitly marked historical - documents, and commit history. A review ledger is append-only while open. - When it closes, promote conclusions into current docs, record settled and - refuted proposals in the [review register](docs/review/register.md), and remove - process narration. Retained superseded proposals must say they are historical - and link to the current contract; they are evidence, not instructions. +- **Review notes** are temporary working memory. Commit them when they help an + active review or handoff, then distill and delete them when that work ends. + The [review lifecycle](docs/review/README.md) owns the policy: unresolved work + stays in one register or active plan, durable reasoning in its current owner, + completed history in Git. Keep dated evidence only for a named ongoing use. **Record deliberate absence once**, at the seam where someone would re-add the mechanism, phrased as a positive design statement with its reason. Avoid removal @@ -400,7 +399,7 @@ See [Running](README.md#running) for the two-phase `setup` / `run` workflow. - Run at least `cargo check` before finishing. - Read the relevant recovery guide and both current TLA+ models before touching recovery code, and the threat model before touching trust-boundary code. -- Check [`docs/invariants.md`](docs/invariants.md) before changing anything it lists as load-bearing, and [`docs/review/register.md`](docs/review/register.md) for open findings in the code you're about to touch and for decisions already settled or refuted. +- Check [`docs/invariants.md`](docs/invariants.md) before changing anything it lists as load-bearing, the owning design for its assumptions, and [`docs/review/register.md`](docs/review/register.md) for unresolved work in the code you're about to touch. Verify review claims against current code. ### Ask First @@ -451,4 +450,4 @@ work reaches another boundary. | Submission fees or oracle pricing | [L1 fee policy](docs/l1-fee-policy.md) — estimation and replacement limits; [threat-model actor table](docs/threat-model/README.md#actors-and-trust) — oracle source and outage assumptions. | | Command setup or deployment configuration | [Running](README.md#running) and [config.rs](sequencer/src/commands/config.rs) — invocation, identity pinning, defaults, and validation. | | Watchdog development or operation | [Architecture](docs/watchdog/README.md); [local dev](docs/watchdog/getting-started.md) for Anvil; [operator deployment](docs/watchdog/operator-deployment.md) for Sepolia/mainnet. | -| A new mechanism, simplification, or work spanning an active track | [Review register](docs/review/register.md) — relevant open findings and settled/refuted reasoning; [coordination tracks](docs/plans/2026-07-coordination-tracks.md) — remaining work and dependencies. Consult dated evidence when its reasoning is needed. | +| A new mechanism, simplification, or work spanning an active track | Owning design and [invariants](docs/invariants.md) — reasons and assumptions; [review register](docs/review/register.md) — unresolved work; [coordination tracks](docs/plans/2026-07-coordination-tracks.md) — priorities and dependencies. Follow the [review lifecycle](docs/review/README.md) when recording conclusions. | diff --git a/README.md b/README.md index 8dfc7cf..72a8306 100644 --- a/README.md +++ b/README.md @@ -240,6 +240,9 @@ After each successfully applied input at offset `X`, persist the claim with including business failures and malformed-direct no-ops. - Recovery stops the process and disconnects subscribers. A reconnect must present its saved claim; offsets alone cannot distinguish a replaced suffix. +- Shutdown or a feed read/send failure may disconnect without a WebSocket + Close frame. Resume from the saved claim after an unexpected disconnect; + a clean close is not required for safe replay. Message shapes: diff --git a/docs/invariants.md b/docs/invariants.md index f9e44ac..422aaff 100644 --- a/docs/invariants.md +++ b/docs/invariants.md @@ -371,7 +371,9 @@ by writer and are write-once (`0001_schema.sql`). same anchor via `open_fresh_tip_in_tx`'s `parent = None` path, after invalidating the old root — so only one *valid* parentless root ever exists, invalidated ones coexisting. -- **Enforced by:** `trg_enforce_nonce_contiguity` — its parentless arm is an +- **Enforced by:** the parent foreign key (enabled on every writer) rejects + dangling parents; `trg_enforce_nonce_contiguity` checks nonce succession. + Its parentless arm is an *exact* match `nonce == (SELECT nonce FROM batch_tree_anchor)` (tighter than a bare "must be 0"), plus an at-most-one-valid-parentless-root guard scoped to `invalidated_at_ms IS NULL`; `compute_next_nonce(None)` reads the diff --git a/docs/plans/2026-07-coordination-tracks.md b/docs/plans/2026-07-coordination-tracks.md index b4781d7..d10ffd4 100644 --- a/docs/plans/2026-07-coordination-tracks.md +++ b/docs/plans/2026-07-coordination-tracks.md @@ -2,7 +2,7 @@ **Status:** active plan of record. Tick / annotate as work lands; when a track completes, move its durable outcomes into the normative docs and -collapse its entry here. +remove its entry here. Track numbers retain their existing identities. Context: Bart is building **libdex**, a native (non-CM) app whose backing storage is an mmap'd flat buffer, and will reimplement the scheduler in C++. @@ -12,14 +12,9 @@ freely at this stage — no backward-compatibility constraints. | # | Track | Owner | Status | |---|-------|-------|--------| -| 1 | WS context fields + L1 provenance (PR #26) | Stephen | **done** — merged to main | -| 2 | Restore `docs/review/` ledger + this plan | us | **done** | | 3 | Feed & replay protocol redesign | us (design) → us/Stephen (impl) | **implemented** — canonical application history, snapshot restore archives, mandatory WS claims, typed refusals, and SDK cutover; [remaining integration gates](2026-07-track3-feed-replay-design.md#remaining-integration-gates) | -| 4 | Storage decode policy | us | **done** — fail-loud for contract-impossible values; the named `saturating_query_bound` only where clamping preserves the predicate (policy lives in `storage/convert.rs` + the invariants check policy) | | 5 | Fee exponentiation LUT | us | **deferred** — decided exact-floor if built (the table *is* the spec, algorithm-free; replay continuity across the upgrade explicitly not preserved); a separate pending design decision may make log-space fees defunct — revisit after syncing with Bart | | 6 | Dump / `Application` API redesign | us + Bart | **interface and reference C binding implemented** — [Application contract](../protocol/application-contract.md); native-engine integration gates remain | -| 7 | LLM context-engineering review | us | **done** — skills/agents/settings homed in-tree; the docs-practice rules live in AGENTS.md | -| 8 | Runtime ownership and terminal stop | us | **done** — owned by the [authority-boundary ADR](2026-08-authority-boundary-adr.md) | **Current campaign order:** @@ -67,8 +62,22 @@ implemented. End-to-end native snapshot-to-live bootstrap remains an integration gate, alongside the private DEX engine when available. Reference bridge conformance cannot establish private-engine correctness. -The [July proposal](2026-07-track6-dump-api-design.md) is historical; the -[September review](../review/2026-09-09-application-lane-dex-review.md) records the -accepted simplifications. Additional public checkpoint primitives or asynchronous -scheduling need a measured requirement. Watchdog extraction from the DEX's -canonical state drive remains separate work. +Remaining checks need the actual consumer: + +- Exercise snapshot-to-live bootstrap and canonical comparison through the C + host in CI; its current smoke test builds and invokes `--help`. A reusable + conformance runner needs engine-supplied genesis and meaningful accepted and + rejected inputs. Compare canonical state files, not recovery-dump layouts. +- Verify the external scheduler's ordering, fee conversion, and recovery + agreement. Publish independent-port fee vectors for the + [current arithmetic](../../sequencer-core/src/fee.rs); a deferred LUT is a + separate semantic change. Watchdog extraction from the DEX's canonical state + drive remains integration work. +- Decide output storage, checkpoint layout, ABI version negotiation, generated + bindings, and linker policy from concrete engine requirements. The current + drain protocol and path callback remain the contract until then. + +Additional checkpoint primitives or asynchronous scheduling need a measured +requirement. Any future microbatch priority scheme must preserve per-account +nonce order, serial execution, and the frame-time contract; the current lane +does not promise priority scheduling. diff --git a/docs/plans/2026-07-track6-dump-api-design.md b/docs/plans/2026-07-track6-dump-api-design.md deleted file mode 100644 index 528ce0a..0000000 --- a/docs/plans/2026-07-track6-dump-api-design.md +++ /dev/null @@ -1,218 +0,0 @@ -# Historical dump / `Application` proposal (Track 6) - -**Status: superseded by the 2026-09-09 design decision.** Preserved as the -original proposal; its public clone/flush machinery was not adopted. The -[current Application contract](../protocol/application-contract.md) specifies -mutable checkpoint creation, independent restore, and the distinction between -recovery checkpoints and canonical comparison bytes. The -[review ledger](../review/2026-09-09-application-lane-dex-review.md) records the -reasoning. The text below is historical. - -## 1. Motivation - -Three forces, one API: - -- **Cost.** `create_dump(&self)` is a full O(state) serialize + 3-fsync - ladder, run synchronously on the single lane thread at every batch close — - soft-confirmation acks stall for the duration. Tolerable for the toy - wallet; not for a multi-GB flat buffer. -- **Bart's app shape.** libdex runs natively against an mmap'd flat state - buffer — the Cartesi Machine's own storage approach (Bart implemented that - feature), whose ecosystem exploits CoW (`clone_stored`: reflink, plus - hardlinks — safe there only because stored images are immutable, see §10 — - with plain-copy fallback; ~2.6 ms vs ~350 ms full store at 533 MB in - Dave's measurements). -- **Verb review.** `create_dump / from_dump / delete_dump / - state_file_in_dump` is a leaky projection of what the lane needs; the - 2026-07 investigation mapped it against the CM/Dave verb set. - -Key investigation results this design builds on: the CM emulator has **no -commit/revert** — those are node-level orchestration over `clone_stored`, -and the sequencer already has both (DB row as commit point; older-dump + -replay as revert) correctly *off* the trait. The one missing primitive is -**cheap clone**. And the CM has since moved our way on durability: -machine-emulator PR #398 adds `rename_stored` and makes it and -`remove_stored` **durable (auto-synced)** — the commit-point idiom our dump -lifecycle hand-rolls. - -## 2. Requirements - -From the lane trace (R) and the wishlist (W): - -- **R1** Checkpoint the current state at batch close — atomic, - crash-durable *before* the DB row lands (invariant I13). -- **R2** Reconstruct at startup: latest checkpoint + replay. -- **R3** Dispose checkpoints (GC + orphan sweep). -- **R4** Serve canonical bytes over HTTP without instantiating the app; - bytes must equal the canonical machine's `inspect_state` output (the - watchdog byte-compares). -- **R5** Genesis construction (off-trait today, stays off-trait). -- **W1** Checkpoints cheap enough to not shape batch policy (CoW). -- **W2** Checkpointing off the ack path. -- **W3** Natural fit for an mmap'd-working-image app without penalizing - pure-RAM apps (WalletApp). - -## 3. The fork, decided: working-image model - -The investigation flagged one load-bearing fork: cheap clones require the -app to run against an on-disk image the lane can flush-then-clone (Dave's -`SHARING_ALL` model); `create_dump(&self)` serializing live RAM can never be -cheap. **This design takes the working-image model** — it is libdex's -natural shape, it is what the CM ecosystem optimizes, and WalletApp adapts -trivially (its "working image" is a file it rewrites on flush; cost -unchanged from today's serialize). - -## 4. Proposed trait - -```rust -pub trait Application: Send + Sized { - // --- execution surface unchanged --- - - /// Open the app on a working image directory. The sequencer owns the - /// directory's lifecycle; the app owns its contents. Called at startup - /// (from a cloned checkpoint) and after genesis materialization. - fn open(working: &Path) -> Result; - - /// Make the working image consistent and durable on disk: flush - /// app-level caches, msync mapped pages, fsync files. After `Ok`, the - /// on-disk image alone reconstructs this exact logical state via - /// `open`. Called by the lane at batch close, before cloning. - fn flush(&mut self) -> Result<(), AppError>; - - /// Clone the (flushed, not currently open) image at `from` into `to` - /// (must not exist). Default: recursive plain copy + fsync ladder. - /// CoW apps override with reflink/hardlink (FICLONE / clonefile), - /// keeping the same durability contract: on `Ok`, `to` survives an - /// immediate kernel crash. - fn clone_image(from: &Path, to: &Path) -> Result<(), AppError> { … } - - /// Delete an image directory the sequencer no longer references. - /// Default: remove_dir_all. - fn delete_image(prefix: &Path) -> Result<(), AppError> { … } - - /// Locate the single canonical state file inside an image without - /// opening the app. Contract unchanged from state_file_in_dump: - /// the file's bytes equal canonical `inspect_state` output (R4). - fn canonical_file_in_image(prefix: &Path) -> PathBuf; -} -``` - -Verb mapping: `open` ≈ CM `load(SHARING_ALL)`; `flush` ≈ the msync the CM's -dirty-page sidecars make optional; `clone_image` ≈ `cm_clone_stored`; -`delete_image` ≈ `remove_stored`. Commit stays sequencer-side (DB row; -sealing by durable rename per PR #398's precedent). Revert stays -sequencer-side (older checkpoint + replay). `from_dump`/`create_dump` -disappear: restore is `clone_image(checkpoint, working)` + `open(working)`; -checkpoint is `flush()` + `clone_image(working, checkpoint)`. - -## 5. Lane lifecycle changes - -Batch close becomes: `flush()` → `clone_image(working, staging)` → -sequencer writes `info.toml` + durable-renames staging into the dumps dir → -DB row in one tx (unchanged commit point). Filesystem-first ordering, the -"orphan dir possible, dangling row never" invariant, promotion, GC, leases, -and the whole `snapshot_dumps.rs` layer are **unchanged** — the storage half -is already representation-agnostic (opaque prefix keys). - -W2 (off-ack-path) falls out for CoW apps: `flush` + reflink is -milliseconds, and the expensive part (page write-back) is the kernel's -business afterward. A dedicated async stage is *not* designed in; if a -non-CoW app's flush is slow, that is the app's cost to fix by adopting CoW. - -Startup (R2): clone the promoted/pending checkpoint into a fresh working -dir, `open`, replay. The working dir is disposable state — never promoted, -never served, deleted on clean start. - -## 6. Crash-safety posture (unchanged, now sharper) - -I13 stays: nothing may reach the DB before the corresponding image is -durable. The split is now explicit: the **app** guarantees durability of -image *contents* (`flush`, `clone_image`); the **sequencer** guarantees -durability of *directory structure* (rename + dir-fsync — exactly what CM -PR #398 now bakes into `rename_stored`/`remove_stored`, validating the -posture). Tell Bart directly: the CM's historical no-fsync stance does not -apply here — a CoW `clone_image` override must fsync what reflink leaves -unsynced, and #398 shows the CM itself now agrees for the rename/remove -verbs. - -## 7. Serving (R4) and the libdex layout constraint - -`canonical_file_in_image` keeps the single-canonical-file contract — the -HTTP snapshot routes, lease protocol, and watchdog byte-compare all survive -untouched. The constraint to put in front of Bart *before* libdex's layout -freezes: the served file must byte-match canonical `inspect_state` output, -so either (a) the flat buffer's layout is itself canonical — fully -normalized, no allocator padding, no free-lists, no pointer-valued fields, -no uninitialized gaps — and the buffer file doubles as the canonical file; -or (b) libdex writes a separate canonical projection during `flush`, which -reintroduces O(state) serialize cost and partly defeats CoW. (a) is the -performant answer and a real design constraint on his buffer format. - -## 8. Migration & cleanups folded in - -- **WalletApp:** `open` mmap-or-reads its file; `flush` = today's - serialize+fsync ladder; defaults cover the rest. No capability lost. -- **Genesis:** concrete-type constructor materializes the initial working - image, then the normal flush/clone path checkpoints it (R5 unchanged). -- **`SafeInputRecord` shim** (`storage/l1_inputs.rs`): collapse - `StoredSafeInput`/`IngestedSafeInput` into one honest row model in a dedicated - cleanup. The provenance/clock decision below is settled; it no longer blocks - that cleanup. -- **Direct-input clock semantics (settled with Track 3):** `block_timestamp` - is persisted and served as feed provenance only; it is not an application - transition input. Directs execute at their exact L1 inclusion block and user - ops at their frame's safe block, as owned by the - [`Application` contract](../protocol/application-contract.md#3-the-safe-block-clock--last_executed_safe_block). - No timestamp-bearing trait change is needed, and this no longer blocks - libdex's state design. - -## 9. Open questions - -1. **Working-image locking:** the CM uses flock to make `SHARING_ALL` - exclusive. Do we require the app to hold an equivalent lock, or does the - sequencer's single-lane discipline suffice? (Lean: sequencer discipline - suffices; a lock is cheap defense — app's choice.) -2. **`flush` durability scope:** must `flush` fsync, or is fsync deferred to - `clone_image`? (Lean: `clone_image` owns durability of the *clone*; - `flush` owns consistency of the *source* — msync yes, fsync optional.) -3. **Non-reflink filesystems:** plain-copy fallback makes checkpoint cost - O(state) again silently. Log loudly at startup when the dumps dir does - not support reflink? (Lean: yes — one probe at bootstrap.) -4. **Dirty-page tracking for hashing/inspect:** the CM's `.dpt` sidecars - make post-clone hashing touch only dirty pages. libdex would need its - own write-barrier to replicate; out of scope for the sequencer API but - worth flagging to Bart as a cost driver for (a) in §7. - -## 10. Review remarks (non-normative, open design) - -These remarks record issues raised during review; they do not replace the -proposed trait or lifecycle above. (An earlier revision promoted the first -remark into §4's `clone_image` contract and misattributed that promotion to -a maintainer instruction; the 2026-08-01 review corrected the record and the -remarks-only posture is restored. Whether hardlinks stay in §4's suggestion -is settled in design review with Bart, alongside the rest of the trait.) - -- Hardlinks are not a valid implementation of the mutable-working-image to - immutable-checkpoint clone. Later in-place or mmap writes through either - path mutate the same inode and therefore the checkpoint. Any accepted - implementation should require a real CoW clone (`FICLONE`/`clonefile`) or - an independent copy, with a test that mutating the reopened working image - cannot change checkpoint bytes. -- Synchronous durability is likely simple and fast enough, and should remain - the baseline while it is measured. Measure the relevant phases separately: - app flush/msync, source synchronization if required, clone or copy, - destination data and metadata synchronization, staging rename, parent - directory synchronization, and DB commit. Measurements should cover - representative state sizes and dirty-page ratios and report tail latency, - not only a best-case reflink time. -- The contract that `clone_image -> Ok` survives an immediate crash is - stronger than the statement in §5 that expensive page writeback may remain - the kernel's work afterward. Until filesystem-specific ordering and sync - requirements demonstrate both claims together, deferred writeback is an - unresolved durability issue rather than work safely removed from the - checkpoint path. -- An asynchronous checkpoint stage need not be designed preemptively. If the - synchronous phase measurements meet the latency budget, keeping the - durability boundary synchronous is preferable. Async staging should be - reconsidered only if measurements show that the required flush and sync - operations materially violate that budget. diff --git a/docs/plans/2026-08-authority-boundary-adr.md b/docs/plans/2026-08-authority-boundary-adr.md index 305e73f..b2e8672 100644 --- a/docs/plans/2026-08-authority-boundary-adr.md +++ b/docs/plans/2026-08-authority-boundary-adr.md @@ -2,10 +2,8 @@ The architecture decision record for how authority — over speculative state, promises, process lifetime, and recovery admission — is owned in the -sequencer. The mechanisms below are landed; the decision history and the -review trail that shaped them live in -[`../review/register.md`](../review/register.md) and the review history it -records. +sequencer. The mechanisms and their reasons below describe the current design; +Git preserves the earlier proposals and review history. ## Context @@ -57,18 +55,23 @@ The lock is released only after every runtime-owned child has actually stopped; a dropped `JoinHandle` detaches rather than stops, so each worker and nested blocking task retains its own lock clone until its closure ends. This prevents two processes on one data directory; it is not distributed -fencing. Cleanup polls every worker concurrently, so one hung drain cannot -hide another worker's terminal exit. Ordinary shutdown has no hard deadline. +fencing. Fresh runtime channels and complete worker shutdown separate runs; +an internal fencing epoch would need a new use case such as overlapping +runtimes or same-process hot replacement. Cleanup polls every worker +concurrently, so one hung drain cannot hide another worker's terminal exit. +Ordinary shutdown has no hard deadline. The reader can cancel a pending RPC read, but awaits any started SQLite append before joining, so the final clean-exit divergence check sees every committed sync. -Runtime construction is prepare → admit → launch: every fallible or awaited -operation happens while zero tasks exist; final admission checks one +Runtime construction is prepare → admit → launch: fallible or awaited +dependency preparation happens before workers launch; final admission checks one consistent fact set; launch spawns every worker in one infallible, non-yielding block, consuming the single-use `RuntimeAdmission` witness. A preparation failure cannot leave a partially launched runtime, and no -refusal or retry can mint the witness. +refusal or retry can mint the witness. This boundary does not promise that +worker initialization succeeds: application restore and catch-up run inside +the launched lane. ### 2. Fact-derived admission and the terminal-fault black box @@ -92,15 +95,23 @@ Every fault whose evidence the boot path reads re-refuses before the first soft confirmation; the residual window is recorded in the threat model. The honesty backstops (rollbackable soft confirmations, the watchdog byte-compare, and the divergence freeze) do not depend on a boot gate. +A durable gate on the previous verdict would require operator acknowledgement +without adding evidence about the current facts. A full integrity sweep would +still miss semantic faults outside its read set. Revisit admission checks for +a specific detectable fault, rather than treating a previous verdict as proof. ### 3. Ordered startup recovery Normal `run` startup inspects local terminal facts, syncs L1, selects a repair from current facts, and checks the result. The flush branch orders flush → sync through the returned safe block → cascade explicitly. There is no -phase driver or progress ledger; the flush witness is a local value. +phase driver or progress ledger; the flush witness is a local value. A restart +must obtain fresh flush/sync evidence; persisting a phase would let it skip +work based on an earlier attempt's observation. Setup/rebuild, maintenance flush, and normal-run recovery retain distinct -typed controllers. The dispatch table, boot-local witnesses, and final +typed controllers because they establish different facts; a universal +controller would represent combinations none of those commands needs. +The dispatch table, boot-local witnesses, and final admission check are owned by [`docs/recovery/README.md`](../recovery/README.md); [`admission.tla`](../recovery/admission.tla) verifies the controller ordering. @@ -145,17 +156,14 @@ write-before-broadcast watermark authorizes an L1 submission; committed version-checked application-input rows authorize the feed output. Effects handed to the network before process termination may still complete remotely. -## Rejected alternatives - -`RunEpoch` (an internal fencing epoch); `EffectGate` / `LiveKernel` (a -universal effect mutex or actor); a generic command controller (one reducer -over setup/rebuild/run/maintenance); a -per-chunk divergence query, provider call, or reader mailbox on the hot -path; a durable recovery-phase ledger; a durable boot gate on terminal -verdicts. Each argument, its evidence, and its revisit trigger live in the -review register's refuted list -([`../review/register.md`](../review/register.md#refuted--do-not-re-propose-without-new-evidence)); -do not re-propose without new evidence. +A global effect mutex or actor would duplicate these owners and put authority +into an additional in-memory coordination layer. A per-chunk divergence read +would see only already-detected accepted-batch divergence, not establish that +every soft confirmation will become canonical; adding a provider call would +also couple acknowledgement latency to L1 availability. The supported reaction +and race bound belong to [I15](../invariants.md#i15-divergence-marker-present--acceptance-frontier-frozen). +Revisit this boundary if a new effect needs authority that its existing owner +cannot establish, or the supported guarantee changes. ## External history @@ -175,12 +183,11 @@ suffix. See the [history contract](../protocol/application-history.md) and ## Performance posture -The product contract is `POST /tx` acknowledgement under 500 ms. Same-host -release sweeps across the cutover found no material regression: ACK p99 at -or below ~50 ms through concurrency 256 with zero rejections, concurrency-1 -HTTP ACK p50 around 13 ms (submit-to-matching-WS-event p50 roughly double — -name which metric "round-trip" means). Same-host numbers are method-specific -regression evidence, never capacity claims: at high concurrency the load -clients contend with the sequencer, so the plateau is machine saturation. A -separate-machine load generator is required for capacity measurement, and -round-trip remeasurement belongs with the public history/API projection. +The product contract is `POST /tx` acknowledgement under 500 ms; the +[benchmark specification](../../tests/benchmarks/BENCHMARK_SPEC.md) defines +the evaluation conditions. The [retained comparison](../review/2026-09-16-track3-validation.md) +records exact revisions, workload, and same-host ACK/WS measurements. They are +regression evidence: client/host contention and excluded startup or backlog +work prevent interpreting them as deployment capacity. Representative latency, +including checkpoint and L1-reconciliation overlap, remains an +[integration gate](2026-07-track3-feed-replay-design.md#remaining-integration-gates). diff --git a/docs/protocol/c-application-binding.md b/docs/protocol/c-application-binding.md index adc06e2..4960464 100644 --- a/docs/protocol/c-application-binding.md +++ b/docs/protocol/c-application-binding.md @@ -38,10 +38,12 @@ output and diagnostic buffers; the adapter copies them before reuse. Fee fields carry base-129/128 exponents. The shared max-fee comparison operates in log space; an application checking balances or charging fees uses a linear -amount. The reference conversion lives in -[`sequencer-core/src/fee.rs`](../../sequencer-core/src/fee.rs). Native and -canonical execution must agree on that conversion, since different amounts can -change rejection decisions and resulting balances. +amount. The exact conversion contract lives on `fee_to_linear` in +[`sequencer-core/src/fee.rs`](../../sequencer-core/src/fee.rs): ports must match +the table, intermediate flooring, and ascending bit order, not just the nominal +exponential formula. [`build.rs`](../../sequencer-core/build.rs) generates the +table and exponent bound. Native and canonical execution must agree on the +conversion, since different amounts can change rejection decisions and balances. The same implementation may be compiled for native execution and the canonical machine. This does not establish equivalent behavior across targets: the diff --git a/docs/review/2026-08-18-over-engineering-review.md b/docs/review/2026-08-18-over-engineering-review.md deleted file mode 100644 index 8cb6195..0000000 --- a/docs/review/2026-08-18-over-engineering-review.md +++ /dev/null @@ -1,46 +0,0 @@ -# Over-engineering review (2026-08-18) - -Full-branch review of the authority-boundary + durable-history-foundation -branch against the project's design goals (readable, auditable, every -mechanism judged against its weight): seven parallel subsystem reviews, each -proposal adversarially cross-examined against the invariants register, the -TLA+ models, and git history, plus an independent premise challenge of the -ADR. 141 mechanisms inventoried: 98 keep, 25 simplify, 6 cut, 12 question. -This review adopted the calibration rule now in AGENTS.md (the complexity -budget belongs to concurrency, mutual exclusion, durability, and hostile-L1 -robustness). - -**Verdict: not over-engineered — unevenly engineered.** The ADR's premise -survived attack; the rejected alternatives left no residue in code. Three -findings cut against "smallest sufficient design": the in-process containment -predicate was still convention at ~11 hand-placed sites (fixed by the -`Authorized` token), ~700 lines of mechanical repetition had accumulated -(harvested), and the lifecycle module implemented the right guarantee with a -heavier representation than needed (re-platformed, then removed — see below). - -**Outcomes** (all landed 2026-08-18/19, each wave adversarially re-reviewed -post-commit; per-item dispositions in [`register.md`](register.md)): - -- Eleven defects fixed (fail-open classification arms, admission bypasses, - containment publication window, snapshot-route gating, typed app-boundary - refusals). -- ~700-line harvest: worker-exit plumbing collapsed with terminality beside - each error type; a real `RuntimeScope` with `ShutdownSignal` reduced to its - name; fee-oracle bootstrap behind its module; recovery type-stack trimmed - to the one `RecoveryProgress` enum; dead surface deleted. -- The `Authorized` externalization token: the containment consult became a - compile-time obligation of the effect functions. -- Module homing: command *brackets* to `commands/` (with their config/error - taxonomy, `RunError` → `CommandError`), the capability substrate alone in - `runtime/`, `L1Config` to `l1/`, the clock to the crate root. -- **Lifecycle arc:** re-platform to singleton + audit trail (decision L1) → - admission gating removed entirely, facts govern (decision L2, 2026-08-19) - → journal narrowed to the terminal-fault black box (decision L3, see - [`2026-08-22-lifecycle-simplification.md`](2026-08-22-lifecycle-simplification.md)). - The surviving design rationale lives in the ADR and the invariants check - policy. - -Still open from this review: the 500 ms latency contract does no design work -(nothing is shaped by it — decide what it is *for*), and the -catch-up ACK-latency measurement owed to the benchmark harness. Tracked in -the register. diff --git a/docs/review/2026-08-22-lifecycle-simplification.md b/docs/review/2026-08-22-lifecycle-simplification.md deleted file mode 100644 index 0a925d2..0000000 --- a/docs/review/2026-08-22-lifecycle-simplification.md +++ /dev/null @@ -1,60 +0,0 @@ -# Lifecycle simplification review — L3 (2026-08-22) - -Fresh-eyes review of what remained of the lifecycle machinery after decision -L2 (admission gating removed, 2026-08-19). Method: two exhaustive read-only -sweeps (a 28-class terminal-fault re-detection map; a journal -weight-and-consumers audit), a six-refuter adversarial verification of every -load-bearing claim before acting, and a fifteen-agent post-landing review of -the diff. - -**Findings that grounded the decision:** - -- The L2 characterization was true: admission was exactly three facts, and - the attempt journal had zero production reads for decisions — its ~490 - production lines across ten files bought only what tracing already - provided, plus the terminal-cause row. In `admission.tla` the journal - variable was a bijective ghost of the controller (removing it left TLC - state counts byte-identical). -- The "terminal faults refuse at re-detection, not at boot" trade is far - narrower than it sounds: everything with durable or deterministically - re-derivable evidence re-refuses before the first soft confirmation, and - the batch/frame spine is re-read within seconds of launch. The verified - residual (cold payload bytes below the lane checkpoint, reachable only via - the WS catch-up window or a pending batch's re-encode; faults with no - durable evidence at all) is recorded as an accepted boundary in the threat - model. - -**Decision L3 (landed):** the journal narrowed to the `terminal_faults` -black box — append-only command+cause rows, best-effort. Principle adopted, -now in the invariants check policy: **telemetry writes are verdict-neutral** -— they sat on the brackets' `?` paths and could change exit codes. -`admit_runtime` collapsed to one consistent inspect + reduce; `RunId` and -the event vocabulary went with the settle plumbing. No boot machinery was -added in the journal's place: neither a durable verdict gate (it needs an -acknowledgement to exit, which carries no information the reducer doesn't -re-derive) nor a boot-time full-integrity sweep (expensive, and blind to -semantic violations outside its read set). - -**Verdict-integrity defects fixed with it** (each adversarially confirmed -first): settle-masking (a settle-step failure could replace a terminal -verdict with exit 1 — and settle was the *sole* exit-code determinant for -most terminal paths); `CommandError::Lifecycle` classified wholesale -terminal (a transient `SQLITE_BUSY` on a lifecycle write paged); signer -misconfiguration classified as unclassified I/O in all three keyed commands. -The misconfig-poison taxonomy question from the 2026-08-18 review closed -with L3: there is no poison to apply; what remained was exit-code accuracy. - -**Post-landing review (8 confirmed / 4 refuted):** one real regression — the -Ok-path divergence refusal had been dropped with `settle_clean`, letting a -clean drain over freshly persisted divergence exit 0 (the one code that -breaks the supervisor's restart-then-refuse rediscovery chain); restored as -an explicit fact check and test-pinned. One missing test pin added; six -doc-staleness items fixed. A claimed re-detection gap at -`finalized_snapshot.inclusion_block` was refuted (the register's L3 block -has the entry). Lesson recorded: when -deleting a mechanism, sweep for its *vocabulary* with review agents, not a -bare grep — one grep pipeline silently returned empty on files that -contained the pattern. - -All landed in the squashed branch commit; per-item dispositions in -[`register.md`](register.md). diff --git a/docs/review/2026-09-03-branch-stocktake.md b/docs/review/2026-09-03-branch-stocktake.md deleted file mode 100644 index 164797a..0000000 --- a/docs/review/2026-09-03-branch-stocktake.md +++ /dev/null @@ -1,847 +0,0 @@ -# Branch stock-take (2026-09-03) - -Stock-take of the authority-boundary branch (PR #28, seven commits on -`f59ec25`) before it leaves draft, asked as: what is this branch for, is it -over-engineered, what next. Method: first-hand reads of the runtime, command, -recovery, lifecycle, and history code; then a read-only fleet of seven -subsystem lenses and five premise challengers (threat model, minimal design, -refuted-list audit, CI root cause, roadmap); then three adversarial refuters -per proposal for the eighteen highest-ranked proposals. Every proposal the -fleet raised is recorded here, including the ones that were not put to a -jury. Per-item dispositions that are actionable now live in -[`register.md`](register.md) (findings 19–31 and the 2026-09-03 refuted -block); this ledger is the full record and will be distilled when it closes. - -**How to read the status tags.** - -- **confirmed** — put to three refuters; at most one refuted it. The recorded - text includes the jury's amendments, which are load-bearing. -- **refuted** — put to three refuters; at least two refuted it on code truth - or on a registered invariant. The reason is recorded so it is not - re-proposed without new evidence. -- **unverified** — raised by one reviewer and not adversarially checked. - Treat as a reviewer's claim: re-verify the cited lines before acting. -- **verified first-hand** — checked directly against the tree during the - stock-take, independent of the fleet. - -## Landed - -Wave 1 (2026-09-03), one theme per commit, on top of the CI fix: - -- **CI red**: both wallet-sequencer binaries style logs only when stdout is - a terminal; the aging-tip scenario asserts exit code 10 instead of grepping - the rendered log (verified: the scenario passes and its log carries no - escape bytes). -- **Prose matches the types**: the token's true scope in `authorize()`, the - ADR, and the register; "boot-local" for the flush witness; the lock - witness's real predicate; three "journal" remnants; the error module's - dated history and removed command; a module doc that argued instead of - instructing; the recovery README's nonexistent type name; a test renamed to - what it asserts. Closes finding 25. -- **Codename sweep finished**: fifteen residual review codes replaced by - the reason or the invariant id. -- **Test-only storage surface gated**: `ensure_open_tip`, - `close_frame_and_batch`, `latest_batch_index`, `ordered_l2_txs_for_batch`, - `promote_finalized` are `#[cfg(test)] pub(crate)`, with their doc links and - the snapshot lifecycle doc pointed at the production paths. Closes findings - 17 and 21 and the second half of 10. -- **Two register nits**: the flusher's healthy retry logs at `warn!` - (finding 4); `fixed_mul`'s comment states what the truncation relies on - (finding 8's comment half). -- **`/tx` 500 body is fixed text**: the application's reason stays on the - lane error and the log (finding 5). -- **Panicking progress constructor deleted**: `ApplicationProgress::new` - had only test callers; `try_new` is the one constructor, tests use it with - `expect`. - -Wave 2 (2026-09-04), the containment diet, each commit refuted by three -read-only reviewers before landing: - -- **Containment writes nothing durable**: the in-scope fault recorder is - deleted; the command bracket's settlement write is the black box's one - writer, so a contained run records one row. The accepted loss (any death - before settlement leaves only the process logs) is stated in the runbook. - `run` logs the last black-box row once at startup, ahead of the preflight, - so the table has its first in-product reader. Closes finding 24. -- **Finalized lease is non-optional**: `acquire_finalized_lease` returns - `FinalizedLease { inclusion_block, dump }`; the impossible-`None` - containment branch in `finalized_state` is gone. Closes finding 26. -- **The reducer's one cycle is cut at the storage boundary**: the - `EnsureOpenTip` phase splits its guard (`TipAlreadyOpen`, retry) and - refuses inside its own transaction rather than commit without a Tip - (`TipMissingAfterOpen`, exit 30); `drive_recovery`'s doc records the - ≤5-phase bound. Closes findings 20 and 23. -- **Workers that only need to stop take the notification half**: the - detector, reader, and fee oracle take `ShutdownSignal` and hold their - own `ProcessLock`; the lane, server, and submitter keep the scope. Their - tests use a bare signal instead of a leaked-tempdir scope. The doc claims - that every worker held a scope clone are restated. Closes finding 22. - -Wave 3 (2026-09-04), the taxonomy, each commit refuted by three read-only -reviewers before landing: - -- **The key file classifies by kind**: a missing, unreadable, non-file, or - non-text batch-submitter key exits 30 like bad key content one call - later, instead of restart-looping at 1; environmental I/O still exits 1. - The typed `BootstrapError::KeyFile { path, source }` names the path and - never the contents. Closes finding 19. -- **Every recovery failure carries one verdict**: `RecoveryFailure::Provider` - is split into `ProviderUnreachable` (retry) and `SignerMisconfig` - (refuse), classified where it is built and pinned, payload and polarity, - against the `BootstrapError` projection. `classify_input_reader`'s doc - says what it can receive and why `Bootstrap` and `Join` flip polarity - between the startup phases and the live worker; both halves are pinned. - Closes finding 27; opens finding 32 (the same L1 misconfiguration exits 1 - under `setup` and 30 under `run`; recorded, not fixed). -- **Exit-code tests table-driven**: the five per-class tests, the verdict - test, the two startup-reader tests, the fee-oracle fatal-math test, and - the two app-bootstrap tests fold into five class functions and one test - asserting class and `is_terminal` per row (71 rows), with the five - verdicts pinned to the integers 10/20/30/40/1 through an exhaustive - match. Two wire-value pins outside the table: `run` on a never-set-up - data directory dispatches to 30 through the real command bracket, and - `stop_expecting_clean_exit` asserts SIGTERM→0 at the healthy stop of - `recovery_after_stale_batches`. Closes the SIGTERM→0 and 30-class halves - of the owed exit-code test. - -Wave 4 (2026-09-04/05), the documentation single-home passes. An -eight-topic read-only mapping fleet first found that the corpus already -elects its homes — the ADR for mechanisms 1, 2, and 4, the recovery README -for the reducer, I15 for the divergence freeze, the register for refuted -proposals, scheduler-semantics for the frame clock — so "the ADR becomes -pointers" was the wrong cut; the fan-out was in AGENTS.md, the check policy, -and the recovery README. Each batch refuted by three read-only reviewers -before landing: - -- **Only what the code enforces**: README's exit-code list gains 40 and - the SIGABRT class; the supervisor is "expected to honor" the contract, not - enforcing it; the lifecycle module doc says which commands preflight - `setup_complete` here and which admit through `commands::setup`; the - frontier writer's cutover forecast, the lane's vacuous "no L1 query", the - reducer's "exactly one phase", the watchdog notes' "structurally frozen" - frontier, the threat model's dangling "non-goals" pointer, and the check - policy's "today" all corrected; I15 gains the clean-exit re-check and - `LocalDivergenceFirst`; I9 owns the accepted false positive; ADR - mechanism 1 owns the hand-placed consult inventory, site by site, with - `runtime/shutdown.rs` and the register pointing at it. Three refuter - passes: the last two amendments of the consult wording were themselves - wrong until checked against a grep of every production consult site. -- **The register owns the rejected alternatives**: the ADR's list is six - names and a pointer; the register's block carries each argument, revisit - trigger, and Evidence line, gains the durable-phase-ledger entry it never - had and the cost datum the distillation dropped (stated as its source - states it, after two passes caught it overreaching), and records that the - boot-gate carve-out is now exercised. Mechanism 3 and G3 reduce to - pointers; the recovery README absorbs the loop, the four-fact inspection, - the witness erasure on Retry and Refuse, and the ≤5-phase bound. -- **AGENTS.md becomes a map**: the hot-path and storage sections are pointer - bullets plus the one rule nothing else owns; every clause only AGENTS.md - carried moved first (into ADR mechanism 4, I2, I3, I20, - scheduler-semantics' revisit trigger, the schema's identity and views - rules, README's closure rule, the design principles); the writer-role - table moves into `docs/invariants.md` corrected — one writer role per - fact, `deployment_identity` and the initial snapshot registration and the - anchor belong to setup, the watermark is shared under I14, startup - hygiene resets leases and collects dumps, the brackets write the black - box. The check policy's admission bullet and the recovery README's - divergence section keep what they own and point for the rest. -- **Six stub ledgers collapse** into the register's Review history table; - the two August ledgers and this stock-take stay for their evidence. The - marker-file containment protocol gets its refuted entry and the 2026-06 - "no architectural restructure" verdict its settled entry. -- **Module docs explain, they do not defend**; the abort bound's number - lives in `runtime/shutdown.rs` and the operator runbook only. - -Not yet landed: the PR title/body, which come last. - -## Verdict - -Proportionate overall, with three named pockets of residue. The maintainer's -fear was quantitatively wrong about scale and right about residue. - -| What | Lines | -|---|---| -| Branch diff | +20,459 / −5,252 across 127 files | -| Authority machinery changed (process lock, scope, lifecycle facts, supervisor) | ~1,700, under 900 production | -| Share of the branch | ~8% | -| Test functions | 418 → 588 | - -The lifecycle machinery that felt over-built was built and removed inside the -branch (decisions L1 → L2 → L3, ~490 production lines); what survived is three -admission facts and one append-only table. The heavy parts of the sequencer -are recovery and storage, which predate the branch and defend in-scope L1 -outage and zombie-transaction threats. - -The three pockets, in descending confidence: - -1. **The black box's write path.** `terminal_faults` has zero production - readers; the in-scope recorder opens a second SQLite writer from inside - containment, is the reason the "arm the watchdog before recording" ordering - hazard exists, and a contained run that drains normally writes two rows. - *(Landed 2026-09-04 as wave 2: the recorder is deleted, a contained run - writes one row, and `run` reads the table at startup — register finding - 24.)* -2. **The exit-code test encoding.** Sixty-seven hand-built projection asserts - pin a pure function four times over, while no test asserts that a real - failing process exits with the promised code; renumbering the terminal code - passes the whole suite. *(Landed 2026-09-04 as wave 3: the table, the - integer pins, and the two wire-value assertions; the stalled-safe-head - (class 20), 40, and 1 failure paths still assert only a non-zero status, - while the backward-clock-jump scenario already pins 20 and the aging-tip - scenario pins 10 — see the owed tests.)* -3. **Documentation fan-out.** Fact-derived admission is described in fourteen - places and the divergence freeze in eleven files; the branch's own L3 - rename missed three "journal" sites. - -Beyond mechanisms, four lenses independently found one defect class: prose -that claims more than the types enforce (see finding 25 in the register). - -**What passed the weight test and should not be cut:** the process lock; the -containment bit and the `Authorized` token at the ack, L1 send, and WS emit; -the pure reducer over one consistent inspection; the `RuntimeAdmission` -witness; the two-second abort watchdog (`/livez` returns 200 unconditionally, -so on a wedged post-containment drain nothing else pages); the clean-exit -divergence re-check. - -## Verified first-hand - -- CI red cause: `tracing-subscriber` 0.3.23 enables ANSI whenever `NO_COLOR` - is unset, with no TTY check (`fmt_layer.rs:743`); the harness inherits the - parent environment and pins only `RUST_LOG`; the e2e assertion at - `tests/e2e/src/test_cases.rs:3151-3157` greps for `status=TipInDanger(` and - the log carries `ESC[3mstatus ESC[0m ESC[2m= ESC[0m TipInDanger(0)`. It is - the only log-grep assertion in the suite. Both wallet-sequencer mains lack a - `with_ansi` call. -- PR metadata is stale: title "Fix storage decode policy", head branch - `feature/review-ledger-and-tracks`, body describing the decode-policy scope - and citing a retired codename; no risk/compatibility paragraph although the - baseline migration is rewritten in place and the `Application` hooks are - renamed. No reviewer comments. No `TODO`/`FIXME`/`unimplemented!` in the diff. -- `/livez` returns 200 unconditionally (`egress/api/health.rs:41-43`). -- `finalized_state` handles an impossible `None` on the `NOT NULL` - `inclusion_block` by escalating to containment (`egress/api/snapshot.rs:139-146`). -- Stale vocabulary: "journal" at `storage/history.rs:201`, - `commands/run/workers.rs:400` and `:680`; an "acknowledge" command at - `commands/error.rs:11`; `docs/recovery/README.md` names - `DangerDetectorExit::DangerDetected` (the type is `WorkerExit::DangerDetected`). -- `RecoveryProgress` derives `Copy`; `docs/recovery/README.md:315`, - `admission.tla:23`, and `recovery/mod.rs:893` call the witness "non-clone". -- `Authorized` is a real signature obligation at three functions - (`submit_batches`, `acknowledge_included`, `send_authorized`); it is minted - and discarded at `snapshot.rs:104/129/182`, `ingress/api.rs:87`, and - `inclusion_lane/mod.rs:211`; the batch-close and reconciliation commits at - `inclusion_lane/mod.rs:165/309` use the raw predicate. -- `terminal_faults` has zero production readers; a contained run appends two - rows (recorder raw cause, then the bracket's prefixed cause); - `latest_terminal_fault` returns the second. -- `ShutdownSignal` on main was 43 lines; `runtime/shutdown.rs` is now 450. - `http.rs` grew a lease-release supervisor with two containment call sites. - -## Confirmed (jury) - -- **cfg-test-gate-ensure-open-tip** (3–0). `Storage::ensure_open_tip` - (`storage/ingress.rs:104`) is `pub` with zero production callers — a new - instance of open finding 17 created by this branch, sitting beside its - guarded replacement. Gate it `#[cfg(test)] pub(crate)` (not private: eight - of nine callers live outside `storage::ingress`), de-link the two intra-doc - references at `ingress.rs:118` and `:486`, correct `:486`'s claim that the - runtime calls this form, and fix `docs/snapshots/lifecycle.md:26-31`, which - still credits it with the production genesis Tip. -- **stale-decision-carries-the-failed-condition** (3–0). - `ensure_open_tip_for_recovery` (`storage/recovery.rs:254-259`) raises - `StaleDecision { expected: Safe, actual: facts.danger }` for a disjunction, - so the `has_open_tip` case renders "expected Safe, found Safe", and - `recovery_tests.rs:114-132` pins that as intended. Add a payload-free - `RecoveryMutationError::TipAlreadyOpen`, split the two checks, add a paired - `RecoveryRetryReason::TipAlreadyOpen` so `classify_mutation` (which today - discards `expected`) carries it to the operator, classify Retry, update the - test and the polarity pin. Roughly +14/−6 across three files. The arm is - production-unreachable under the process lock; this is diagnostics. -- **detector-takes-shutdown-signal-not-runtime-scope** (2–1). - `DangerDetector` and `InputReader` use the scope for exactly one thing, - `wait_for_shutdown`, and each already carries a construction-required - `ProcessLock`. Narrow `start`/`start_preflighted`/`run_forever` to - `ShutdownSignal` and pass `scope.signal()` at the two launch sites. The - change is incomplete without restating three doc comments that assert the - property it relocates: `workers.rs:100-105` and `:1120-1124` ("every spawned - worker retains a RuntimeScope clone, which also retains the process lock") - and `shutdown.rs:96-99` ("workers that touch the data directory take a - scope", already false for the submitter). Restate as: workers that - externalize or contain take a scope; data-directory ownership is a separate - construction-required `ProcessLock`. The dissent would narrow only the - detector. -- **app-with-progress-wrapper** (2–1, a do-not-adopt). Moving - `ApplicationProgress` into a sequencer-owned wrapper (deleting both - capabilities, the seal, three trait methods, all three asserts, ~130 lines) - is not viable: the pair is inside the canonical SSZ bytes - (`examples/app-core/src/wallet_snapshot.rs:41-42`) that `create_dump` writes, - `/finalized_state` streams, and the watchdog byte-compares, and the canonical - machine advances it inside its own state transition; cockroach recovery reads - the clock out of a dump into a wiped database (`commands/setup/mod.rs:433-451`, - `Checkpoint::load`). The rationale for the clock exists at - `docs/snapshots/format.md:113-118`; the missing piece is the composition. - Add one sentence to `docs/protocol/application-contract.md` §4 and to - `ApplicationProgress`'s doc comment, cross-referencing `format.md`, and - extend `format.md`'s "must live in the canonical state bytes" sentence to - cover `executed_input_count`. -- **bound-or-prove-drive-recovery-termination** (2–1). `drive_recovery` - (`recovery/mod.rs:288-313`) is an unbounded loop with one cycle: - `Repaired` + `Safe` + `!has_open_tip` → `EnsureOpenTip` → `Repaired`. No - watchdog exists on the boot path (the scope is constructed in `prepare`, - after recovery). Main could not spin. Take the postcondition, not the loop - bound: after `open_fresh_tip_in_tx` in `ensure_open_tip_for_recovery`, - re-read `has_valid_open_batch` and return a new typed variant classified - `RecoveryError::refuse` (exit 30) — not a `debug_assert` (compiles out in - release; the file's existing postcondition at `ingress.rs:538-541` is one), - and not `StaleDecision` (maps to retry and relocates the non-termination into - the supervisor). Record the ≤5-phase bound in `drive_recovery`'s doc. - Alternative accepted by two jurors: make the reducer's `Repaired`+`Safe`+no-tip - arm a terminal `Refuse`, removing the cycle from the pure function. The - dissent notes the antecedent is unreachable by SQLite semantics; the - counter-argument that carries is fidelity — `admission.tla:271-286` - hardcodes `hasOpenTip' = TRUE`, so TLC proves termination of a model whose - postcondition the code does not enforce. -- **typed-key-source-io-error** (2–1). `resolve_key_source` - (`commands/config.rs:243-254`) returns a bare `std::io::Error` that lands in - `CommandError::Io` → exit 1 ("restart with backoff") for a missing or - unreadable key file, while bad key content in the same file exits 30 via - `SignerMisconfig`. Kind-filter rather than blanket-map, mirroring - `referenced_artifact_io_is_terminal` (`dump_info.rs:49-58`): NotFound, - PermissionDenied, InvalidData, IsADirectory, NotADirectory terminal; - everything else operational (a not-yet-mounted secret must not consume the - do-not-restart code). Prefer a distinct `BootstrapError::KeySourceUnreadable - { path, kind }` over reusing `SignerMisconfig`; never echo file contents. - Roughly +30–40 lines with the predicate and tests. Consider the same - treatment for `create_dir_all` or record why not. -- **table-drive-exit-code-tests** (2–1). Fold the five per-class tests and - the duplicating verdict test (`error.rs:782-828`) into one `const CASES` - table keeping every distinct error shape and every reason string as the - assert message; drop the verdict column (derivable from the bijection); - assert `is_terminal()` per row, which extends a five-shape pin to ~54. Do not - invent rationales for rows that carry none. Realistic saving is 110–150 - lines, not 250. The point is the spend: no test asserts a real failing - process's exit code (the four failure-path e2es assert only `!success()`), - and the `EXIT_*` values appear as literals only at their declarations, so - renumbering `EXIT_TERMINAL` to 31 passes the suite. Add SIGTERM → 0 (the - harness already waits and discards the status) and one 30-class failure → 30 - (`run` on a never-set-up data directory needs no new lever), asserting integer - literals. Correction from the jury: composition is pinned in-crate at - `workers.rs:1209/1228`, `run/mod.rs:237`, `startup_hygiene.rs:168/191`, - `commands/mod.rs:330`, `process_lock.rs:159`; the gap is the process-level - projection at `harness.rs:117-119`. -- **dedupe-terminal-fault-rows** (2–1 for documenting; 0–3 against skipping - by variant). Document the two-row shape at `record_terminal_fault`, in the - ADR's black-box paragraph, and in the runbook's postmortem line, including - the asymmetry: a clean contained drain yields two rows; a controller panic or - watchdog abort yields one. Do not skip the bracket write when the error is - `StorageInvariantViolation`: the recorder swallows both `open_writer` and - `record_terminal_fault` failures into a `warn!`, so the variant does not - prove a row landed, and the post-drain bracket write is the attempt more - likely to succeed after `SQLITE_FULL` or contention (5 s `busy_timeout` - against a 2 s abort deadline). If one row per fault is wanted later, - condition the skip on evidence (an `AtomicBool` the recorder sets on - success), not on the variant. - -## Refuted (jury) — do not re-propose without new evidence - -- **supervise-workers-with-joinset** (3–0). `select_first_exit` and `finish` - deliberately read the same worker return two ways: `WorkerStop::from_select` - maps `Ok(Ok(()))` to `StoppedUnexpectedly` (the runtime is live), while - `from_shutdown` maps it to `Ok(())`. Feeding both phases from one `JoinSet` - of `wait_for_*_shutdown` futures collapses them into the shutdown reading, so - a worker that dies silently while live yields a value `FirstExit` cannot - represent; the available completions are "run with a dead lane" or "drain - and exit 0", the one code `run/mod.rs:82-87` names as breaking the - supervisor's rediscovery chain. `into_supervision(self)` would also drop - `ShutdownOnDrop`, requesting shutdown milliseconds after launch; the - conditional fee-oracle push relocates rather than deletes; and - `FirstExit::detector` plus its mapping tests disappear (one of the four - reasons the register already refuted this shape on 2026-08-23). **Survives:** - the `swap_remove` hazard at `workers.rs:653-655` is real and unwritten in - types; a cleanup-only `JoinSet` built inside `finish`, with the live race - untouched, was not what the jury examined. -- **collapse-preparedruntime-into-boot** (3–0). The load-bearing claim ("zero - tasks during fallible work is reviewer-visible, not type-enforced") is false: - `fn launch(self, _admission: RuntimeAdmission) -> Workers` (`workers.rs:288`) - is non-async and non-`Result`, so a `?` or `.await` between admission and the - six spawns is a compile error today. `async fn boot(..) -> Result<..>` makes - both silently legal, reopening a guarantee registered in the check policy, - ADR mechanism 1, and AGENTS.md, and the linearization argument at - `recovery/mod.rs:477-483`. Under `boot` nothing consumes `RuntimeAdmission` - (`let _a = admit_runtime()?` satisfies `#[must_use]`). Commit 02a2b34 ran - this pass and deliberately stopped here. **Survives:** the test - `preparation_outliving_clean_facts_cannot_launch` (`workers.rs:1161`) never - calls `launch`; rename it to what it asserts. -- **make-authorized-token-uniform** (3–0). `LeasedDumpBody` does not exist - (the primitive is `stream_body(file, guard)` at `snapshot.rs:214`); - `finalized_inclusion_block` (`snapshot.rs:101-120`) has no streaming - primitive to receive a token; and `ingress/api.rs:87` is not a pre-check — - its comment calls it the publication gate, it runs after the lane's ack - resolves, and it immediately precedes the success body that is the soft - confirmation leaving the process, the ack family the ADR names as a token - site. Only the `inclusion_lane/mod.rs:211` half survives (a fast-turn entry - gate; the real ack boundary re-consults at `:248`). **Survives:** the doc - tightening — the token proves "consulted at some point in this borrow", not - "at this effect boundary" (the poster mints at `worker.rs:243` and then does - a chain-id RPC, fee estimation, and a nonce fetch before its own re-checks); - and the coverage claim in the ADR/register should say three compile-forced - primitives plus hand-placed consults at the HTTP 200 gate and the two lane - mutation commits, or the 200 body should take the token (~8 lines). -- **recovery-polarity-unconstructible** (3–0). Diagnosis exact: - `RecoveryError::retry(RecoveryRefusalReason::CanonicalDivergence{..})` - compiles today and would project the absorbing refusal to exit 20 (bounded to - one restart by the next boot's preflight). But `recovery` is `pub mod` and - both enums have public variants, so deleting the `#[from]` impls removes the - shortest spelling, not the route: `RecoveryError::retry(RecoveryFailure::PolicyRefusal(r))` - still compiles, and that longer spelling is the dominant idiom at all 15 call - sites. Cost ~16 renames for a property not achieved. The invoked precedent - (1fcb9aa) deleted the violating value from the type; this does not. -- **single-table-per-error-type** (3–0). `From - for BootstrapError` (`error.rs:671-683`) performs no terminal/transient - decision — it selects among three variants with distinct fields, and the - verdict is taken later over the `BootstrapError` taxonomy, whose variants - have four other producers. "Have both sites read one `is_terminal`" is not - implementable; the result is a third table. Part (b)'s premise is false: - `reader.rs:96-104` states the phase-dependence for `Bootstrap` and `Join`. - Renaming to `is_terminal_in_worker` is wrong for `FlushError` (no - `WorkerExit` arm), and "phase in the type" is blocked because the phase - belongs to the caller (`create_provider(..).map_err(InputReaderError::Bootstrap)` - appears identically in `sync_to_current_safe_head` and `run_loop`). - **Survives:** `classify_input_reader` (`recovery/mod.rs:526`) carries no doc - comment and its `Bootstrap`/`Join` refusals are pinned by no test; and a - pre-v3 InputBox exits 1 under `setup` but 30 under `run` — a separate - finding. -- **flatten-recovery-error-to-one-enum-with-is-retryable** (3–0). A flat - `is_retryable(&self)` must be total over the value, and one variant carries - two verdicts: `ProductionRecoveryDriver::flush` (`recovery/mod.rs:429-438`) - maps `VerifiedSignerProviderError::ChainIdRpc` → retry and `::Create` → - refuse into the same `RecoveryFailure::Provider(String)`. Either resolution - regresses (a bad RPC URL restart-loops forever, or a transient chain-id - timeout pages), nothing pins either arm, and dropping the `Box` risks the - deliberately managed `CommandError` footprint under `result_large_err`. - **Survives:** split `Provider(String)` into two verdict-determined variants - as an independent fix; the wrapper's doc claims a context-sensitivity the - `classify_*` functions do not use. -- **drive-recovery-owns-phase-to-progress-mapping** (2–1). The replacement - is not total: `(RecoveryPhase::Flush, PhaseOutcome::Done)` has no target - because `Flushed { observed_safe_block }` needs a block number the loop does - not hold; the pre-existing implementation of exactly this mapping - (`PhaseCompletion` + `transition_after_phase`) was deleted by `ed41f9b`, whose - message pre-answers the argument. **Survives:** delete - `RecoveryDriver::admitted` (`recovery/mod.rs:283`, a production trait method - whose only implementor pushes a string into a test trace; `drive_recovery` - returns `Ok(())` only from the Admit arm); and the five trace tests exercise - the double's copy of the mapping while production's copy is pinned by no unit - test. -- **merge-stringly-bootstrap-variants** (2–1). `FeeOracleMisconfig` has two - further producers (`setup/mod.rs:144` and `:171`) passing bare strings whose - "fee oracle misconfiguration" words exist only in the variant's Display, and - the black box stores `error.to_string()`, so merging attributes an operator's - Uniswap mistake to a trusted-code fault in the one postmortem artifact. Part - (b) demotes a compile-forced classification to an unchecked `&'static str` - discriminant, the shape the register refuted twice on 2026-08-23. Real delta - ~−15, not −25. -- **terminality-trait-for-workerstop** (3–0). Inverts its goal: - `WorkerExit::is_terminal` already matches all seven variants by name; the - trait admits `fn is_terminal_invariant(&self) -> bool { false }` exactly as - plausibly as `|_| false`. `impl TerminalityOf for std::io::Error { false }` - installs a crate-wide answer for a type the codebase has decided has no - context-free answer (`dump_info.rs:49-58` classifies several kinds terminal); - a `pub(crate)` trait in a public type's bound trips `private_bounds` under - `-D warnings`; and `is_terminal_invariant` is inherent on eight types, only - five of them `WorkerExit` payloads. Delta inverts to +4..+15. -- **prune-duplicate-tla-actions-and-run-check-admission-in-ci** (3–0). - "Nix already provides tlc" is false for CI: no Nix expression or `.envrc` is - tracked, `ci.yml` provisions tools by hand, and `just` is absent from the - `rust` job; TLC also checks the spec against itself and says nothing about - spec-vs-Rust drift. The three deletions are state-space-neutral (`Crash` - subsumes every settle action), but the rationale is false: `decision` - records Retry/Refuse, `DecideRetry`/`DecideRefuse` have distinct guards, and - `InspectRetry` encodes its own comment ("a known local divergence cannot be - masked by the retry edge"). **Survives:** put the 860-state model under CI as - a properly pinned standalone `formal` job (JDK + `tla2tools.jar` pinned by - version and sha256 in `toolchain-pins.env`). - -## Unverified (raised by one reviewer, not put to a jury) - -Re-verify the cited lines before acting on any item below. - -### Runtime authority and containment - -- **drop-in-containment-fault-recorder** (threat challenge, Δ−110). Delete - the `FaultRecorder` alias, field, `set_fault_recorder`, and the recorder - invocation from `runtime/shutdown.rs`; delete `install_terminal_fault_recorder` - from `workers.rs`. Containment becomes three non-blocking steps (set the - cause, arm the watchdog, request shutdown), removing the "either may block" - ordering hazard and the second SQLite writer inside containment. The bracket - write at `run/mod.rs:90` keeps recording every contained fault that settles. - Loss: a fault whose drain hangs past 2 s and exits via SIGABRT leaves no row - — already the documented status for unclean deaths - (`docs/watchdog/operator-deployment.md:392-395`). Note the confirmed - dedupe verdict above: the bracket write is a genuine retry, which argues for - keeping one writer rather than two, and for the bracket one. *(Landed - 2026-09-04, wave 2; register finding 24.)* -- **delete-terminal-faults-black-box** (threat challenge, Δ−330) and - **cut_black_box** (minimal design, Δ−185). Delete the table, its two - triggers, `TerminalFault`, `record_terminal_fault`, `latest_terminal_fault`, - `LifecycleCommand::parse`, `record_terminal_fault_best_effort` and its four - call sites; replace the runbook's `SELECT * FROM terminal_faults` paragraph - with the log-and-exit-code instruction. New evidence against decision L3: - zero production readers; no CLI or API surface; the write path spans four - files; cockroach recovery wipes the database in exactly the incident class - where the postmortem matters. Counter-argument neither author could dismiss: - a Kubernetes Deployment restarts regardless of exit code, so a terminal - fault restart-loops and the first cause could rotate out of the logs while - the black box retains it. Judgment call, not a defect. -- **close-or-correct-the-token-coverage-claim** (threat challenge, Δ+8). - Either make `ingress/api.rs:85-92`'s success response take `Authorized` - (mirroring `acknowledge_included`), or amend ADR mechanism 1 and the - register's settled entry to the true scope. Do not extend the token to the - lane's mutation commits (they sit inside `&mut self.storage` borrows). The - jury's refutation of the "uniform" proposal above endorses this framing. - *(Resolved 2026-09-03 by restating the ADR and the register to the true - scope; register finding 25.)* -- **sketch_boot_shutdown** (minimal design, Δ−260) — a reference sketch that - keeps the lock, scope, token, `ShutdownOnDrop`, reducer, hygiene, lifecycle - facts, and the typed `Workers`/`FirstExit`, and deletes `ShutdownSignal` - (fold into the scope), `RuntimeAdmission`, `PreparedRuntime`/`WorkersConfig`, - and the `WorkerId`/`ComponentShutdown`/`next_component_shutdown`/six-waiter - drain in favour of `Option` fields taken at the winning select - arm and one `tokio::join!` over a generic `drain`. Partly overtaken: the - jury refuted the `PreparedRuntime` collapse and the `ShutdownSignal` half is - contradicted by the confirmed narrowing (the slim half gains two consumers). - The `Option`-take drain is the one part not examined by a jury; it avoids - all four grounds of the 2026-08-23 refutation (named fields, named arms, no - `swap_remove`, no empty-list race). -- **cut_admit_runtime** (minimal design, Δ−60). Delete `RuntimeAdmission`, - `admit_runtime`, `AdmissionChanged`, and the `launch(_admission)` parameter; - keep the fallible-then-infallible ordering. Argument: between the reducer's - `Admit` and launch, the lock excludes every other process and zero tasks - exist, so only wall-clock drift can change the facts, and those arms are the - Retry class re-derived by the detector within one 2 s poll. Adjacent - refutation applies in part: the jury defended the witness as the thing that - makes "launch only from a fresh admit" a compile fact and the basis of the - linearization at `recovery/mod.rs:477-483`. Low priority. -- **taxonomy_min** (minimal design, Δ−145). Regroup `BootstrapError`'s - seventeen variants into verdict-uniform groups (`Misconfig(..)` uniformly 30, - `Transient(..)` uniformly 20, `Recovery`, `SetupNotComplete`, `SetupRefuse`, - `OpenStorage`), move `IdentityError::FirstBootRequiresL1` into the transient - group so `IdentityError` becomes uniformly terminal, delete - `CommandFailureVerdict` (five variants in bijection with five constants, two - consumers), delete `WorkerId` if the drain no longer needs identity. The - taxonomy has exactly one consumer outside the crate (`harness.rs:117`). - Partly overtaken: the jury upheld keeping `is_terminal()` as the thing the - black-box write gates on. -- **leased_dump_inclusion_block_non_optional** (refuted-list audit, Δ−4; - premise verified first-hand). Stop sharing `LeasedDump` between the - finalized and latest lease queries (own return type or `LeasedDump`), and - delete the `let Some(inclusion_block) = leased.inclusion_block else { - contain(..) }` branch at `snapshot.rs:139-146`. The column is `NOT NULL` - (`0001_schema.sql:855-856`); the L3 review refuted a boot assert on it and - left a heavier runtime branch that maps a type artifact to exit 30, contrary - to the check policy's "no `Option`-handling for can't-be-`None`". -- **startup_log_last_terminal_fault** (refuted-list audit, Δ+8). After the - preflight in `run`, read `latest_terminal_fault` and emit one `warn!` when - present. Explicitly not a gate: no acknowledgement, no branch on the value. - The recorded refutation argues against a gate on a verdict; a read that - changes no decision is untouched by it. This would give the black box its - first in-product reader; if declined, say in the register that the black - box is an out-of-process artifact by design. *(Landed 2026-09-04 as - `warn_on_previous_terminal_fault`, ahead of the preflight; register - finding 24.)* -- **startup_hygiene_single_finalized_read** (refuted-list audit, Δ−6). - `require_finalized_snapshot` and `restamp_finalized_promotion` each query - `finalized_dump()`; fetch once in `run_snapshot_hygiene` and pass the row. - Ordering of the five steps is unchanged. -- **reconsider-release-supervisor-weight** (lane lens, ~175 lines). The - supervised lease-release queue in `http.rs:157-235,325` (unbounded MPSC of - boxed closures, a `JoinSet` supervisor with a two-armed `select!`, a drain - awaited inside `axum::serve`, containment on both joins, `ReleaseScheduler` - changed to `Arc` plus a second reporter) defends a real but narrow - hole: a `StatementChangedRows` on release means the leased row vanished, - which nothing else re-detects. Two lighter shapes: (a) keep the - classification, drop the supervisor, accept that a release racing the very - end of shutdown may miss classification (~−120 lines); (b) keep the drain - but move it to the egress snapshot module that owns leases, so `http.rs` - stops hosting a runtime component. The queue is unbounded and bounded only - by concurrent snapshot requests, which have no cap. -- **drop-token-ceremony-at-bool-sites** (lane lens, Δ−10). Overlaps the - refuted uniform-token proposal: the `/tx` publication-gate half is refuted - (it is the ack); the snapshot half survives only as "push the token into - `stream_body`" for the two streaming routes. Separately worth a maintainer - decision: the `/tx` gate returns 503 for an operation that is durably - committed and may still reach L1; the API contract should state the - client-visible semantics of "503 after commit". - -### Error taxonomy - -- **fee_price_stamp_surface_or_drop** (refuted-list audit, Δ+10). - `log_gas_price_updated_at_ms` is written on every refresh - (`storage/fee_oracle.rs:29-40`) and read only by tests, yet the threat model - cites it as the honest telemetry that justifies having no expiry gate. - Either surface the age in `GET /healthz` as an informational field, or - include `retained_price_age_ms` in the existing transient-refresh warn, or - drop the column and fix the threat-model sentence. No threshold, refusal, or - lifecycle effect is proposed. -- **register_provenance_and_wording** (refuted-list audit, Δ+6). Applied in - the register on 2026-09-03: the fee-price-age refutation was added by - 143a290 (2026-08-25) but filed under "From the ADR re-evaluation - (2026-08-01/02)"; the boot-gate refutation's "(in any form)" is broader than - the argument it rests on; the drain-merge entry says "Scope-narrowed - 2026-08-23" while the landing commit is dated 2026-08-24. - -### History foundation and the application boundary - -- **drop-panicking-progress-constructor** (Δ−12). `ApplicationProgress::new` - panics on an incoherent pair and has only test callers; its own doc says to - use `try_new` on the only path that constructs one from data. Delete it; - tests become `try_new(..).expect(..)`. Register finding 17's category. -- **defer-era-newtypes-to-track3** (Δ−110). The schema slice (`history_state` - columns, five triggers, the generation bump inside `cascade_and_reopen`) is - cheap to carry and expensive to retrofit, and should stay. The Rust surface - (`EraId` with Display/Debug/TryFrom, a three-variant parse error, - `RecoveryGeneration`, `HistoryVersion`) has zero production readers; the WS - feed destructures the coordinate away with `..` (`l2_tx_feed/mod.rs:299-315`), - and Track 3 says the era leg is explicitly unconfirmed by the consumer. - Alternative: represent the era as `[u8; 16]` at the storage boundary and let - Track 3 introduce the newtypes beside the wire codec. Low cost either way. -- **drop-uuid-version-variant-checks** (Δ−30). `mint_era_id` - (`open.rs:242-261`) stamps v4/RFC-4122 bits into a random blob, then the Rust - constructor and a SQL `CHECK` verify that self-imposed constant at three - points, for a token whose only semantics is equality. Keep the 16-byte - newtype, length check, and hyphenated Display; drop the version/variant - stamping and checks (+6 bits of entropy). The one argument for keeping it is - a future strict-UUID consumer, a Track 3 wire concern; if kept, reword the doc - from "must carry" to "presentational contract for the future wire form". -- **single-enforcement-for-mapping-contiguity** and - **drop-duplicate-offset-assert** (Δ−15). `attach_executed_inputs_in` - (`history.rs:116-133`) recomputes the exact predicate - `trg_executed_inputs_contiguous` (`0001_schema.sql:562-575`) enforces, on the - accepted user-op path with the latency contract — one `query_history_state` - read plus one `MAX(executed_input_offset)` probe per chunk. Its own comment - says the schema independently enforces the rule. Keep one enforcement point, - preferably the trigger (cannot be bypassed by any writer, aborts the - transaction rather than unwinding a panic through the lane). For directs the - offset is checked a third time by the derive-and-compare below. -- **drop-terminal-fault-typed-reader** (Δ−55). `latest_terminal_fault`, - `TerminalFault`, `LifecycleCommand::parse`, and the two `Malformed` variants - that only report a malformed black-box row exist to serve three test - assertions; an empty cause is already impossible at the engine - (`0001_schema.sql:628-630`). Contradicted in part by - `startup_log_last_terminal_fault` above, which would give the reader a - production caller; decide the black box's reader story once. *(Withdrawn - 2026-09-04: the startup read landed, so the typed reader has its - production caller.)* -- **single-admission-implementation-for-setup** (Δ−25). - `preflight_lifecycle_command` has two callers (`run`, `flush`); setup and - rebuild go through `admit_setup_lifecycle` (`setup/mod.rs:361-388`), which - re-implements the same two facts with different semantics (an - already-complete plain setup is a no-op success there, `NotAdmissible` in - the lifecycle module). Make `preflight_lifecycle_command` return a three-way - admission for setup/rebuild and have setup call it. Also `run` calls the - preflight (which refuses without `setup_complete`) and then - `load_setup_identity` re-checks completion with a different error type; drop - the second check. - -### Lane and storage - -- **narrow-direct-attribution-cross-check** (Δ−8). `persist_frame_direct_sequence` - (`mutations.rs:168-181`) re-derives every direct's sender over the whole - drained range and asserts vector equality with the lane's receipts inside the - reconciliation commit; the lane read the same rows moments earlier. Cheaper - shapes: carry the skipped-submitter count and assert - `executions.len() + skipped == range.len()` plus first/last offset, or keep - the derive behind `cfg(debug_assertions)`. The honest answer depends on the - catch-up ACK measurement the register already owes (5,000 directs over a - 7,200-block jump in one turn). - -### Tests and e2e - -- **gate-remaining-test-only-storage-api** (Δ−40). `latest_batch_index` - (`l1_submission.rs:97`), `ordered_l2_txs_for_batch` (`:129`), and - `promote_finalized` (`snapshot_dumps.rs:182`) are `pub` with only test - callers; `promote_finalized` can promote without the lane's inclusion-block - and lease invariants. Delete the first two (fold into their tests), gate the - third. This is what the in-crate test move was supposed to unlock (register - finding 17). -- **drop-tryfrom-accepts-tests** (Δ−35). Five `*_accepts_*` tests in - `storage/convert.rs` assert that `std::convert::TryFrom` is correct; keep - every `should_panic` twin (they pin the settled decode policy) and - `prepare_time_sql_failures_classify_persistent_in_both_spellings`. Also - `era_id_displays_canonical_lowercase_hyphenated_form` pins a Display string no - consumer parses. Record in the register that the 21 new - `#[should_panic(expected = ..)]` attributes are accepted panic-message - coupling. -- **unify-harness-chain-clock** (Δ+60). Four notions of block time exist: - `SECONDS_PER_BLOCK = 12` duplicated at `rollups.rs:264` and - `sequencer.rs:876`, `LIVE_L1_BLOCK_INTERVAL_SECONDS = 1`, `BOOT_L1_MINE_INTERVAL - = 1 s`, and the sequencer's configured `seconds_per_block = 12`; e2e - correctness depends on their unwritten relationship staying under the 12 s - clock-usability threshold. `advance_live_frame_until_covers` - (`test_cases.rs:480-514`) can drive L1 roughly 20× ahead of the process - clock per iteration. Give the harness one `ChainClock` owned by the devnet - stack, constructed from the value passed to the sequencer, with - `advance(Duration)` and `mine_live(n)` deriving from it, and one post-mining - check `l1_head_timestamp − faketime_now < seconds_per_block` so drift fails - loudly in the harness. Two of the three re-staged scenarios are principled - and strictly stronger (`sequencer_outage_danger_zone_tip_cascade` now asserts - an invalidation; `wall_clock_backward_jump_retries_then_recovers` is the only - per-variant exit-code e2e in the suite). -- **replace-timewarp-tip-injection** (Δ−10). `aging_open_tip_runtime_danger_zone_exit_test` - injects a wedged lane by mining 1,150 blocks with wall time frozen (a chain - 3.8 h in the future), then compensates with `mine_live_l1_blocks(1)` plus an - absolute faketime offset, and greps the log to prove the future-dated view did - not route into the clock-fallback arm. Replace the injection with a - lane-level one (a `--freeze-frame-clock` test dial beside the existing - batch-open dial), advance wall and L1 together, and delete the compensation - and the log assertion. Minimum fix: assert exit code 10 instead of the log. - Also `set_faketime_offset` resets the cumulative counter while leaving an - absolute offset in the rc file, so a later `advance_wall_and_mine` in the - same scenario would regress the child clock. -- **replace-watchdog-sleep-assertions** (Δ−20). Two watchdog tests are - negative assertions implemented as `recv_timeout(250 ms).is_err()`; - unfalsifiable by slowness. Expose `is_watchdog_armed()` under `#[cfg(test)]` - or have the injected abort action record whether a deadline was scheduled. -- **merge-detector-arm-mapping-tests** (Δ−25). Three tests cover the - 13-line `FirstExit::detector`; merge into one table test over the four join - shapes. The composed containment tests the 2026-08-23 refutation protects are - untouched. -- **document_second_half_clock_assumptions** (Δ+12). Record at - `test_cases.rs:3168-3176` that the `+1` alignment margin is not the real - margin (Anvil block timestamps track wall time) and that the single - `mine_live_l1_blocks(1)` refresh has one block of headroom only because Anvil - runs with `--slots-in-an-epoch 1`. -- Also open: `RuntimeScope::default()` (`shutdown.rs:258-267`) leaks one temp - directory per construction via `mem::forget`; worth asking whether a - `(RuntimeScope, TempDir)` guard should be the only shape. - -### Documentation corpus - -- **refuted-evidence-grades** (Δ+10). Give each refuted entry an `evidence:` - line naming the file/line or measurement a reader can re-run; demote entries - that cannot produce one to "declined, no evidence recorded". Restore the - deleted cost datum to the per-chunk divergence-query entry (the - pre-distillation ADR read "every roughly 14-ms user-op chunk"; "14 ms" - appears in zero markdown files now). Name the select arm in the - homogeneous-list entry's title, since the `Vec<(WorkerId, ComponentShutdown)>` - shape now exists in the tree for cleanup. *(Partly landed 2026-09-05: - the ADR-list block carries Evidence lines and the cost datum is restored - as its source states it; the other refuted blocks and the select-arm - title remain.)* -- **collapse-six-stubs** (Δ−110, −6 files). Six of the eight dated ledgers are - 15–20 line stubs carrying a verdict plus a pointer; collapse them into a - "Review history" table at the bottom of the register. Keep the two August - ledgers (they carry the only re-verifiable evidence in the corpus). - *(Landed 2026-09-05; this stock-take stays too, as the branch's ledger. - The six files totalled 106 lines, not 110.)* -- **adr-dedupe-vs-register-and-invariants** (208 → ~70 lines). Each ADR - mechanism is also described in the invariants check policy, AGENTS.md, the - recovery README, the threat model, the runbook, and module docs; five of six - rejected alternatives are also in the register's refuted list, and the two - point at each other circularly. Cut the ADR to context, the policy statement, - and four mechanism names with pointers; move the rejected-alternatives - arguments into the register so there is one home. *(Landed 2026-09-05 - with a correction: the mapping pass found the corpus already elects the - ADR as the home for mechanisms 1, 2, and 4, so the ADR keeps those and - points for mechanism 3, G3, and the rejected list; the register owns the - arguments.)* -- **single-home-divergence-freeze** (Δ−60). `docs/invariants.md:353-372` and - `docs/recovery/README.md:398-415` are the same four sentences; the ADR's G3 - and `AGENTS.md:264` are third and fourth compressions. I15 owns the runtime - reaction and race bound; the others link. *(Landed 2026-09-05.)* -- **agents-hotpath-to-pointers** (50 → ~20 lines). `AGENTS.md:255-304` - restates I2, I3, I9/I15, I17, I18, and the admission policy in fifteen - paragraph-length bullets, violating its own line-475 rule; the good pattern - is already used at `AGENTS.md:118-121` and `:324`. *(Landed 2026-09-05, - with the storage section and the writer table, which moved to - `docs/invariants.md` corrected.)* -- **module-docs-explain-not-defend** (Δ−15). Strike the four defensive - clauses (`workers.rs:21-27` "they are the enforcement, not style"; - `error.rs:10-13`'s dated `RunError` history and stale "acknowledge"; - `shutdown.rs:20-23`; `storage/recovery.rs:15-23`), and fix the three "journal" - usages. *(Landed 2026-09-05; the "journal" usages went in wave 1.)* -- **finish-the-codename-sweep** (~14 one-line edits). Residue: `history.rs:202` - ("L2"), `l1_inputs.rs:41` ("H6"), `wallet.rs:101` ("D10") added by this - branch; `provider.rs:160,234`, `e2e_sequencer.rs:411,429,537`, - `tests/harness/src/sequencer.rs:33,38,302,305,583`, `test_cases.rs:3039,3924` - predate it. Track 6's requirement labels R1–R5 collide with the codename - map's R1–R5. *(Landed in wave 1; a 2026-09-04 re-check of the residue - list found only interval notation left.)* -- **proportionality-measured**. Measured: ~8,026 lines of standalone doc/spec, - ~6,792 comment lines, ~21,200 lines of production Rust, roughly 0.7 prose - lines per code line; the branch's own margin is one doc line per six code - lines. Volume is defensible; the unstated fan-out is not. Either adopt a - single-home rule with a named canonical copy per mechanism, or write down - that redundancy is deliberate and name the canonical copy. *(Decided - 2026-09-05: the single-home rule, with the homes named, is a settled - register entry.)* -- Also open: `docs/plans/` is listed as timeless in AGENTS.md but the tracks - board is a dated status board; the deleted terminal-containment plan's - marker-file protocol has no refuted entry anywhere. *(The marker-file - entry landed 2026-09-05; the `docs/plans/` question stays open.)* - -### CI - -- **binary_disables_ansi_when_not_a_tty** (Δ+4, rank 1). Add - `.with_ansi(std::io::stdout().is_terminal())` to both wallet-sequencer mains - (`IsTerminal` is std). Independent of the test: a daemon writing to a pipe, - file, or journald must not emit SGR escapes. -- **assert_exit_class_not_log_text** (Δ−4, rank 2). Replace the log grep with - `exit.code() == Some(10)`; `TipInDanger` projects to 10 and the clock - fallback to 20. Caveat: 10 does not separate `TipInDanger` from - `ClosedBatchInDanger`; if that matters, keep one check anchored on the - never-styled substring `TipInDanger(` alone. -- **harness_pins_no_color** (Δ+2, rank 3). Pin `NO_COLOR=1` beside the - `RUST_LOG` pins at `sequencer.rs:1300` and `:1394` as hermeticity, not as the - fix. -- **strip_ansi_in_assertion** — rejected by its author; recorded so it is not - re-proposed. -- **centralize_tracing_init_in_run_main** (Δ−10, optional). The two mains are - byte-identical apart from the config constructor; `run_main` already owns the - exit-code contract and could own log rendering. A library installing a - global subscriber is a deliberate boundary decision, not part of the CI fix. - -### Roadmap items (plan, not simplification) - -- **pr-body-rewrite** — a ~270-word draft exists in the fleet output; the - title should name the actual scope and the body must carry the two breaking - changes (baseline migration rewritten in place; `Application` hooks renamed). -- **pre-draft-ci-fix**, **pre-draft-metadata** — the two blockers; plus - `docs/plans/2026-07-coordination-tracks.md`'s "ready for its PR against - main" line and the register's verification date. -- **fold-before-merge** — findings 4 (flusher `error!` → `warn!`), 5 (`/tx` - 500 body echoing `AppError` strings), 17 and the second half of 10 (gate the - test-only storage surface), and the false `debug_assert` comment at - `sequencer-core/src/fee.rs:228`. -- **followup-1-submitter** (findings 1–3, ~250 lines), **followup-2-schema** - (findings 9, 11, 18, while the baseline-rewrite window is open), - **followup-3-measure** (the 500 ms objective and catch-up ACK p99), - **followup-4-harness** (the owed levers and the e2es they unlock, split by - lever), **followup-5/6/7-track3** (typed history foundation and - `GET /history-version`; finalized replay routes, gated on consumer decisions - 2 and 3; the `/ws/subscribe` cutover absorbing findings 6 and 7), - **followup-8-track6** (working-image `Application` API; breaks the same trait - this PR breaks). - -## Recommended split - -**In this PR before it leaves draft:** the CI fix (binary ANSI plus the exit -code assertion), the PR title, body, and breaking-change notes, the two doc -lines that go false on merge, the prose-versus-types honesty sweep (token -coverage claim, "non-clone", the witness wording, the three "journal" words, -the "acknowledge" mention, the README type name, the two-row black-box shape), -the five mechanical register items, gating `ensure_open_tip`, and deleting the -panicking progress constructor. Each touches files the branch already rewrites -and none changes behaviour a reviewer has not already seen. - -**A focused successor PR:** the remaining confirmed items — `TipAlreadyOpen`, -the notification-half narrowing with its doc restatements, the kind-filtered -key-file error, the tip postcondition refuse, the exit-code table with the two -process-level assertions — plus whichever unverified runtime items survive -their own re-verification (the lease-release supervisor's home, the -non-optional inclusion block, the in-scope recorder). These change behaviour -or exit-code classification and deserve their own adversarial pass and tests. - -**Later, in order:** the doc single-home passes; submitter pacing; schema -hardening before first deployment; the latency measurement; the harness -levers; Track 3; Track 6. diff --git a/docs/review/2026-09-09-application-lane-dex-review.md b/docs/review/2026-09-09-application-lane-dex-review.md deleted file mode 100644 index a4bfa34..0000000 --- a/docs/review/2026-09-09-application-lane-dex-review.md +++ /dev/null @@ -1,389 +0,0 @@ -# Application, inclusion lane, and DEX integration review - -Status: accepted and implemented locally. The investigation below records the -pre-change evidence. Current contracts live in -[application-contract.md](../protocol/application-contract.md) and the snapshot -docs. The reference C bridge is ported on a separate integration branch, not -cherry-picked into the main implementation. - -## Accepted follow-up decisions - -- Keep `Send`; remove unused `Clone + Sync`. Independent state forks use - checkpoint/restore, not a required `Clone`. A non-Clone, non-Sync runtime - fixture exercises actual preparation and launch. -- `progress()` returns engine-owned count/clock by value. Apply hooks advance - it; the shared boundary preflights overflow and verifies successful - transitions. Validation distinguishes rejection from fatal engine failure. -- Mutable `create_dump` permits backing-resource changes while preserving - logical state. File and directory prefixes remain opaque. Durable immutable - checkpoints, independent restores, and source-deletion independence are - required; public flush/clone/reopen machinery is deferred. -- Recovery state and canonical comparison bytes may differ. The wallet uses - the same binary SSZ file for both; the DEX design compares `M` with its - canonical drive. The current watchdog's drive extraction remains follow-up. -- Canonical inspection is a separate trait. Lane bookkeeping is simplified - without changing ordering, attempt limits, commit/ACK, or drain/promotion - atomicity. CORS and Lua executable parity remain a separate focused commit. - -The private DEX scheduler and native engine are still unavailable. Reference -bridge tests verify the proposed seam, not private engine conformance. DEX -conformance and the [Track 3 history API](../plans/2026-07-track3-feed-replay-design.md) -remain follow-ups. This review establishes reference integration coverage; -it does not establish that the Application surface is production-proven. - -Reviewed on 2026-09-09: - -- Local PR #28 work: `7d6238ab2ae1aadee6a858806747b042d87b896c`. -- DEX integration branch: `c31bf18413d8e9677ad9d663b9a045f44fafa4a3`, - compared from common ancestor `993d3310ac0da63380a62d6d3c93bff22d2317ff`. -- [C application bridge, PR #32](https://github.com/cartesi/sequencer/pull/32): - `0fa1755a882ce8aeeb6ba7877ba4ea7c479da9d1`, based on `01030fd7107e360d9a4d7e0e2852183eadb4b327`. -- Uncommitted lane annotations in the main checkout were read without alteration. - Its unresolved index entry was not resolved by this review. - -The actual DEX bridge and C++ scheduler have not been shared. PR #32 is evidence -of the intended integration approach, not proof of the private implementation. - -## Findings and proposed Application boundary - -### 1. Native progress ownership fits the supplied bridge - -The current trait calls progress scheduler-owned, stores it inside the app, -requires immutable and mutable references to it, and controls the latter with a -separate capability. Execution then checks that application hooks did not change -it and that the getter and mutator agree. See -[`Application`](../../sequencer-core/src/application/mod.rs). - -PR #32 describes a different, coherent ownership model: native execution advances -and persists count and clock, and two C functions read those values. The adapter -holds an opaque engine pointer. Mapping this directly onto the current trait -would conflict with the assertion that native execution must leave progress -unchanged. A Rust shadow value would introduce two representations of the same -fact and require synchronization at save/load boundaries. - -Recommend treating `Application` as the complete execution engine: - -- Read progress by value: `progress(&self) -> ApplicationProgress`. -- Let successful native execution update its own complete state, including - progress. The protocol still defines the transition. -- The shared Rust execution boundary preflights the checked count successor, - invokes execution, checks the exact expected count/clock after success, and - returns the pre-execution offset. -- Remove the mutable progress accessor and both capability types. Remove - progress-only validation-purity and post-error coherence checks; an execution - error already defines no successor and the instance must be discarded. -- Preserve `clock = max(previous_clock, input_block)`, count zero implying clock - zero, durable round trips, replay attribution checks, and deterministic - application behavior. - -PR #32 can implement the value read using its existing two scalar getters. No -C ABI change is required for that read, and the single-owner handle prevents -concurrent mutation between them. Rust implementations may share a small -progress-transition helper; the C++ implementation follows the same formula. - -This deliberately gives up capability-enforced routing for safe Rust callers. -Production call sites must use the shared boundary and conformance tests must -verify the native contract. The current capabilities never prove validation: -`execute_valid_user_op` accepts a publicly constructible `ValidUserOp` because -trusted replay needs to bypass admission. They are not an FFI isolation boundary. - -The earlier blanket rejection of `AppWithProgress` in the review register was -too broad. A wrapper shared by canonical execution, the lane, and recovery could -own progress correctly, with codecs preserving existing canonical bytes. The -invariant rejects an off-chain-only sidecar, not composition. Nevertheless, that -would require coordinated ownership and codec changes, while the native-owned -value interface fits PR #32 without inventing a second owner. Prefer the latter. - -### 2. Remove unused Clone and Sync requirements - -The entry chain in [`harness.rs`](../../sequencer/src/harness.rs), -[`run/mod.rs`](../../sequencer/src/commands/run/mod.rs), and -[`workers.rs`](../../sequencer/src/commands/run/workers.rs) requires -`Application + Clone + Sync`. Prepared runtime state holds no application value; -the lane constructs the application inside its blocking thread. - -This already has a concrete cost in PR #32: `EngineApp` supplies a panicking -`Clone` and an `unsafe Sync` justified by the current runtime never sharing it. -The C ABI forbids concurrent use of a handle. A public `Sync` implementation -licenses shared-reference calls from multiple threads, so current call-site -discipline is not a sound general justification for that implementation. - -Removing all five `Clone + Sync` bounds compiled across the complete workspace -and all targets in an isolated archive. Remove the bounds and those adapter -workarounds. PR #32 explicitly permits moving handles, so `Send` is supported by -this ABI; there is no need to redesign thread confinement for this integration. -Its necessity belongs to the async host, rather than canonical transition -semantics, if a future engine is thread-affine. - -### 3. Preserve all three validation outcomes - -Rust validation currently returns only success or `InvalidReason`. PR #32's C -validation function returns OK, INVALID, or INTERNAL. The adapter therefore has -to abort when validation fails internally. - -Recommend `Result`, where `ValidationOutcome` is -`Accept` or `Reject(InvalidReason)`. The shared protocol guard still checks -`max_fee >= current_fee` first. Expected rejection remains a nonmutating response; -internal failure propagates to the host's failure policy. - -Keep validation and execution separate for now. Catch-up currently replays -`ValidUserOp { sender, fee, data }`, without nonce or max-fee fields. Combining -the methods would require rebuilding original operations and checking their -admission result on replay. That is possible, but PR #32 already supports the -split, so it has no demonstrated benefit here. - -The bridge's rationale for aborting execution errors predates the local scheduler -fix: the reviewed scheduler propagates application errors instead of swallowing -them. The adapter can now report execution failures as `AppError`; process policy -belongs to the host. Exceptions must still never unwind across the C ABI. - -### 4. Fix the rejection contract before asking clients to implement it - -The current application contract recommends `ExecutionOutcome::Invalid` for -malformed application payloads, but application execution hooks cannot return -that type. The wallet consumes nonce and fee before decoding a method; malformed -methods and unsuccessful business operations return `Ok` with no outputs. -Malformed or unsupported direct inputs likewise execute as counted no-ops. - -The distinction to document and test is: - -| Outcome | Included? | Progress | Meaning | -|---|---|---|---| -| Admission rejection | No | Unchanged | Invalid nonce, insufficient fee balance, or max fee below frame fee | -| Included business failure/no-op | Yes | Advances once | Method fails under application rules; fee/nonce behavior remains the application's defined included semantics | -| Internal execution failure | No canonical successor | Instance discarded | A bug or unrecoverable execution failure, never an ordinary bad DEX order | - -Do not change fee/nonce or rejection semantics as part of this interface cleanup. -Also correct two smaller documentation errors: ingress does enforce the declared -payload bound, and the wallet no longer repeats the shared max-fee check. - -### 5. Keep checkpoint requirements, remove representation assumptions - -Retain the simple lifecycle: load state; create an immutable, crash-durable -checkpoint; locate its canonical file; dispose of obsolete state. A returned -checkpoint must be durable before SQLite references it. HTTP readers retain -their leases, including for directory-shaped dumps. - -PR #32 describes private live mutations over an immutable source image and an -explicit durable save. It does not require a caller-managed write-through -working image. The Track 6 draft's `open/flush/clone` lifecycle should not be -imposed on this integration. CoW and sparse writing can remain engine details. -Measure checkpoint tail latency before adding asynchronous staging. - -The sequencer-owned outer dump is a directory containing `info.toml`. Its -application prefix can already be treated opaquely by create/load/delete and -canonical-file lookup. Explicitly allowing that prefix to be either a file or a -directory would accommodate PR #32 without requiring a dummy subtree. Pin this -with a single-file lifecycle fixture if adopted. Preserve directory support. - -The load contract should explicitly state how an instance remains usable after -its source checkpoint is collected. PR #32 promises that property through its -private mapping; other implementations must provide equivalent independence. -Do not generalize that implementation into an assumption that every app consists -of one mapped file. - -Durable deletion is unnecessary for the sequencer's safety: SQLite references -are removed first, and orphan files after a crash are acceptable. Durable creation -remains necessary. Preserve meaningful load-error classification: PR #32 maps -every IO_ERROR to `ErrorKind::Other`, losing the missing-artifact distinction the -current host uses to refuse a broken referenced checkpoint. Carry the needed -typed distinction across the ABI rather than parsing diagnostic text. - -### 6. Put optional capabilities on their actual consumers - -`export_state` has no generic Rust consumers; keep human-readable debugging on -the concrete app. `canonical_snapshot_bytes` belongs to canonical inspection, -not every native engine adapter. The Rust scheduler's inspection method should -require an inspection capability where used. A separate C++ canonical scheduler -can provide inspection itself while the sequencer serves the canonical file. -Do not turn the current default runtime error into a globally mandatory bridge -method merely to make the trait uniform. - -Execution and durable checkpointing are distinct contracts with actual distinct -consumers. Separating those traits is reasonable if it makes the implementation -clearer, but no generalized capability or lifecycle framework is needed. - -## Inclusion lane - -No new supported canonical-order or acknowledgment correctness defect was found -in the reviewed lane, replay, snapshot, and storage paths. - -The useful simplifications are local: - -1. Collapse `ChunkOutcome`, the accepted-count return, and `FastTurnSummary` - into one bounded-turn result. Preserve the cap on attempted requests, so a - rejected flood cannot starve reconciliation; preserve commit-before-ACK. -2. Store `next_safe_input_index` rather than the previous complete - `last_drained_direct_range`, whose end is the only subsequently used value. - Advance it only after a successful commit. -3. Consider one storage frame-transition operation with an optional promotion - argument, replacing the duplicated promoting/nonpromoting entry points. - Drain attribution, progress mapping, frame creation, and promotion must - remain in one transaction. The observation accumulator remains useful. - -Answers to the in-tree annotations: - -- `frontier_min_interval` limits SQL observation frequency (default one second). - User-op chunks continue during that interval. The five-safe-block criterion - controls logical frame advancement and deposit visibility; it is a separate - policy. Removing it changes behavior while saving little code. -- The durable divergence check must precede the clock threshold, including when - no new frame is due. It detects a poisoned accepted-batch projection. -- The reintroduced `is_storage_invariant_contained` check belongs to the old - global terminal mechanism removed from the reviewed HEAD. It historically - checked a different signal, not the same database fact twice. -- The divergence check is polling-based diagnosis. SQL does not fence every - post-marker ordinary frame/user-op append; do not describe it as doing so. - -Whole-range reconciliation remains appropriate under the explicit capacity -assumption: fix one safe frontier, execute its direct prefix, and atomically -attribute it to the advanced frame before later user ops. Paging bounds payload -scratch memory, not the full receipt vector or turn duration. Reconciliation, -checkpoint creation, and GC can delay overlapping requests. Measure them with -the DEX engine before introducing preemption or resumable ordering state. - -## DEX branch parity - -The five feature-only commits do not establish a large missing runtime surface. - -| Capability | Local status | -|---|---| -| WS user-op nonce, frame safe block, batch nonce | Present; same wire fields | -| Direct-input input index, batch nonce, block timestamp, transaction hash | Present; same wire fields and encodings | -| WS catch-up close reason carrying live-start offset | Present | -| Avoid historical getLogs when the safe input count has not changed | Present | -| HTTP transaction request and acknowledgment | Same schema | -| Browser POST /tx | Works; CORS policy differs | -| Explicit Lua 5.4 executable selection | Not carried over everywhere | - -The CORS discrepancy is concrete. The feature branch allows any origin, POST, -and request headers, with a 3600-second preflight cache, on ingress only. Local -`http.rs` applies `CorsLayer::permissive()` to the merged ingress/egress router, -including internal reads, and configures no preflight max-age. The branch's -egress-isolation expectation fails locally. Restore the narrow scope without -waiting for a port split; carry its error-path and preflight contract tests. - -Reconcile Lua executable selection with the supported Nix/native environments; -blindly replacing every invocation can break an environment that exposes its -pinned Lua 5.4 as `lua` rather than `lua5.4`. - -These larger items are absent from both public API variants, not lost fork -features: - -- Recovery-aware public history: local storage has canonical count and - era/generation, but WS and snapshot headers still expose physical rowids. -- Full remote recovery-dump export: local dumps support multiple files, while - snapshot HTTP routes serve one canonical file. -- Public application-output delivery: current WS sends inputs and POST returns - the inclusion acknowledgment, not notices or vouchers. - -Keep the established history-protocol work separate. Add archive export or an -output stream only for a concrete consumer requirement. PR #32's C engine -binding is also separate integration work, not already provided by the CORS -branch. - -## Scheduler integration and approach - -Sharing the DEX scheduler between its canonical machine and cockroach recovery -would remove an important independent implementation. The Rust scheduler should -not be presumed more correct. However, compile-time selection alone does not -remove all possible disagreement: the live lane and SQLite acceptance/nonce -projection still encode protocol assumptions and must agree with that scheduler. - -The eventual scheduler interface should be separate from per-application input -execution and should cover the actual recovery needs: restore a checkpoint, -seed pending directs, process L1 inputs, drain at the recovery stop, and return -application state and next batch nonce. Determine that interface from their real -scheduler rather than encoding the Rust implementation's convenience methods -into a new requirement now. - -Future microbatch priority is a different decision. It can select an order among -pending, unacknowledged operations before validation/execution and persist that -chosen order; it need not create a protocol frame every 500 ms. Preserve direct -drain attribution and validate sequentially against the chosen order. Under the -current nonce contract, a cancel at nonce 11 cannot simply move before a new -order at nonce 10 from the same sender. A full 500 ms collection window also -spends the entire advertised acknowledgment budget before execution and commit. -These are design constraints for that later work, not reasons for a policy -framework today. - -Recommended sequence: - -1. Simplify Application ownership and error outcomes, remove unused bounds, - and port the reference C bridge alongside the Rust wallet. Preserve canonical - bytes and existing rejection behavior. -2. Polish the lane's local bookkeeping with its current ordering intact. -3. Close the narrow CORS/tooling parity differences in a focused integration - change. Keep public history cutover as its own API change. -4. Integrate the real scheduler once available, then consider measured ordering - policy requirements. - -The main process improvement is to use the native adapter as an acceptance test -for an interface change. One source engine exercised through Rust, the C ABI, -replay, dump/load, and canonical execution exposes more useful integration -mistakes than additional capabilities around two self-trusted fields. Include -admission rejection, included no-ops, exact progress, output order, snapshot -immutability, and error classification. Cross-language scheduler tests should -compare behavior against the protocol, not automatically bless either side. - -## Validation and limits - -All commands used the pinned environment via -`direnv exec /Users/gcdepaula/projects/cartesi-dev/sequencer` from the reviewed -checkout unless stated otherwise. - -- `cargo check --workspace --all-targets --locked --offline`: passed. -- `cargo test --offline --locked -p sequencer --lib ingress::inclusion_lane -- --nocapture`: - 50 passed. -- `cargo test --offline --locked -p sequencer-core -p app-core --lib`: - 107 core and 23 app tests passed. -- Isolated archive, removal of all five `Clone + Sync` bounds: - full workspace/all-target check passed. -- Isolated archive, two real-listener tests adapted from the feature branch's - CORS expectations: reproduced both differences (`/livez` returns allow-origin - `*`; `/tx` preflight has no max-age). These are expected repro failures, not - failures of the existing suite. - -The existing cheap-app 5,000-direct backlog test took about 47 ms in this run. -That is neither a DEX benchmark nor a concurrent ACK latency measurement. -No private DEX engine/scheduler, native bridge runtime, or canonical-machine -end-to-end execution was validated in this pass. - -## Implementation validation (2026-09-09) - -The local implementation passes `cargo check --workspace --all-targets`, -strict workspace/all-targets/all-features Clippy, and formatting checks. -`cargo test --workspace --exclude canonical-test -- --test-threads=1` passed -697 tests; one pre-existing doc example remains ignored. Wallet SSZ golden -bytes are unchanged. The watchdog Lua 5.4 suite passed 62/62. - -New coverage includes pure validation, native progress mismatch and overflow, -fatal validation propagation, no reads after a failed apply hook, independent -wallet and file/directory checkpoint restores, atomic frame/promotion rollback, -a Send-but-not-Clone-or-Sync runtime, ingress-only CORS on success and rejection, -and POST preflight policy. The CORS fixture initially queried an uninitialized -snapshot service and correctly triggered a terminal fault; it now seeds the -required finalized checkpoint. An unrelated lock-lifetime test failed once in -a parallel run and passed alone and in the final serial workspace suite. - -CORS and Lua invocation changes close the reviewed public branch's remaining -narrow parity gaps. This does not deliver the separately planned history API, -remote full-dump export, output stream, or private scheduler integration. - -Real-process follow-up rebuilt the devnet binaries at `5773b833` and passed -`restart_and_replay_test` with deposit, transfer, withdrawal, and restart replay. -`setup_recovery_round_trip_test` restored checkpoint `B=26, N=1`, accepted the -continuing nonce, passed its anchor/divergence checks, and finalized the resumed -snapshot at block 35. The test then failed during watchdog initialization: -`expected "archive_version" 7 (got 6)`. The installed in-process Lua Cartesi -binding expects the newer archive, while the repository image is pinned to -CM 0.20. The verified CM 0.20 CLI shim does not affect that Lua binding. This -E2E remains incomplete; no emulator pin or image was changed to make it pass. - -The separate reference bridge integration also exposed a concrete host need: -lazy genesis construction must be fallible, and a custom CLI must reuse the -library's command-task exit projection. Its integration branch makes the -factory return `Result` and exposes `run_command` for parsed -commands. Completed setup still avoids opening the original genesis. This -keeps file-load errors in the existing bootstrap error policy without adding -a second setup-admission check in the C host. diff --git a/docs/review/2026-09-16-track3-validation.md b/docs/review/2026-09-16-track3-validation.md index d44cdb7..56aeaa0 100644 --- a/docs/review/2026-09-16-track3-validation.md +++ b/docs/review/2026-09-16-track3-validation.md @@ -5,13 +5,15 @@ Scope: validate application-history commit a complete cold replica, and a same-host latency comparison against `91e25780854bb641c63135751f951f9f7ee1e744`. +Retained for the [Track 3 integration gates](../plans/2026-07-track3-feed-replay-design.md): +this is the wallet baseline against which native-engine and deployment results +can be assessed. Replace or delete it when those decisions no longer use these +measurements. It describes the named revisions, not ongoing validation of HEAD. + ## Environment and canonical agreement -The shared development flake was switched to emulator 0.20.0 using its previously -recorded source, generated-files, and uarch hashes. The CLI, Lua module, and -native library all resolved to the same Nix package. Existing unrelated Foundry -edits were preserved; the shared flake lock was unchanged. This environment edit -lives outside the sequencer repository. +The CLI, Lua module, and native library used emulator 0.20.0 from the same Nix +package in the shared development environment outside this repository. Rust 1.95.0, Lua 5.4.7, and Foundry 1.5.1 were used. A fresh devnet canonical image was built from this checkout with the pinned cross image and kernel; @@ -34,14 +36,12 @@ All four selected canonical-machine gates passed: | `setup_recovery_round_trip_test` | Real `/finalized_snapshot` download, database wipe, `setup --recovery`, resumed execution, and independent CM comparison | 15.36 s | These are selected integration gates, not a claim that the entire E2E suite or -private DEX adapter was tested. The existing 693-test host-suite result belongs -to the implementation record in the register. +private DEX adapter was tested. ## Cold replica -Added `cold_replica_snapshot_backlog_live_recovery_test` and two small wallet -replay helpers. The scenario passed in 18.12 seconds, including its test-owned -120-second deadline. Its claims come from HTTP headers and consumed inputs, +`cold_replica_snapshot_backlog_live_recovery_test` passed in 18.12 seconds, +including its test-owned 120-second deadline. Its claims come from HTTP headers and consumed inputs, without querying storage for the consumer's history identity. The test restores a nonempty tar archive, deletes the downloaded source, and @@ -58,17 +58,6 @@ The expected replacement branch retains the accepted prefix and replays only its retained L1 directs. Optimistic transfers disappear, and a new transfer at the recovered nonce succeeds. -## Tooling fixes - -- `just doctor` preserves Lua's configured search paths, matching the production - watchdog. Its forced Linux-only environment variables hid the Nix Cartesi - module. The corrected doctor loads both lcurl and the new machine image. -- Benchmark CLI/recipe defaults use `max_fee=2000`. The former 1200 default was - below the self-contained frame fee of 1356, rejecting every request. The stale - `--from-offset` help example was also corrected. An initial benchmark attempt - with 1200 was discarded during warmup; both compared revisions use an explicit - 2000 limit. - ## Latency comparison Four release-build runs used an ABBA order: baseline, current, current, @@ -77,6 +66,8 @@ baseline. Each had 5 seconds of warmup and a 45-second measured window, explicit max fee of 2000, a 3-second request deadline, and a 5-second WS deadline. The host was an Apple M5 Max (18 logical CPUs, 36 GiB) running macOS 26.6.2. No builds, correctness tests, or injected network shaping ran during measurement. +An initial attempt with max fee 1200 (below the frame fee of 1356) was discarded +during warmup; all four compared runs used the explicit 2000 limit. Both exact revisions used their matching SDK/protocol and the same fresh machine image, Anvil fixture, and toolchain. Per-request latency excludes funding, diff --git a/docs/review/README.md b/docs/review/README.md new file mode 100644 index 0000000..93c8e98 --- /dev/null +++ b/docs/review/README.md @@ -0,0 +1,60 @@ +# Review notes and their lifecycle + +Reviews help us investigate code and hand work over. Their conclusions must +survive where future contributors need them; their working notes need only +survive while useful. Git is the archive. + +## During a review + +Create a note only when the investigation or handoff needs one. A small review +can live in the conversation, PR, or commit description. Commit a note when +sharing the ongoing reasoning is useful; committing it does not make it a +permanent document. + +Notes are mutable. Correct, consolidate, and remove superseded claims instead +of appending a conversation transcript. Record the reviewed revision, scope, +evidence, uncertainty, and next question. Check behavior in code and tests; +documentation and earlier verdicts are leads, not proof. Separate observed +defects from coverage gaps, accepted tradeoffs, and unverified hypotheses. + +## Closing or handing off + +Give each surviving conclusion one home: + +| Conclusion | Home | +|---|---| +| Current contract, invariant, or durable design reason | Its owning design document, source comment, or test | +| Unresolved defect or investigation | [Current register](register.md), unless already owned by an active plan | +| Coordinated implementation or integration work | The relevant active plan; link to it from the register if useful | +| Measurement or other evidence still used by a decision | A dated record with the consumer, exact revision, method, result, limits, and retirement condition | +| Completed discussion, superseded proposal, closed finding | Git history; remove it from the working tree | + +An unresolved entry states the impact or question, supporting source/test, +last-checked date and revision, and next action or revisit condition. A shared +verification stamp is sufficient for entries checked together. A missing test +needs a specific unverified behavior; an absent test name alone creates no +obligation. Proposed machinery needs an invariant and supported assumptions. + +Preserve the reason for a deliberate tradeoff at its design seam, together with +the assumptions that would change the decision. A previous rejection is not a +permanent ban on an alternative. Avoid a second register of settled/refuted +decisions, closed-item tombstones, or review-codename maps. + +Delete the finished note after checking that useful unresolved work and unique +evidence have a home, and update incoming links. Apply the same rule to +completed plans. Revisit retained notes when related code changes, at handoff, +or when they stop informing a decision; no calendar-driven archive is needed. +Cleanup does not imply the remaining code work is complete. + +## Recovering earlier reasoning + +The last version of a deleted note remains available without loading it into +every agent's baseline context: + +```sh +git log --all -- docs/review/FILE.md +git show REMOVAL_COMMIT^:docs/review/FILE.md +``` + +Treat that version as evidence about its reviewed revision. Recheck its +premises before carrying a conclusion into current work. diff --git a/docs/review/register.md b/docs/review/register.md index 8942aa8..90c9713 100644 --- a/docs/review/register.md +++ b/docs/review/register.md @@ -1,709 +1,123 @@ -# The Review Register - -The distilled outcome of every dated review ledger in this directory: what -is still **open**, what was **settled** (with its reasoning's current home), -and what was **refuted** and must not be re-proposed without new evidence. -Check the open section before touching related code; check the refuted -section before proposing a simplification or a new mechanism. Every -review's date, scope, and verdict is in the Review history table at the end -of this file; the two August 2026 ledgers and the 2026-09-03 stock-take stay -as separate files because they carry measurements and refutation evidence -recorded nowhere else. Process detail beyond that lives in git history. - -Statuses of findings 1–18 were verified against the tree on 2026-08-25. -Findings 19–31 and the 2026-09-03 refuted block come from the -[branch stock-take](2026-09-03-branch-stocktake.md), which records every -proposal that review raised, including the ones no jury examined. - -The 2026-09-07 maintainer-approved simplification changes two earlier -premises: terminal runtime faults abort immediately, and startup recovery -uses an ordered procedure. Earlier dated containment/token and phase-driver -entries below record their historical rationale; those mechanisms and their -test-only obligations are superseded by the current ADR and recovery design. - -## Findings - -Code findings, oldest first (file references are starting points, not exact -lines). Numbers are stable identifiers: a closed finding keeps its number and -is reduced to a one-line closure note, so citations in git history and the -remaining dated ledgers stay valid. - -1. **Submitter confirmation-timeout defeats pacing** — a watch timeout maps - to a successful `Submitted` tick, so the next tick immediately re-sends - the same payloads at the same nonces (usually "replacement underpriced") - before any sleep. `l1/submitter/poster.rs` + `worker.rs`; add a distinct - outcome that sleeps. -2. **A transient `SQLITE_BUSY` costs the submitter a full respawn** — 50 ms - reader `busy_timeout` plus every non-poster error ending the run. It now - classifies restartable rather than terminal, but the respawn+recovery - cost stands. Retry BUSY or use the writer-grade timeout for these reads. -3. **An undecodable own-sender payload stalls submission for the safe-lag** - — the poster hard-fails decode where both scheduler mirrors - skip-and-continue, so an operator's manual tx from the submitter EOA - wedges ticks until the block passes the safe head. Skip undecodable - own-sender payloads. -4. **Closed** (2026-09-03): the flusher's healthy retry pass logs at - `warn!`, not `error!`. -5. **Closed** (2026-09-03): the application-error 500 body is the fixed - "application internal error"; the reason stays on the lane error and the - log. -6. **WS session hygiene** — a mid-session transient read error tears down - without a close frame. Ahead-of-head admission is closed (2026-09-16): a - typed HTTP409 refuses it before upgrade. -7. **Closed** (2026-09-16): mandatory era/generation/application-count claims - reject resume across recovery; snapshot headers provide cold-bootstrap - coordinates. Current application suffix replacement is atomic with the - generation bump. See the [history contract](../protocol/application-history.md). -8. **Fee-determinism contract under-specified** — the LSB-first - floor-after-each-multiply order is implemented but not stated as contract - (`sequencer-core/src/fee.rs`). Load-bearing for the C++ scheduler port; - interacts with the deferred fee-LUT track. (The `fixed_mul` comment that - claimed a nonexistent `debug_assert` now states what the truncation - relies on: the `MAX_EXPONENT` bound upstream — closed 2026-09-03.) -9. **`trg_enforce_nonce_contiguity` NULL hole** — a dangling parent makes - the comparison NULL and the trigger silent; mitigated by `foreign_keys=ON` - on every writer connection, but the trigger itself is not NULL-safe. -10. **Closed** (2026-09-16): batch sealing asserts that the next frame retains - the durable Tip clock; complete L1 reconciliation owns clock advancement. -11. **Partially closed** (2026-09-16): `safe_accepted_batches.inclusion_block` - drives accepted snapshot selection and export. `first_frame_safe_block` - remains audit-only and may be removed in a separate cleanup. -12. **`direct_q` is unbounded in the shared scheduler** — an adversarial - deposit flood is bounded in time (force-drain) but not bytes; a - per-input cap or byte budget closes a (very expensive) guest-OOM vector. -13. **`MAX_BATCH_METADATA_BYTES` (71) understates real SSZ per-op overhead** - (~83+ with offsets) — byte budgeting undercounts ~15% for max-payload - ops (`sequencer-core/src/user_op.rs`). -14. **Wallet snapshot decode accepts unsorted entries** while encode sorts — - enforce strictly-ascending addresses (subsumes the duplicate check) or - drop the canonical-decode pretense (`app-core/src/wallet_snapshot.rs`). -15. **Reader and submitter re-open `Storage` per tick** — a held connection - per worker drops per-tick overhead. Low priority. -16. **`should_retry_with_partition` substring-matches the Debug format** — - consciously accepted and regression-pinned against alloy's format; - revisit with structured JSON-RPC codes. -17. **Closed** (2026-09-03): `latest_batch_index`, - `ordered_l2_txs_for_batch`, and `promote_finalized` are `#[cfg(test)]` - (gated rather than deleted — the first two have callers across three test - modules). `safe_input_end_exclusive` has a live reader-path caller and - stays. -18. **`frames` lacks the immutability triggers `batches` got** — `fee` and - `safe_block` are documented immutable but convention-protected only. -19. **Closed** (2026-09-04): the key-file read returns - `BootstrapError::KeyFile { path, source }`, classified by kind - (`config::key_file_io_is_terminal`: missing, unreadable, not a file, or - not text → 30; environmental I/O → 1); the message names the path, never - the contents; both halves pinned. The sibling `create_dir_all` at each - command's start keeps `CommandError::Io` → 1: it creates rather than - reads, so a missing path is not an operator mistake there; a read-only or - wrong-type parent is, and is not yet separated. -20. **Closed** (2026-09-04): `ensure_open_tip_for_recovery` splits its - disjunction — a non-`Safe` danger raises `StaleDecision`, an already-open - Tip raises the payload-free `TipAlreadyOpen`, paired with a retry reason - so the operator sees it; the guard test and the polarity pin assert it. -21. **Closed** (2026-09-03): `Storage::ensure_open_tip` is - `#[cfg(test)] pub(crate)`; the two intra-doc links and the snapshot - lifecycle doc then named the reducer's guarded `EnsureOpenTip` phase. - That phase-driver description was superseded by ordered recovery on - 2026-09-07; the guarded storage operation remains. -22. **Closed** (2026-09-04): `DangerDetector`, `InputReader`, and the - fee-oracle worker (narrowed with them: same single use, same - construction-required lock) take `ShutdownSignal`; `launch` passes - `shutdown.signal()` for the three and the scope to the lane, server, and - submitter. Data-directory ownership is each worker's own `ProcessLock`, - so the watchdog's weak witness is unaffected; the doc comments now say - workers that externalize or contain take a scope. - Historical containment/watchdog rationale: superseded by the - [2026-09-07 immediate-abort model](../plans/2026-08-authority-boundary-adr.md#1-runtimescope-structured-process-ownership). -23. **Closed** (2026-09-04): the guarded `EnsureOpenTip` phase re-reads - `has_valid_open_batch` inside its own transaction and returns - `TipMissingAfterOpen` (classified `refuse`, exit 30) rather than commit - without a Tip; `drive_recovery`'s doc records the ≤5-phase bound and - `admission.tla`'s comment names the enforced postcondition. - The phase driver and its bound were superseded by ordered recovery on - 2026-09-07; the storage postcondition remains. -24. **Closed** (2026-09-04): the in-scope recorder is deleted, so a - contained run writes exactly one `terminal_faults` row — the command - bracket's, at settlement. Containment writes nothing durable. The - accepted loss, stated in the runbook: any death before settlement (an - abort at the terminal abort deadline, a controller panic, SIGKILL) leaves - only the process logs, and the single write has no second attempt. The - row is telemetry; restart policy is the exit code. - Historical settlement behavior: superseded on 2026-09-07. Terminal - runtime faults now abort immediately without a drain or terminal row; - terminal command errors returned through the bracket still get a - best-effort row. -25. **Closed** (2026-09-03): the token's doc, the ADR, and the settled - entry state its true scope (three compile-forced primitives; the rest - hand-placed); "non-clone" became "boot-local"; the witness comment names - every holder; the "journal", "acknowledge", and `DangerDetectorExit` - remnants are gone. - Historical token documentation: the token was removed by the - 2026-09-07 immediate-abort model. -26. **Closed** (2026-09-04): `acquire_finalized_lease` returns its own - `FinalizedLease { inclusion_block: u64, dump: LeasedDump }`, so the - `NOT NULL` column is no longer an `Option` and the impossible-`None` - containment branch in `finalized_state` is gone. Corrupt-row containment - is unchanged (the persistent-storage classifier and the decode panic, as - `corrupt_finalized_snapshot_trips_terminal_storage_fault` pins). - The lease type remains; the historical containment response was - superseded by immediate terminal abort on 2026-09-07. -27. **Closed** (2026-09-04): the two-verdict `RecoveryFailure::Provider` - is split into `ProviderUnreachable` (retry) and `SignerMisconfig` - (refuse), constructed by a pure `classify_signer_provider` pinned on all - three arms; `classify_input_reader` documents its phase-dependent - polarity and pins `Bootstrap`/`Join` refused at startup, non-terminal - live. The setup-versus-run asymmetry it exposed is finding 32. -28. **Setup admission is implemented twice with different semantics** — - `preflight_lifecycle_command` (used by `run`/`flush`) and - `admit_setup_lifecycle` (`commands/setup/mod.rs`) each check the same two - facts; an already-complete plain setup is a no-op in one and - `NotAdmissible` in the other, and `load_setup_identity` re-checks - completion with a third error type. Unverified. -29. **`http.rs` hosts a runtime component** — the ~175-line supervised - lease-release queue with two containment sites belongs in the egress - snapshot module that owns leases; its queue is unbounded and capped only by - concurrent snapshot requests. Unverified; weight, not safety. -30. **Harness block-time notions are unreconciled** — four constants with an - unasserted relationship; `advance_live_frame_until_covers` can drive L1 - ~20× ahead of the process clock; `aging_open_tip_runtime_danger_zone_exit_test` - stages an impossible chain and greps the rendered log (the CI red). One - `ChainClock` authority plus a post-mining drift check; assert exit 10 - instead of the log. The log grep is verified first-hand; the rest is - unverified. -31. **`log_gas_price_updated_at_ms` is production-write-only** — written on - every refresh, read only by tests, yet the threat model cites it as the - telemetry that justifies having no expiry gate. Surface it (health field - or the transient-refresh warn) or drop it and fix the sentence. - Unverified. -32. **A deterministic L1 misconfiguration exits 1 under `setup` but 30 - under `run`** — `setup` wraps every non-`Provider` input-reader failure - — from `InputReader::new` (`setup/mod.rs:107-127`) and from both - `sync_to_current_safe_head` calls (`setup/mod.rs:260-267,535-542`) — as - a worker exit, whose `InputReaderError::is_terminal_invariant` treats - `Bootstrap` as the live-worker case (non-terminal), so it projects to 1. - `run` meets the bad-RPC-URL half through `classify_input_reader` and - refuses at 30; the discovery-time half (wrong contract, pre-v3 InputBox) - has no `run` counterpart, because `run` builds its reader with - `from_parts` and never re-runs discovery. `setup` is an - operator-run one-shot, so the harm is a wrong hint, not a restart loop. - Candidate fixes: classify `setup`'s bootstrap failure through the same - phase table as `run`, or give `setup` its own terminal variant for L1 - misconfiguration. Surfaced by the 2026-09-04 refuters; not yet decided. -33. **Closed** (2026-09-07): ordinary shutdown no longer cancels the reader's - in-flight blocking append. The reader joins it before returning, so the - final clean-exit check sees committed divergence and append failures - reach the supervisor. The actual worker-loop regression fails when - cancellation around the append is restored. - -Open maintainer decisions: - -- **What the 500 ms acknowledgement contract is *for*** — it is 8× above the - worst measured value and shapes nothing today; either it encodes - catch-up-overlap headroom (then measure that) or restate the objective. -- **Catch-up ACK-latency measurement** owed to the benchmark harness: ACK - p99 *during* an epoch-sized catch-up reconciliation turn (the in-crate - seed test digests 5,000 directs over a 7,200-block jump in one turn). -- **Track 6 with Bart**: the hardlink-suitability dispute and the - changed-era bootstrap contract (see the tracks doc). -- **C engine follow-up boundary** (2026-09-11): agree output storage and - checkpoint layout with Bart before replacing the drain protocol with output - arrays or the path callback with a static suffix. ABI version negotiation, - checked-in bindings, and linker policy also need concrete integration - requirements. Keep these separate from the port and the agreed contract - reductions; the private DEX engine and scheduler remain unavailable for - conformance testing. - -## Owed tests - -Statuses swept 2026-08-22 and updated through 2026-09-04. - -- **Arm-ordering discriminating test**: both `ClosedBatchInDanger` and - `TipInDanger` genuinely in danger; assert Closed wins (today pinned only - incidentally by an equally-aged fixture). -- **Fail-loud halves**: no test references `CatchUpError::NoSnapshot` or - `InclusionLaneError::NoOpenTip`. -- **`EstimatedBatchInDanger` e2e** (recipe: mine ~800 blocks, faketime - +30 min without mining, respawn → refusal with zero invalidations). -- **Process-level divergence scenario** via `respawn_until_stable` (storage - and reducer coverage exists; the end-to-end freeze/refuse loop does not). -- **Per-variant exit-code e2e assertions for classes 20/40/1** (those - failure-path e2es still assert only `!success()`). Landed 2026-09-04: 10 - in the aging-tip scenario, 0 through `stop_expecting_clean_exit` at the - healthy stop of `recovery_after_stale_batches`, and 30 in-crate through - the real command bracket (`harness.rs`, `run` on a never-set-up data - directory); the five verdicts are pinned to their integers in - `commands/error.rs`, so renumbering `EXIT_TERMINAL` fails the suite. -- **Closed** (2026-09-07): the phase→progress mapping and scripted driver - were removed. Procedure tests now exercise real SQLite inspections and - mutations, replacing only external Sync/Flush operations. The - `classify_input_reader` polarity pins remain. -- **Full-tear cascade on a recovered (anchor = `N'`) tree** re-rooting at - `N'` (anchor unit mechanics are covered; this end-to-end shape is not). -- **Uniswap-mode fee oracle end-to-end**: every fixture and e2e pins fixed - mode, so no Uniswap-mode sequencer boots in tests. Setup validation, - RPC-free runtime source construction, transient quote retention, and - terminal misconfiguration are source-boundary-pinned in-crate as of - 2026-08-25; a real E2E still needs a mock pool — decide whether that extra - harness is worth its weight. -- **True same-block direct-input ordering end-to-end**: the renamed - `multi_deposit_reconciliation_test` covers multiple accumulated directs, but - default Anvil automining puts its portal deposits in distinct blocks. A real - same-block test needs queued portal sends, one explicit mine, equal receipt - block assertions, and WS order/block attribution. -- **Verify-then-write-or-strike** (status uncertain on 2026-08-22): the - encoded-wire-frame stamp at an advanced safe head; the wallet - insufficient-balance silent no-op and replay-determinism pins; the - young-never-submitted-batch cascade-policy pin; the `recover_aging_tip` - torn/no-Tip entry; the cascade-with-backward-clock pin. -- **Closed** (2026-09-16): I7's colliding-artifact test asserts that failed - snapshot registration rolls back the seal, successor Tip, and cached head. - Snapshot endpoint tests cover referenced artifact deletion as a terminal fault. -- **Harness levers to build with their tests**: pending-tx capture + - re-inject (`txpool_content`/raw-tx before `drop_all_pending_txs`, then - `eth_sendRawTransaction`) → unlocks the zombie e2e, the headline - adversarial scenario; snapshot lifecycle observability (DB readers for - the snapshot tables + dump-dir inspection) → the take/promote/GC/lease - e2e, plus finally *asserting* warm-resume-from-dump (every restart test - exercises it, none asserts it — a silent fall-back to genesis replay - would pass everything, just slower); a bare second Anvil - (`--chain-id `) + mid-run endpoint override → the wrong-chain - e2e; kill-at-log-marker → the flush-completion/cascade-commit crash - window; SQLITE_BUSY injection → the submitter/WS BUSY items. - **Recorded do-not-build**: a split-view/response-rewriting L7 proxy - (unit-level provider mocks instead; e2e validates only the passing - path) and fsync/power-loss WAL-rewind injection (out of scope — - state-construction variants cover the detectable halves). -- **External-engine conformance** (2026-09-11): a reusable runner needs - application-supplied genesis and meaningful accepted/rejected inputs; compare - canonical comparison files rather than byte-identical recovery dumps. - Native repeatability alone does not establish native/canonical-machine - equivalence. Publish fee-conversion data and vectors for the C++ integration - (finding 8), and add targeted devnet E2E coverage through the C host in CI. - -## Settled decisions - -Each entry: the decision, its reason, and where the reasoning now lives. - -- **Application-only current history** (2026-09-16): retain every raw L1 input - and original batch/frame/user-op record, but replace the invalidated flattened - application suffix. The recovered prefix is opaque. Mandatory offsets and - versioned claims replace the mixed replay log and sparse mapping. Acceptance - facts select immutable per-batch snapshots without promotion or restamping; - per-batch cadence and end-of-block watchdog comparison remain. The complete - model lives in [application history](../protocol/application-history.md), I5–I11, - I18/I20, and the snapshot lifecycle. - -- **No architectural restructure** (2026-06-10): one file per writer role, - `*_in(tx)` free functions composing into larger transactions, - storage-owns-SQLite / lane-owns-filesystem — the layout is sound and is - defended, not redesigned → AGENTS.md "Sequencer module layout", the - storage module docs (`storage/recovery.rs`), `docs/snapshots/lifecycle.md`, - and the do-not-simplify list in `docs/invariants.md`. -- **Write-before-broadcast watermark** (2026-06): the flush's completion - anchor is durable, not the local pool's memory → I14. -- **Content-identity check, gated on full acceptance** (2026-06): accepted - landings compare by content hash; content-equal copies are effect-equal, - so no batch identifier is needed; detection freezes the frontier - atomically with the detecting sync → I9, I15. -- **`synchronous=FULL`** (2026-06): externalization rides on commits, so - every commit fsyncs; noise-level cost on NVMe → `storage/open.rs` doc. -- **Cockroach recovery's flush is best-effort by construction** (2026-06): - the wiped DB destroys the watermark, so the flush resolves only what the - provider remembers, and plain `setup`'s detection gate shares the same - false negative. Accepted because the content-identity check turns the - residual zombie from silent divergence into a detected freeze (repair: - wipe and re-run). Recorded option if ever needed: an operator-supplied - flush floor from the old DB's watermark — fail-safe under corruption - (too high wastes no-ops; too low degrades to exactly best-effort) → - `cockroach.md` step 2. -- **Exit-code contract** (2026-06, panics-terminal amendment 2026-07): the - orchestrator must not parse logs; 10/20/30/40 by restart productivity → - `commands/error.rs` + the operator runbook. -- **Fail-loud check policy replaces "no defense-in-depth"** (2026-06): the - line is loud-vs-silent, not self-doubt; assertions must check real - invariants (the wall-clock CHECK cautionary tale) → the invariants check - policy. -- **Scoped pending clear** (2026-06): delete only pending rows at/above the - cascade pivot, in the cascade's transaction → I5. -- **Batch-tree anchor, not a sealed sentinel** (2026-06-25): the parentless - root carries the anchor nonce, exact-matched by the contiguity trigger → - I16. -- **`N` is trusted; no recovery-time verifier** (2026-06-26): a - sequencer-produced finalized dump cannot carry a wrong `N` by - construction; only wrong-low is caught at `run` → `cockroach.md` data - dictionary. -- **Anchor-aware frontier; recovery defers population** (2026-06-26): - below-anchor landings are trusted collapsed history → I15. -- **Recovery drain caps at `C`** (2026-06): `(C, H1]` deposits stay - undrained so `run` leads them exactly once → `cockroach.md` steps 3/6. -- **Don't resurrect TEST_PLAN.md** (2026-06): the scenario matrix rotted - once; owed tests live here as a dated, finite list. -- **The authority boundary** (2026-08, re-evaluated 2026-09-07): four - mechanisms — process ownership and terminal abort, fact-derived admission, - ordered startup recovery, SQLite-centered runtime with the two-regime lane → the - [ADR](../plans/2026-08-authority-boundary-adr.md). -- **Storage decode policy** (2026-07): fail-loud for contract-impossible - values; the named `saturating_query_bound` only where clamping preserves - the predicate → `storage/convert.rs` + the check policy. -- **The calibration rule** (2026-08-18): the complexity budget belongs to - concurrency, mutual exclusion, durability, and hostile-L1 robustness → - AGENTS.md design principles. -- **Terminal runtime abort** (2026-09-07, supersedes the 2026-08-18 - `Authorized` token): a diagnosed terminal fault logs and aborts, so there - is no terminal runtime to drain or gate. Ordinary shutdown signals, - concurrent joins, and process-lock ownership remain. Dedicated-process - hosting and prompt diagnostic logging are the supported assumptions; - orderly terminal requests and settlement are explicitly given up → ADR - mechanism 1. -- **Snapshot leases remain** (2026-09-07): application dumps may contain a - directory of state files. GC racing a download is an ordinary supported - operation; relying on open-file unlink behavior is not the general dump - lifetime contract → snapshot lifecycle, leases. -- **C bridge contract reductions** (2026-09-11): the linked engine reports its - stable payload bound at runtime, including a valid zero bound; C progress is - one count/clock record. This removes duplicated build configuration and - presents the protocol pair together without introducing another state owner. - Checkpoint artifacts are self-contained beneath their supplied prefix and - require only ordinary filesystem deletion, so the sequencer recursively - removes its enclosing dump directory without an application deletion hook. - Reader leases, SQLite-first deletion, and restored-engine independence remain - required → [C binding guide](../protocol/c-application-binding.md) and - [Application checkpoint contract](../protocol/application-contract.md#6-checkpoint-lifecycle). -- **C host failure boundaries** (2026-09-11): snapshot-path callback panics - enter the host's terminal-abort boundary; a failed HTTP task alone must not - leave sequencing active after an application invariant violation. Missing or - malformed genesis dumps and absent required options are terminal bootstrap - failures; operational I/O remains retryable. Genesis stays lazy: completed setup may - succeed after its original source has been deleted. A pre-command mandatory - state-file check would violate that lifecycle contract → snapshot handlers, - command error classification, and genesis harness tests. -- **Execution-offset continuity has one enforcement point** (2026-09-07): - the SQLite trigger rejects a noncanonical offset inside the application-input - transaction. The duplicate Rust loop was removed; rollback, invalidation, - and offset-reuse tests remain → I20. -- **Module homing** (2026-08-19): command brackets in `commands/` (with - config + the `CommandError` taxonomy), the capability substrate alone in - `runtime/`, `L1Config` in `l1/`; a full merge was refused because the - substrate is consumed crate-wide. `http.rs` stays whole until the - ingress/egress listener split forces it apart. -- **Lifecycle: facts govern; the black box records** (L1 2026-08-18 → - L2 2026-08-19 → L3 2026-08-22): admission is three facts; no state - machine, no acknowledgement (it carried no machine-consumed decision); - telemetry writes are verdict-neutral; terminal faults refuse at - re-detection with the residual recorded in the threat model → the ADR, - the invariants check policy, and the two dated ledgers. -- **Misconfig-poison taxonomy** (opened 2026-08-18, closed by L3): there is - no poison; misconfig is terminal by exit code only, and a fixed config - boots cleanly. -- **Submitter-key redaction** (2026-08-24): the key enters the process as - `SubmitterKey` at the clap edge — `Debug` redacts, no `Display` exists, - and the raw hex is reachable only through `expose_secret`, so every - consumer of the secret is greppable. The key's public identity is the - pinned `batch_submitter_address` beside it. Closes the former - Debug-derive open finding → `l1/mod.rs`. Deferred separately: the startup - log prints the full RPC URL, which the help-leak test treats as - token-bearing. -- **One home per mechanism** (2026-09-05, updated 2026-09-07): every mechanism has one canonical - statement, and every other site is a pointer or an explicitly scoped - partial. The homes: the authority-boundary ADR for `RuntimeScope` and - terminal abort (mechanism 1), fact-derived admission and the black - box (2), and the SQLite-centered runtime and the two-regime lane (4); - `docs/recovery/README.md` for the procedure; `docs/invariants.md` for the - cross-module invariants, the check policy, the writer roles, and the - divergence freeze (I15) with the check's completeness scope (I9); - `docs/protocol/scheduler-semantics.md` for the frame clock; - `docs/protocol/application-contract.md` §5 for the digestibility - assumption; `README.md` for the API contract and the storage model; the - schema for the write-once batch lifecycle; `commands/error.rs` for the - exit-code contract, with the operator runbook and README carrying the - operator- and user-facing lists; `runtime/shutdown.rs` and the runbook for - terminal stop policy; this register for refuted proposals and the review - history. Code-side exceptions, where the argument is falsifiable - only at the code: the stack-local flush observation, the admission - linearization (`admit_runtime`), and the one-transaction inspection - (`RecoveryInspection`) → - AGENTS.md "Documentation Practice". - -## Refuted — do not re-propose without new evidence - -From the 2026-06 reviews: - -- **`scheduler_accepts` omitting the two structural rejections is a bug** — - deliberate self-trust; the simulator runs only over our own well-formed - batches; the worst case is covered by the content-identity check - (documented in `scheduler-semantics.md`, duality-test-pinned). -- **Sealed `N'-1` sentinel batch** for recovery rooting — a valid closed - sentinel is a legal cascade pivot; nothing stops a runtime cascade from - invalidating it, after which recovery ABORTs. Its safety rested on an - unenforced assumption. -- **Recovery-time `N` cross-check** — circular: every cheap recomputation - seeds from the `N` it would check; the only independent check is a - from-genesis L1 replay, deliberately not built. -- **`FoldInputSource` abstraction** — a wrapper over a single call site; - revisit only if a second fold-input source appears. -- Wire-fee-exponent panics, stalled-WS DoS, operator `WalletConfig` dead on - warm start, snapshot bytes history-dependent — each verified as - deliberate/out-of-scope (see the threat model's scoping). - -From the ADR's rejected list (opened by the 2026-08-01/02 re-evaluation); -the 2026-08-18 premise challenge found these alternatives left no residue in -code: - -- **`RunEpoch`** (a globally threaded internal fencing epoch) — the OS lock - plus structured task lifetime plus fresh per-scope channels already make - an old sender unable to reach a new receiver, and there is no in-process - hot restart to fence against; persisted rows cannot distinguish a live - owner from a stale one, a kernel-held lock can. Revisit only if in-process - restart or multiple admitted runtimes under one lock are introduced. - Evidence: `runtime/process_lock.rs` (module doc); no epoch type exists in - `sequencer/src`. -- **`EffectGate` / `LiveKernel`** (a universal effect mutex or actor, with a - reader mailbox) — would duplicate the role-local linearization points the - system already needs and force the reader and the latency-critical lane - through a new in-memory authority protocol, adding a second state machine - without making the narrow content-identity check a complete divergence - oracle; SQLite stays the durable coordination plane. The `Authorized` - token is not this — see - [ADR mechanism 1](../plans/2026-08-authority-boundary-adr.md#1-runtimescope-structured-process-ownership). - Evidence: - `runtime/shutdown.rs` (`Authorized`); [I9](../invariants.md)'s - completeness boundary. -- **A generic command controller** over setup/rebuild/run/maintenance — - their facts are unrelated; combining them enlarges the cross-product state - machine without closing an enforcement hole, and a flush has no admission - state to restore or erase. Evidence: the per-command controllers are - separate — `recovery/mod.rs` (`drive_recovery`), `commands/setup/mod.rs` - (`admit_setup_lifecycle`), `commands/flush.rs` (the flush body); the one - shared piece is a *fact* gate, `commands/mod.rs`'s - `preflight_lifecycle_command` (used by `run` and `flush`; `setup` reads - its own two facts inline), which checks admission facts and reduces - nothing. -- **A per-chunk divergence query, provider call, or reader mailbox** on the - hot path (formerly proposed per user-op) — the content-identity check is - complete only for at/above-anchor accepted-batch content identity - ([I9](../invariants.md)), so a query paid on every user-op chunk would buy - no complete safety boundary. Cost is not the argument: the `POST /tx` - round trip that carries a chunk is about 13 ms at concurrency 1 - (concurrency-1 HTTP ACK p50 13.231 ms against submit-to-matching-WS-event - p50 25.313 ms in the same harness session — the ADR's - [performance posture](../plans/2026-08-authority-boundary-adr.md#performance-posture) - carries the surviving figures; the maintainer's earlier informal "roughly - 14 ms" localhost round-trip observation carried no metric qualifier), so - the query would be cheap and still incomplete; a provider call on the same - path would put L1 liveness inside the acknowledgement path. Evidence: - `ingress/inclusion_lane/mod.rs` (the bounded chunk and the time-gated - frontier read); I15's runtime reaction. -- **A durable recovery-phase ledger** — the flush and post-flush-sync - witnesses are boot-local by design; persisting them would re-create a - state machine whose only effect is skipping an idempotent flush, and would - let a restarted attempt trust a half-remembered phase. Evidence: - `recovery/mod.rs` (`RecoveryProgress` is memory-only and `drive_recovery` - its only writer; the pin - `reconstructed_controller_cannot_reuse_a_post_flush_sync_witness`); - `admission.tla` (no durable per-attempt record gates anything). -- **A marker-file containment protocol** (2026-08-01; hardened by that - review, then deleted wholesale — git history) — a filesystem side-channel - for containment state, superseded by the in-process containment bit and - the settlement-written black-box row for containment state (ADR - mechanism 1) and by fact re-detection at boot for the durable verdict - (ADR mechanism 2); the kernel process lock guarded it and outlived it. Do - not reintroduce one: SQLite is the durable coordination plane and the - process lock is the exclusivity primitive. - -From the 2026-08-18 adversarial pass: - -- **Merging `Workers::finish`'s two drain modes by re-awaiting the primary** - — the winning select arm consumed the handle's completion (the select - borrows `&mut self.server` etc., so the handle is still in the cleanup - set); re-polling it panics ("JoinHandle polled after completion"), - unwinding through `ShutdownOnDrop` into containment — a benign stop - becomes a poisoned data directory. Scope-narrowed 2026-08-23: the 2026-08 - source read "as sketched" / "the naive merge"; distillation dropped the - qualifier. A merge that removes the primary from the cleanup set before - draining — keeping the `expect` that it was present — is settled, not - refuted (landed as `finish`'s one-loop/two-phase shape). -- **Removing the post-commit accessor-coherence assertion** — it is the - only guard in the two contexts with no database backstop (canonical - RISC-V fold, `fold_replay`). -- **"The three-variant frame-drain writer family is bloat"** — backwards: - the raw physical writers are `#[cfg(test)]`-demoted; production has one - way to write a frame. -- **`FuturesUnordered` for cleanup polling** — not worth promoting a - dev-only dependency tree to delete one small hand-written future. - -From the 2026-08-23 run-glue simplification pass: - -- **`Workers` as a homogeneous component list** (a `Vec<(WorkerId, - ComponentShutdown)>` built at launch, one select arm racing the list) — - it makes the "no `.await` between `Poll::Ready` and `swap_remove`" - property `select!`-load-bearing and untested (violation loses a worker - exit *and* panics on re-poll); converts the asserted primary-in-set - precondition into an unchecked cross-function assumption whose failure is - the benign-stop-to-exit-30 outcome; deletes the composed detector - select-mapping tests; and makes a zero-component race representable. The - real hole it targeted (select arms were the one per-worker site not - compile-forced) is closed by the exhaustive `let Self { .. }` destructure - in `select_first_exit` instead. -- **`UniswapConfig::pinned(&identity)`** — the exhaustive - `FeeOracleIdentity` match in the run bracket IS the launch decision for - the optional oracle worker; moving it behind an `Option` - constructor makes a future identity variant compile while silently - launching no worker, and the chain-id-pairing guarantee it claimed is - already structural now that the identity travels whole inside `L1Config`. - -From the L3 review (2026-08-22): - -- **A durable boot gate on terminal verdicts** (a gate on a *verdict*, i.e. - a non-fact) — it needs an acknowledgement to exit, and the acknowledgement - carries no information the fact-derived reducer doesn't re-derive. A - verdict-neutral startup *read* of the black box is not covered by this - entry, and is now exercised: `run` logs the latest row once at startup and - branches on nothing (`warn_on_previous_terminal_fault`, 2026-09-04). -- **A boot-time full-integrity sweep** — expensive machinery that still - cannot catch semantic violations outside its read set; the residual - window is recorded and bounded instead. -- **A boot assert on `finalized_snapshot.inclusion_block`** — the column is - `NOT NULL` at the engine; the claimed gap does not exist. (The runtime - `Option` branch on the same column was finding 26, closed 2026-09-04.) - -From the 2026-08-25 fee-oracle lifecycle pass (143a290; a single-pass -decision, not an adversarial review): - -- **Fee-price age as a runtime lifecycle gate** — setup owns the required - live quote; run starts from the persisted price and refreshes it best-effort. - Shared-endpoint staleness is already detected from safe-head progress, while - a pool-only outage is an explicitly accepted economic residual. Reintroduce - an expiry gate only with an independently derived economic bound and action, - not by borrowing the L1 liveness threshold. The telemetry this entry leans - on is production-write-only (finding 31). - -From the 2026-09-03 branch stock-take (three refuters per proposal; full -reasoning in the [ledger](2026-09-03-branch-stocktake.md)): - -- **`Workers` on a `JoinSet` fed by the shutdown waiters** — `from_select` - and `from_shutdown` deliberately read a worker's clean `Ok(())` two ways - (live: stopped unexpectedly; drain: graceful); one set collapses them and a - silently dead lane becomes exit 0. Evidence: `commands/error.rs:606-627`, - `commands/run/workers.rs:339-390`. The `swap_remove` hazard is real; a - cleanup-only set built inside `finish`, with the live race untouched, was - not examined. -- **Collapsing `PreparedRuntime` into an `async fn boot`** — - `fn launch(self, RuntimeAdmission) -> Workers` is non-async and non-`Result`, - so `?` and `.await` after admission are compile errors today; `boot` makes - them legal and the witness degenerates. Evidence: `workers.rs:288`, - `recovery/mod.rs:477-483`; 02a2b34 stopped here deliberately. -- **Making the `Authorized` token "uniform" by demoting the `/tx` 200 gate - to a bool** — the 200 body is the acknowledgement leaving the process; - `LeasedDumpBody` does not exist and `finalized_inclusion_block` has no - streaming primitive. Evidence: `ingress/api.rs:84-92`, - `egress/api/snapshot.rs:102-121,209-214`. The doc tightening survives (finding - 25). -- **Deleting the `#[from]` impls so a refusal reason cannot be typed into a - retry** — the enums are public with public variants; the longer spelling - still compiles and is the dominant idiom at all 15 sites. Evidence: - `recovery/mod.rs:45-49,122-128`. -- **One `is_terminal()` on `VerifiedSignerProviderError` read by both - tables** — the `From` impl performs no classification; the verdict is taken - later over `BootstrapError`, whose variants have four other producers. - Evidence: `commands/error.rs:688-700,220-273`; `l1/reader.rs:96-104` - already documents the phase pair. -- **Flattening `RecoveryError` into one enum with `is_retryable()`** — - refuted 2026-09-03 because `RecoveryFailure::Provider(String)` carried two - verdicts, so no total function over the value existed. That premise was - retired 2026-09-04 when finding 27 split the variant; every - `RecoveryFailure` now determines its verdict from its value. The entry - keeps its place on its remaining ground: the `Retry`/`Refuse` wrapper is - the reducer's verdict at birth, not a predicate a consumer recomputes, and - dropping its `Box` grows the `CommandError` footprint managed against - clippy's `result_large_err` (`commands/error.rs:537`). Re-propose only - against those. Evidence: `recovery/mod.rs:122-128,545-583`. -- **Moving the phase→progress mapping into `drive_recovery`** — `(Flush, - Done)` has no target without the observed block; this is - `PhaseCompletion`/`transition_after_phase`, deleted by ed41f9b. Deleting - `RecoveryDriver::admitted` survives. -- **Merging `FeeOracleMisconfig`/`FeeOracleFatal` and - `ChainIdRpc`/`DetectionNonceRead`** — bare-string producers at - `commands/setup/mod.rs:144,171` would misattribute an operator mistake in - the black box; the second merge demotes a compile-forced classification to a - string discriminant (refuted twice on 2026-08-23). -- **A `TerminalityOf` trait for `WorkerStop`** — admits the same wrong - classification as `|_| false`, installs a crate-wide answer for `io::Error` - that `dump_info.rs:49-58` contradicts, trips `private_bounds` under - `-D warnings`, and splits an inherent convention across eight types. -- **Wiring `check-admission` into CI as one line, and pruning three - "duplicate" settle actions** — no Nix or `.envrc` is tracked and `just` is - absent from the `rust` job; TLC checks the spec against itself; the actions - are state-neutral but `InspectRetry` encodes a recorded commitment. A - properly pinned standalone `formal` job survives as a proposal. -- **A sequencer-owned `AppWithProgress` wrapper replacing the progress - capabilities** — the pair lives inside the canonical SSZ bytes the watchdog - byte-compares and the canonical machine advances it inside its own - transition; cockroach recovery reads the clock from a dump into a wiped - database. Evidence: `examples/app-core/src/wallet_snapshot.rs:41-42`, - `commands/setup/mod.rs:433-451`. Record the composition in the application - contract. **2026-09-09 refinement:** keep native progress in the engine and - expose it by value; remove the mutable accessor and capabilities. This - preserves canonical checkpoint bytes without requiring a Rust-side mirror. - See the [Application/lane review](2026-09-09-application-lane-dex-review.md). - -Also standing, from the same reviews: the **do-not-simplify list** now -lives beside the invariants it protects -([`docs/invariants.md`](../invariants.md), "Do-not-simplify"), and these -deliberate declines keep their reasons — egress single-poller fan-out (no -need at current subscriber counts), `LeaseGuard` shared release channel, -`finalized_state` ETag on `l2_tx_index` (no reachable collision), -stale-skip on-chain report (scheduler protocol change; queue behind the -scheduler library), `Storage::read` commit-vs-rollback (no behavioral -difference). - -## Historical codename map - -Older commit messages and the pre-distillation ledgers (in git history) use -these codes; their concepts now live here: - -| Code | Concept | Current home | -|---|---|---| -| R1a | write-before-broadcast watermark | I14 | -| R1b | cockroach recovery's best-effort flush | `cockroach.md` step 2 + settled above | -| R2 | content-identity check | I9, I15 | -| R3 | `synchronous=FULL` decision | `storage/open.rs` | -| R4 | exit-code contract | `commands/error.rs`, runbook | -| R5 | fail-loud check policy | invariants check policy | -| F1–F10 | 2026-06 correctness findings | settled above; F7 = the closed "WS invalidation/rollback contract" finding | -| I1–I20 | invariants (stable, still in use) | `docs/invariants.md` | -| D1–D11, H1–H14, S-A, P1–P8 | 2026-08-18 defects / harvest / structural fix / premise items | settled above + ADR | -| WP1–WP11 | 2026-06 work packages (all landed) | settled above | -| L1, L2, L3 | the lifecycle decisions (not Layer 1/2) | ADR mechanism 2 + the two August ledgers | -| S1–S7, A1–A12, B1–B5 | 2026-06 simplification queue / owed tests | open remnants above | - -## Review history - -One row per dated review ledger, oldest first. Single-pass decisions that -produced refuted entries without a ledger (the 2026-08-23 run-glue pass, the -2026-08-25 fee-oracle pass) carry their own dated blocks under Refuted. The -rows without a file were stubs whose every fact already lived here or in a -living doc; their originals are in git history, in the parent of the -distillation commit ("docs: distill the corpus — living docs timeless, -history in the register", f1b4b07): `git show f1b4b07^:docs/review/` -for `2026-06-10-correctness-review.md`, `2026-06-10-simplification.md`, -`2026-06-10-test-coverage.md`, `2026-06-25-cockroach-recovery-rooting.md`, -`2026-06-26-branch-deep-review.md`, and -`2026-08-01-containment-adr-review.md`. - -| Date | Scope | Verdict | Where its content lives | -|---|---|---|---| -| 2026-06-10 | Whole-project correctness review: twelve parallel module reviews plus line-by-line passes, every medium/high concern adversarially verified | The sequencer/scheduler duality was sound; the confirmed problems clustered at the boundary with the infrastructure underneath (fsync semantics, the local node's mempool memory, RPC fleet coherence, the subscriber protocol). Ten findings (F1–F10) and five design resolutions (R1–R5); every F-finding fixed except F7, the WS invalidation contract | R1–R5 → Settled decisions and the codename map; F7 → the open WS invalidation/rollback finding (Track 3); the robustness and hygiene backlog the review left → findings 1, 2, 3, and 6 | -| 2026-06-10 | Simplification and refactoring review (companion) | No architectural restructure: the layout is sound and is defended, not redesigned; the weight was unpinned cross-file invariants, test-only surface presenting as production API, and duplicated semantics | Created `docs/invariants.md`; the settled entry above; open remnants and declines above | -| 2026-06-10 | Test-coverage review (companion): what the suite pins, what it misses, which harness levers exist | Recovery is the best-tested subsystem (the full dispatch matrix at unit and e2e level, libfaketime clock jumps, respawn loops, TCP-proxy outage injection, Anvil mempool control); the one structural hole, the duality having no direct mechanism, is closed by the watchdog non-genesis byte-compare e2e and the I1 agreement table | Owed tests and harness levers still to build above; do not resurrect a TEST_PLAN scenario matrix | -| 2026-06-25 (follow-ups closed 2026-06-26) | Design session with an adversarial panel: how `setup --recovery` roots the rebuilt batch tree at the resume nonce | The `batch_tree_anchor` singleton: the parentless root carries the anchor nonce, exact-matched by the contiguity trigger and frozen once setup completes; no sentinel batch row | I16 and `docs/recovery/cockroach.md`; the sealed sentinel and the circular recovery-time cross-check → Refuted; the follow-up e2e's self-divergence against the empty rebuilt tree → the anchor-aware frontier on I15 | -| 2026-06-26 | Deep branch review: the setup/run split, the scheduler-library extraction, the fold engine, `setup --recovery`, plus a multi-agent adversarial sweep | Eight findings confirmed and fixed; three refuted | The two protocol contracts (`docs/protocol/scheduler-semantics.md`, `docs/protocol/application-contract.md`) were written during this review; the recovery spec is `docs/recovery/cockroach.md`; per-finding dispositions and the declined `FoldInputSource` above | -| 2026-08-01/02 | Containment ADR review: two rounds on the terminal-containment cutover, then re-evaluation with the maintainer | The architectural turn accepted; the unimplemented `LiveKernel`/reader-mailbox design rejected on the completeness/cost boundary: the content-identity check is a narrow backstop, not a divergence oracle, and SQLite remains the durable coordination plane | Every mechanism it shaped → the [authority-boundary ADR](../plans/2026-08-authority-boundary-adr.md); the divergence race bound → I15; `RunEpoch`/`EffectGate`/`LiveKernel` and the marker-file protocol → Refuted | -| 2026-08-18 | Over-engineering review: the full branch, seven parallel subsystem reviews plus an independent premise challenge of the ADR | Not over-engineered, unevenly engineered; 141 mechanisms inventoried (98 keep, 25 simplify, 6 cut, 12 question) | [`2026-08-18-over-engineering-review.md`](2026-08-18-over-engineering-review.md), kept for the inventory, the ~700-line harvest, and the eleven defects | -| 2026-08-22 | Lifecycle simplification, decision L3 | The attempt journal bought only what tracing already provided; it narrowed to the `terminal_faults` black box, and telemetry writes became verdict-neutral | [`2026-08-22-lifecycle-simplification.md`](2026-08-22-lifecycle-simplification.md), kept for the journal weight audit, the `admission.tla` ghost-variable result, and the verdict-integrity defects | -| 2026-09-03 | Branch stock-take of the authority-boundary PR: first-hand reads, then a read-only fleet of seven subsystem lenses and five premise challengers, then three refuters over the eighteen highest-ranked proposals | Proportionate overall, with three residue pockets; every proposal recorded, the jury-refuted ones listed above | [`2026-09-03-branch-stocktake.md`](2026-09-03-branch-stocktake.md), the ledger of the current branch, with its "Landed" section | -| 2026-09-07 | PR #28 premise review and maintainer-approved simplification | Ordered recovery replaces the phase driver; diagnosed terminal runtime faults abort immediately; ordinary shutdown and snapshot leases remain; reader drain race and duplicate offset check fixed | Current ADR and recovery design; finding 33 and settled decisions above. Validation: 692 host tests, seven targeted restart/outage E2Es, workspace check, strict Clippy, formatting, and admission TLC passed. The broader stale-batch recovery E2E reached its watchdog comparison but was blocked by the host Lua emulator 0.21 loading the pinned 0.20 image (archive version mismatch); no protocol pin was changed. | -| 2026-09-09 | Application, inclusion lane, and public DEX integration branch | Native progress ownership, typed validation failures, mutable independent checkpoints, optional canonical inspection, and lane bookkeeping simplified; ingress CORS and Lua 5.4 parity restored. Reference C bridge port kept separate. | [Application/lane review](2026-09-09-application-lane-dex-review.md); current Application and snapshot contracts. Workspace check, strict Clippy, 697 host tests, and 62 watchdog tests passed; private DEX conformance remains unverified. | -| 2026-09-11 | Reference C bridge port and review boundary | Keep the current Application contract, runtime payload bound, paired progress, filesystem-owned checkpoint disposal, and host failure fixes together. Engine-dependent API refinements and external-engine conformance remain follow-ups. | Settled decisions and owed tests above; [C binding guide](../protocol/c-application-binding.md), Application contract, and snapshot lifecycle. | -| 2026-09-16 | Application-history and Track 3 implementation, with independent storage/recovery/snapshot review | Replaced mixed replay and sparse attribution with current application inputs; complete atomic baselines; acceptance-derived immutable snapshots; HTTP restore/recovery archives and mandatory WS claims. Review narrowed equal-block recovery to empty genesis to avoid losing same-block pending directs. | [History design](../protocol/application-history.md), current invariants/API/snapshot/recovery docs. Validation: 693 workspace tests, strict Clippy, formatting, 62 watchdog tests, admission TLC (155 distinct states), and the Anvil recovery/old-claim refusal gate passed. Full canonical watchdog comparison remains blocked by the local emulator 0.21 vs repository 0.20 environment; no pin changed. | -| 2026-09-16 | Track 3 integration validation | Nonempty HTTP cold replica, concurrent backlog/live consumption, stale recovery/rebootstrap, and four real canonical-machine gates pass under emulator 0.20. Tooling fixes preserve Lua paths and make benchmark fee defaults admissible. | [Validation and latency evidence](2026-09-16-track3-validation.md); native bridge/DEX and representative deployment latency remain separate gates. | +# Unresolved review work + +Current follow-ups, maintained under the [review lifecycle](README.md). +Contracts and settled design reasons belong to their owners; completed review +history is in Git. Entries below were checked against code and test sources on +2026-09-17 at `85f033b0768de527312ff876a29bca67ee2f9316`. This was a source audit, +not a new run of every cited test. Recheck an entry before acting on it. + +## Confirmed discrepancies + +### Batch-size accounting omits SSZ overhead + +The lane estimates each operation as `71 + max_method_payload_bytes()`. +The SSZ layout takes `83 + actual_payload_bytes` per operation, including its +list offset, plus 12 bytes per batch and 18 per frame. Thus the estimate +understates a maximum-size operation; smaller actual payloads can mask it. +This is a batch-target accounting discrepancy, not a demonstrated protocol-size +overflow. The discrepancy is not a universal percentage. + +Evidence: [`SignedUserOp`](../../sequencer-core/src/user_op.rs), +[`Batch` / `Frame` / `WireUserOp`](../../sequencer-core/src/batch.rs), and +`user_op_count_to_bytes` in the [lane](../../sequencer/src/ingress/inclusion_lane/mod.rs). +Next: compare the intended bound with serialized batches across payload/frame +counts, then correct the estimate and check the separately configured +`batch_policy.log_user_op_bytes` used for fee accounting. + +### Setup misclassifies some deterministic L1 configuration failures + +Setup wraps reader bootstrap failures as live-worker failures, yielding exit 1; +normal-run startup classifies deterministic reader bootstrap failures as +terminal, exit 30. Malformed RPC URLs exercise this distinction. Discovery +failures such as a wrong InputBox also occur in setup, but run does not repeat +that discovery. The impact is a misleading operator hint in a one-shot command. + +Evidence: [`setup`](../../sequencer/src/commands/setup/mod.rs), +[`InputReaderError::is_terminal_invariant`](../../sequencer/src/l1/reader.rs), +and `classify_input_reader` in [startup recovery](../../sequencer/src/recovery/mod.rs). +Next: classify failures by the setup phase and pin the external exit code; +preserve genuinely transient provider failures. + +### Startup logs the full RPC URL + +The `sequencer startup` event includes `eth_rpc_url` verbatim. Operator URLs can +carry credentials in userinfo, paths, or query parameters; private-key +redaction does not cover this field. + +Evidence: [`commands/run/mod.rs`](../../sequencer/src/commands/run/mod.rs). +Next: omit the field or define a safe endpoint representation, and check +diagnostic/help paths with a synthetic credential-bearing URL. No credential +exposure in an actual deployment was established by this review. + +## Bounded investigations and cleanup + +- **Transient SQLite contention stops the submitter.** Read handles use a + 50 ms busy timeout; a storage/open failure escapes the submitter loop. + BUSY/LOCKED are nonterminal but project to unclassified exit 1, causing + respawn/recovery under a restarting supervisor. Frequency and benefit of + local retry are unmeasured. Check contention before choosing a bounded + retry or timeout change; preserve other errors. Evidence: + [`storage/open.rs`](../../sequencer/src/storage/open.rs), + [`submitter/worker.rs`](../../sequencer/src/l1/submitter/worker.rs), + [`commands/error.rs`](../../sequencer/src/commands/error.rs). +- **Canonical direct-input queue capacity.** The shared + [`Scheduler`](../../sequencer-core/src/scheduler/mod.rs) retains queued + payloads without a byte budget. Force-drain bounds age in the observed L1 + timeline, not bytes. Determine the supported L1-window volume and guest + memory cost before claiming an OOM vulnerability or proposing a limit. + Dropping or capping canonical inputs would change protocol semantics. +- **Overlapping admission policies.** The generic + [`lifecycle` preflight](../../sequencer/src/storage/lifecycle.rs) contains + setup/rebuild branches, but production calls it only for run/flush. + [`setup`](../../sequencer/src/commands/setup/mod.rs) has its own admission + path with an intentionally tested already-complete no-op. Consolidate or + narrow the unused branches when next changing admission; there is no + demonstrated conflicting live route. +- **Fee-observation visibility.** + [`log_gas_price_updated_at_ms`](../../sequencer/src/storage/fee_oracle.rs) + is persisted, with no production reader. Decide whether operator SQL + inspection suffices or a real consumer needs exposure before adding an + endpoint or removing the stamp. The accepted + [oracle outage policy](../threat-model/README.md#actors-and-trust) is an + economic tradeoff, independent of whether a health field exposes age. +- **Recovery across external effects.** Model/storage tests do not replace + process-level zombie re-injection or a restart between flush completion and + cascade commit. Before adding harness machinery, identify the missing + observation: safe nonce consumption must exclude a later original landing, + and a restarted recovery must rederive facts without reusing the previous + attempt's flush witness. Use the [recovery model](../recovery/README.md) to + bound a scenario and assess whether existing component tests suffice. + +## Verification gaps + +These are specific behaviors whose coverage remains incomplete, not a mandate +to build a general fault-injection framework. Add a discriminating assertion +at the smallest useful boundary when working on that behavior. + +| Boundary | Existing evidence and remaining check | +|---|---| +| Elapsed-time danger | Storage/procedure coverage exists. Add a process scenario isolating `EstimatedBatchInDanger` from stale-view refusal: retry at exit 20 without a speculative cascade. Start in [recovery](../../sequencer/src/recovery/mod.rs) and the [E2E scenarios](../../tests/e2e/src/test_cases.rs). | +| Canonical divergence | Storage freeze and startup refusal are covered separately. Compose accepted divergent input, runtime stop, and refusal after respawn in a process test; frontier must remain frozen. See [I15](../invariants.md#i15-divergence-marker-present--acceptance-frontier-frozen). | +| Rebuilt anchor | Anchor unit mechanics and rebuild round-trip are covered. Exercise a later full-tear cascade after a nonzero-anchor rebuild and verify submission resumes at that anchor. See [I16](../invariants.md#i16-the-batch-tree-has-exactly-one-valid-parentless-root-carrying-the-deployments-anchor-nonce). | +| Same-block directs | `multi_deposit_reconciliation_test` accumulates directs in separate blocks. Queue portal sends, mine once, assert equal receipt blocks, and verify canonical/WS order and attribution in the [E2E scenarios](../../tests/e2e/src/test_cases.rs). | +| Wallet business failure | Mixed replay is covered. Pin insufficient transfer/withdrawal amounts after successful fee validation: fee/nonce/progress advance, the transfer/withdrawal has no further effect, and replay agrees. See the [wallet implementation](../../examples/app-core/src/application/wallet.rs) and [Application contract](../protocol/application-contract.md#2-replay-safety--rejection-inclusion-and-failure). | +| Boot failure exits | Exit 20 now has a process assertion; exits 40/1 still need exact assertions in applicable setup/failure scenarios, so the supervisor receives the intended recovery/retry hint. Start in the [E2E scenarios](../../tests/e2e/src/test_cases.rs) and [exit contract](../../sequencer/src/commands/error.rs). | +| Uniswap-mode boot | Source-boundary tests pin setup validation, lazy runtime refresh, and transient quote retention. Existing sequencer E2Es use fixed mode. Decide whether a mock-pool boot scenario warrants its harness cost when changing oracle integration. | + +Warm restore already has state/cursor agreement tests and snapshot downloads +have lease/GC coverage. If restart cost becomes a requirement, add an assertion +that distinguishes restoring a dump from a correct but expensive genesis +replay; no new snapshot lifecycle is implied. + +## Integration work owned elsewhere + +- [Track 3 integration gates](../plans/2026-07-track3-feed-replay-design.md): + native snapshot-to-live replication and representative deployment latency, + including checkpoint creation and complete L1-reconciliation turns. +- [Track 6](../plans/2026-07-coordination-tracks.md#track-6--dump--application-api-redesign): + external engine/scheduler agreement, C-host end-to-end coverage, independent + fee-conversion vectors, and consumer-driven ABI/checkpoint decisions. + +The [2026-09-16 validation record](2026-09-16-track3-validation.md) supports +those integration decisions within its stated scope. It is not proof of +private-engine conformance or deployment capacity. diff --git a/sequencer-core/src/fee.rs b/sequencer-core/src/fee.rs index 81234fd..902f1ec 100644 --- a/sequencer-core/src/fee.rs +++ b/sequencer-core/src/fee.rs @@ -6,24 +6,14 @@ //! All fees in the protocol (frame `fee_price`, user-op `max_fee`, DB `recommended_fee`) //! are represented as **log-space exponents** with base 129/128. //! -//! An exponent `n` represents a linear value of `(129/128)^n` smallest-token-units. -//! Exponent 0 = 1 unit (minimum, effectively free). There is no special sentinel. +//! An exponent `n` nominally represents `(129/128)^n` smallest-token-units. +//! [`fee_to_linear`] owns the exact integer conversion contract, including +//! intermediate rounding and operation order. Exponent 0 is 1 unit; there is +//! no special sentinel. The encoding uses two bytes and no floating point. //! -//! This encoding: -//! - Fits any token denomination in a u16 (range up to ~10⁷⁷) -//! - Eliminates integer overflow in the DB (fee derivation becomes pure addition) -//! - Compresses fees to 2 bytes on the wire -//! - Gives ~0.78% precision per step -//! - Uses **no floating-point arithmetic** — all conversions are pure integer ops -//! -//! The key trick: multiplying by 129/128 in integer math is `x + (x >> 7)`. -//! Exponentiation uses a precomputed table of 15 entries with binary -//! exponentiation (at most 15 fixed-point multiplications). -//! -//! The precomputed table and [`MAX_EXPONENT`] are generated at build time by -//! `build.rs` using exact integer arithmetic (iterated fixed-point squaring). -//! Any reimplementation (e.g. in C++) must use the same table values to -//! guarantee bit-identical fee calculations. See `build.rs` for the algorithm. +//! `build.rs` generates the 15-entry fixed-point table and [`MAX_EXPONENT`]. +//! Independent implementations must reproduce the conversion bit-for-bit: +//! fee differences can change validation decisions and application state. use alloy_primitives::U256; @@ -51,19 +41,27 @@ type U512 = alloy_primitives::Uint<512, 8>; /// Convert a log-space fee exponent to a linear [`U256`] value. /// -/// `fee_to_linear(n)` = `floor((129/128)^n)`. +/// Let `S = 2^64`. `build.rs` generates `T[0] = 129 * 2^57` and +/// `T[i] = floor(T[i-1] * T[i-1] / S)` for `i = 1, …, 14`. /// -/// Uses a precomputed table with binary exponentiation: at most 15 fixed-point -/// multiplications, no floats. +/// Start `R = S`. Visit bits `i = 0, …, 14` in ascending order; for each set bit +/// of `n`, replace `R` with `floor(R * T[i] / S)`. Return `floor(R / S)`. +/// Every multiplication widens its 256-bit operands to a 512-bit product +/// before shifting right by 64. Intermediate flooring and accumulation order +/// are part of the contract: using the same table in a different order, or +/// computing the exact rational power and flooring only once, is not a +/// compatible replacement. /// /// # Panics /// -/// Panics if the result would overflow `U256` (exponent > [`MAX_EXPONENT`]). +/// Panics for `n > MAX_EXPONENT`. The bound protects the full fixed-point +/// accumulator, including its 64 fractional bits, rather than just the final +/// integer result. pub fn fee_to_linear(log_fee: u16) -> U256 { fee_to_linear_fixed(log_fee) >> FRAC_BITS } -/// Compute `(129/128)^n` in fixed-point representation (64 fractional bits). +/// Compute the rounded fixed-point accumulator specified by [`fee_to_linear`]. /// /// Used internally for higher-precision comparisons in binary search. fn fee_to_linear_fixed(log_fee: u16) -> U256 { @@ -82,7 +80,9 @@ fn fee_to_linear_fixed(log_fee: u16) -> U256 { /// Convert a linear fee value to the nearest log-space exponent. /// -/// `fee_from_linear(v)` = `round(log_{129/128}(v))`. +/// Choose the exponent whose rounded fixed-point value is closest to `value`, +/// breaking ties toward the smaller exponent. Values at or above +/// `fee_to_linear(MAX_EXPONENT)` saturate to [`MAX_EXPONENT`]. /// /// Returns 0 for `value <= 1` (since `(129/128)^0 = 1`). ///