[Patrick-Ehimen]

What

Makes a dropped WebSocket lossless: the indexer now reconnects with exponential backoff and resumes from a persisted per-chain high-water mark instead of going dark until the process restarts.

Closes #2.

Why

The issue's premise turned out not to hold, which changed the design.

#2 assumes a dropped connection surfaces as the log stream ending. It usually doesn't. In alloy 2.1 PubSubService catches the disconnect itself, reconnects with its own capped exponential backoff (max_retries: 10, retry_interval: 3s) and re-issues every active subscription (alloy-pubsub-2.1.0/src/service.rs:71-110). The broadcast channel survives, so stream.next() never observes the drop — but the logs produced during the gap are gone with no signal. with_max_retries(0) doesn't opt out either: reconnect() is attempted once before the retry count is checked.

Two consequences:

1. Resume-on-stream-end would rarely fire. It only triggers after alloy burns all 10 retries — the case in the issue's pasted error. Every shorter blip silently loses events.
2. There's a second silent-loss path. SubscriptionStream swallows broadcast Lagged errors with a debug! and a comment saying "This is OK" (sub.rs:392-395). Since process() awaits a DB insert inline per matching log, a slow database drops logs with no error.

So a supervisor and a high-water mark are both necessary but neither delivers "no events are missed" on its own.

How

The live subscription is a latency optimisation; a periodic reconcile sweep is the correctness guarantee. Both run under tokio::try_join! in ChainRunner::run. Each interval the sweep re-scans eth_getLogs from the persisted cursor to the tip, which closes gaps from silent reconnects, broadcast lag and cold restarts through a single code path. Inserts are already idempotent via ON CONFLICT DO NOTHING, so the overlap between the two costs nothing.

• supervisor.rs rebuilds the provider and runner on any fault. 1s → 30s capped backoff, reset after a 5 minute stable window so a long-lived process doesn't keep waiting the maximum delay after one blip.
• backoff.rs holds that policy as pure logic, so it is testable without I/O.
• chain_cursors persists the per-chain high-water mark. The upsert uses GREATEST so a runner restarting mid-sweep cannot rewind progress another runner already made.
• resolve_start_block prefers the cursor, then EVM_START_BLOCK, then the tip. This retires the u64::MAX sentinel that made backfill a silent no-op by default.

Two bugs found while reviewing the above, both fixed here:

• EVM_RECONCILE_INTERVAL_SECS=0 panicked every chain task. tokio::time::interval asserts a non-zero period (interval.rs:74). Now clamped to a 1s floor.
• The cursor turned transient insert failures into permanent silent loss. process() logged and continued, but the cursor then advanced past the unwritten event. fetch_chunk now propagates, so a failed write holds the cursor back and the next sweep retries. Live-path failures still only warn, since the sweep covers them.

Cursor writes land only after a chunk fully completes, so tearing the sweep down mid-flight costs at most one chunk of rework — cancellation-safety falls out of the ordering rather than needing explicit handling.

Testing

make ci green: fmt, clippy -D warnings, 43 unit tests (33 existing + 10 new covering the backoff curve, the stability reset, and the interval clamp).

Verified against Postgres 16 in Docker, not just compiled:

• Migration 0004 applies incrementally on top of an existing 0001–0003 database — the same situation Supabase is in — and is recorded success = t.
• shared/tests/chain_cursors.rs covers the absent-cursor case, round-trip, monotonicity, and a block number past u32::MAX to exercise the u64 → BIGINT conversion.
• The monotonicity test is not vacuous: replacing GREATEST(...) with a plain EXCLUDED.last_block makes it fail with left: Some(150), right: Some(200), then reverted.
• Tests are rerun-safe (each resets its row) and confirmed by running twice back to back.

Those tests are #[ignore]d so CI stays green without a database — make ci with DATABASE_URL unset gives 43 passed, 5 ignored. Run them with docker compose up -d postgres && make test-db.

⚠️ Do not deploy before #10

watched_addresses is still vec![], so the filter is unconstrained. The reconcile sweep now re-pulls every log on every chain on a timer, which multiplies the cost of that considerably. There's a FIXME(#10) at the construction site. I deliberately did not wire contracts::all_addresses here since that is #10's scope, but the two are now coupled in a way they weren't before.

Known trade-off

Failing the chunk on insert error means one permanently-unwritable event (say a constraint violation) stalls that chain's cursor: 5 failed sweeps → runner errors → supervisor restarts → same row. That's a loud backoff loop rather than quiet data loss, which I think is the right default for an explorer, but a dead-letter or skip-after-N would fix it properly. Worth a follow-up issue.

Checklist

Always

• cargo clippy --workspace --all-targets passes with no warnings
• cargo test --workspace passes
• .env.example updated if new env vars introduced — EVM_START_BLOCK (semantics changed) and EVM_RECONCILE_INTERVAL_SECS (new)

New protocol adapter

• N/A — no adapter changes.

Database / schema changes

• Migration file added under shared/migrations/ — 0004_create_chain_cursors.sql
• sqlx-data.json regenerated — N/A. The project uses runtime sqlx::query/query_as, not the compile-time-verified macros, so there is no sqlx-data.json/.sqlx to regenerate. Worth noting separately that CI sets SQLX_OFFLINE: true despite this, which implies a guarantee that isn't there.
• Backward-compatible or includes a rollback plan — purely additive. No existing table is touched, and the migration is CREATE TABLE IF NOT EXISTS. Rollback is DROP TABLE chain_cursors;, after which every chain falls back to EVM_START_BLOCK or the tip.

Note: Cargo.lock moves because tokio is added as a dev-dependency of shared for the integration tests — it must stay in the commit or CI's --locked will fail.