Test phase 7: fault points and kill-and-restart; four durability fixes

.github/workflows/scoreboard.yml

@@ -12,6 +12,11 @@ name: Scale scoreboard
 # Nightly with the full ten-minute stream; on pull requests that touch
 # locking, watching, saving or limits with a two-minute stream, and the
 # table in the job summary.
+#
+# The second job kills the server with SIGKILL at random moments (during a
+# build, a load, a stream of edits, a burst, a shutdown save), starts it
+# again and checks the index comes back right (examples/chaos.rs). Pull
+# requests use a fixed seed; other runs a new one, printed in the log.
 
 on:
   schedule:
@@ -30,7 +35,10 @@ on:
       - "src/diagnostics/**"
       - "src/config.rs"
       - "src/main.rs"
+      - "src/index/persistence.rs"
+      - "src/index_lock.rs"
       - "examples/scoreboard.rs"
+      - "examples/chaos.rs"
       - "examples/support/**"
       - "benches/scoreboard/**"
       - ".github/workflows/scoreboard.yml"
@@ -93,3 +101,48 @@ jobs:
             scoreboard.json
             ${{ runner.temp }}/score/run/server-*.log
           if-no-files-found: ignore
+
+  chaos:
+    name: Kill and restart
+    runs-on: ubuntu-latest
+    timeout-minutes: 90
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install Protoc
+        run: sudo apt-get update && sudo apt-get install -y protobuf-compiler
+
+      - name: Setup Rust toolchain (pinned by rust-toolchain.toml)
+        uses: dtolnay/rust-toolchain@master
+        with:
+          toolchain: "1.98.1"
+
+      - name: Cache cargo
+        uses: Swatinem/rust-cache@v2
+
+      - name: Build the server and the harness
+        run: cargo build --release --bins --example chaos
+
+      - name: Kill and restart
+        run: |
+          target/release/examples/chaos --dir "$RUNNER_TEMP/chaos" \
+            --files 20000 --rounds "$ROUNDS" --seed "$SEED" \
+            --markdown chaos.md --json chaos.json
+        env:
+          ROUNDS: ${{ github.event_name == 'pull_request' && '40' || '100' }}
+          SEED: ${{ github.event_name == 'pull_request' && '1' || github.run_id }}
+
+      - name: Report
+        if: always()
+        run: test -f chaos.md && cat chaos.md >> "$GITHUB_STEP_SUMMARY" || true
+
+      - name: Keep the rounds and the logs of failed ones
+        if: always()
+        uses: actions/upload-artifact@v7
+        with:
+          name: chaos
+          path: |
+            chaos.md
+            chaos.json
+            ${{ runner.temp }}/chaos/run/failed-*.log
+          if-no-files-found: ignore

CHANGELOG.md

@@ -8,6 +8,26 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## [Unreleased]
 
 ### Fixed
+- **Every save killed half way left a file as large as the index.** A
+  save writes a temp file next to the index and renames it into place; a
+  server killed during the save (`kill -9`, a power cut, the OOM killer)
+  left the temp file, and nothing removed it. The server now removes them
+  when it starts.
+- **A save that failed was forgotten.** With `save_after_updates`, the
+  count of unsaved updates was reset whether the save worked or not, so
+  after a save onto a full disk the shutdown skipped its save and the
+  saved index lacked those updates (the next start re-read the files, so
+  no search was wrong, but it took longer). The count now stays until a
+  save works.
+- **A failed rename could delete the index.** When moving the new index
+  into place failed, the old index was deleted before trying again, so a
+  second failure left no index at all. The old index is now moved aside
+  and put back if the second try fails.
+- **A panic while reading a file left it out of the index without a
+  word.** During the build, a file whose reading or parsing panicked was
+  skipped and not reported. It is now tried again at the end of the
+  build, and reported as unreadable if it keeps failing, like a file that
+  cannot be read.
 - **Every search waited 4 to 5 s soon after a restart.** An index loaded
   from disk keeps its unresolved imports (`import os`, packages outside
   the indexed folders) but not each file's import list. At the end of the
@@ -142,6 +162,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   made up from the corpus (`--generate-regex`), and
   `tests/discovery_oracle.rs` compares the indexed files and the FILES box
   with `rg`.
+- **Kill and restart.** `examples/chaos.rs` kills the server binary with
+  SIGKILL at random moments (during a build, a load from disk, a stream of
+  edits, a periodic save, a burst of changes, and the save at shutdown),
+  starts it again and checks that it loads the saved index, leaves no temp
+  file, catches up with every change and passes `/api/check`. The "Scale
+  scoreboard" workflow runs it nightly and on pull requests that touch
+  saving or watching (job "Kill and restart").
+- **More fault points** (`--features failpoints`, `src/faults.rs`): out of
+  file handles, a file gone between listing and reading, a slow disk, a
+  panic while reading or parsing a file, events lost by the OS, a save
+  onto a full disk and a failed rename, each with a test in
+  `tests/fault_injection.rs`.
 - **A scale scoreboard.** `examples/scoreboard.rs` runs the server binary
   over HTTP in seven scenarios (cold build, warm restart, branch switch, a
   ten-minute stream of edits, overloaded searching, saves under load, and

docs/book/src/how-it-works/persistence.md

@@ -12,6 +12,26 @@ named temporary file next to the target, fsynced, and renamed into place.
 Two saves cannot interleave, and a crash at any point leaves the previous
 file intact. Checkpoints during a long build use the same path.
 
+- **A save that is killed** (`kill -9`, a power cut, the out-of-memory
+  killer) leaves its temporary file (`<index_path>` with `.tmp.<pid>.<n>`
+  in place of its extension, as large as the index). The next server to
+  take the index lock removes them before it loads ("Removed temp files
+  left by an index save that did not finish" in the log).
+- **A failed rename** (Windows refuses to replace a file another process
+  has open) is tried once more with the old index moved aside; if that
+  fails too, the old index is put back. The save fails, and the index on
+  disk is the previous one.
+- **A failed save** (disk full) is logged as an error, and with
+  `save_after_updates` the updates it held still count: the next save,
+  after more updates or at shutdown, writes them. The index on disk is
+  older until then, so the next start re-reads the files changed since.
+
+`tests/fault_injection.rs` fails the write and the rename on purpose
+(fault points `save.write` and `save.rename`), and `examples/chaos.rs`
+kills the server binary during saves, builds and shutdowns and checks that
+the next start loads the index and catches up (see
+[How fcs is tested](testing.md#killed-at-any-moment)).
+
 ## One server per index
 
 A server takes an exclusive lock on `<index_path>.lock` before it loads

docs/book/src/how-it-works/testing.md

@@ -26,11 +26,12 @@ wrote every line and knows the answer.
 | `tests/rank_snapshot.rs` | the first 20 hits and the total of about a hundred searches; the same corpus at two places on disk must rank the same | `tests/snapshots/rank_order.txt`, changed only on purpose | `cargo test --test rank_snapshot` |
 | `tests/discovery_oracle.rs` | which files are indexed (the build and the watcher's per-file check) on random trees of `.gitignore`, `.ignore`, `.rgignore` and `.git/info/exclude` rules; the FILES box with a regex | `rg --files`, and `rg` over the path list | `cargo test --test discovery_oracle` (needs `rg`) |
 | `tests/lock_sim.rs` | the engine lock and the watcher's install steps under minutes of made-up load (back-to-back half-second searches, one 20 s search among quick ones, a busy server during a branch switch): every search answered, no search waits past its hold-back plus one step, backlogs drain at half speed or better, a slow search delays only the writer | bounds derived from the protocol, on a simulated clock (`fast_code_search::sim`): the real lock code, the same schedule on every run and OS, four minutes in about a second | `cargo test --test lock_sim` (`-- --ignored --nocapture` prints the tuning table) |
-| `tests/fault_injection.rs` | failed reads, panics while applying changes, lost changes: each is retried, reported and repaired | expected recovery | `cargo test --features failpoints --test fault_injection` |
+| `tests/fault_injection.rs` | failed reads (permission, out of file handles, a file gone between listing and reading), a slow disk, panics while reading, parsing or applying changes, lost changes and lost events, a save onto a full disk or a failed rename: each is retried, reported or repaired, searches are not held up, and the saved index stays whole | expected recovery | `cargo test --features failpoints --test fault_injection` (fault points are listed in `src/faults.rs`) |
 | `examples/rg_backtest.rs` | fixed and generated queries and regexes on tokio, Django and fixtures, with paging and default limits, on an index built in one go, restarted from disk (`--history reload`) and filled by the watcher (`--history watch`) | `rg` | see [Correctness against ripgrep](../benchmarks/correctness.md) |
 | `examples/graph_oracle.rs` | the import graph on Django, Vite and generated deep trees | grimp, TypeScript's resolver | CI job "graph oracle" |
 | `examples/ref_oracle.rs`, `examples/impact_oracle.rs` | Python references and diff impact on real repositories | Jedi | CI jobs "ref oracle", "impact oracle" |
 | `examples/scoreboard.rs` | the server binary over HTTP in seven scenarios (cold build, warm restart, branch switch, ten-minute stream, slow mix, saves under load, event-queue overflow): searches turned away, latency, time until the index matches the disk, memory, CPU, diagnostic captures, crashes, `/api/check` | targets in `benches/scoreboard/targets.toml` | nightly and on pull requests that touch locking or watching (workflow "Scale scoreboard"), see [Scale scoreboard](../benchmarks/scoreboard.md) |
+| `examples/chaos.rs` | `kill -9` of the server binary at random moments during a build, a load, a stream of edits, a periodic save, a burst and the shutdown save; after each, a restart must load the saved index (never reject it), leave no temp file, catch up with every change and pass `/api/check` | the files on disk | nightly and on pull requests that touch saving or watching (workflow "Scale scoreboard", job "Kill and restart"), see [below](#killed-at-any-moment) |
 | `examples/watch_load.rs`, `examples/limits_bench.rs` | searches answered and latency while a large tree changes; where limits bite | measured numbers | by hand, see [Incremental updates](incremental.md#how-this-is-tested) |
 
 Every test that drives the index also runs its self-checks
@@ -90,6 +91,51 @@ NODE_PATH=/tmp/pw/node_modules FCS_REQUIRE_BROWSER=1 cargo test --release --test
 Without Playwright the leg prints a note and is skipped; CI's Linux test
 job sets `FCS_REQUIRE_BROWSER`, so there it fails instead.
 
+## Killed at any moment
+
+`examples/chaos.rs` starts the server binary on a generated tree of 20,000
+files and kills it with SIGKILL, round after round, at a random moment of
+one of these:
+
+| Kind | Killed |
+|---|---|
+| `build` | during a build from scratch, which saves a checkpoint every quarter of the tree |
+| `load` | while it loads the saved index and checks the tree |
+| `stream` | while 300 files a second are rewritten, created and removed, and saved every 300 updates; the edits go on for a second after the kill |
+| `save` | 0 to 20 ms after one of those saves begins (it watches for the save's temp file every millisecond) |
+| `burst` | while the watcher installs a tenth of the files rewritten at once |
+| `shutdown` | 0 to 20 ms after the save at shutdown (SIGTERM) begins |
+
+After each kill it starts the server again and requires: the saved index
+is loaded, or there was none (a kill must never leave one the next start
+rejects); no temp file of a killed save is left once the server answers;
+the server becomes ready and catches up (nothing pending, the changed
+files indexed as they are on disk); `/api/check?all=true` finds nothing;
+and it does not crash. The seed is printed, and `--seed N` replays the
+same kill moments; a failing round keeps its server log as
+`failed-<round>.log`.
+
+60 rounds on a 4 vCPU VM, 2026-10-23 (seed 4):
+
+| Killed during | Rounds | Killed mid save | No index after | Index rejected | Temp files left | Slowest ready | Slowest catch-up | Check problems |
+|---|---|---|---|---|---|---|---|---|
+| build | 10 | 1 | 3 | 0 | 0 | 1.8 s | 1.8 s | 0 |
+| load | 10 | 0 | 0 | 0 | 0 | 0.3 s | 0.5 s | 0 |
+| stream | 10 | 1 | 0 | 0 | 0 | 0.4 s | 0.8 s | 0 |
+| save | 10 | 10 | 0 | 0 | 0 | 0.3 s | 0.8 s | 0 |
+| burst | 10 | 0 | 0 | 0 | 0 | 0.3 s | 0.8 s | 0 |
+| shutdown | 10 | 10 | 0 | 0 | 0 | 0.4 s | 0.8 s | 0 |
+
+"No index after" is a build killed before its first checkpoint: the next
+start builds from scratch. A build killed later loads the checkpoint and
+indexes the rest. CI runs 40 rounds on pull requests (seed 1) and 100
+nightly (a new seed).
+
+This tests a killed process, not a power cut: the kernel still writes out
+what the process wrote. A save fsyncs its file before the rename and, on
+Linux and macOS, the folder after it, so a power cut should leave either
+the old index or the new one; nothing here simulates that.
+
 ## Reproducing a failure
 
 - proptest suites print the shrunk sequence or query that failed, which is
@@ -133,6 +179,11 @@ generated Python trees with grimp (job "graph oracle").
   while the import graph's parked imports were retried in one lock hold;
   a `backlog` diagnostic fired under steady edits the watcher kept up with
   (scale scoreboard).
+- A save killed half way left a temp file as large as the index, for
+  good; a save that failed (disk full) was forgotten, so the shutdown
+  skipped its save; a failed rename deleted the index before trying
+  again; and a panic while reading a file during the build left it out
+  without a word (fault points and the kill-and-restart run).
 - Earlier probes found a crash when a large file was rewritten during a
   search, "server unavailable" at the end of a build, and lost changes
   after failed reads or a panic; the fault and watch tests cover them.

docs/book/src/reference/index-format.md

@@ -35,7 +35,8 @@ live ids on load, so ids stay stable across restarts.
 
 - **Atomic.** A save writes a unique temporary file next to the target,
   fsyncs it, and renames it into place; saves are serialized within the
-  process. A crash leaves the previous index intact.
+  process. A crash leaves the previous index intact, and the next start
+  removes the temporary file a killed save left.
 - **Detects drift.** Each file's mtime and size at read time are stored;
   on load a file whose stat differs is re-read.
 - **Pinned.** `tests/fixtures/index-v8.fcsidx` is compared byte for byte