Skip to content

Test phase 7: fault points and kill-and-restart; four durability fixes - #190

Merged
jburrow merged 1 commit into
mainfrom
claude/project-thread-bssnnv
Oct 23, 2026
Merged

jburrow merged 1 commit into
mainfrom
claude/project-thread-bssnnv

Conversation

@jburrow

@jburrow jburrow commented Oct 23, 2026

Copy link
Copy Markdown
Owner

Phase 7 of the test plan (reviews/test-strategy-2026-10-23.md): failpoints at the remaining I/O and panic sites, and a kill -9 chaos run against the real server binary.

Fault points

src/faults.rs (failpoints feature): parse.read now also returns OS errors (EMFILE) and delays (slow disk); new parse.panic, parse.symbols, watch.overflow, save.write, save.rename. Ten new tests in tests/fault_injection.rs: a file gone between listing and reading, out of file handles (build and watcher), read panics (once and always, build and watcher), a parser panic, lost OS events made up by a rescan, a slow disk not holding up searches, a failed save or rename keeping the old index whole, and updates whose save failed being saved at shutdown.

Kill and restart

examples/chaos.rs kills the server with SIGKILL during a build, a load, a stream of edits, a periodic save (0–20 ms after it begins), a burst and the shutdown save, then restarts and requires: the saved index loads (never rejected), no temp file left, the index catches up with every change, /api/check?all=true clean, no crash. 60 rounds on 20,000 files locally: all recovered, 22 kills landed mid-save. New job "Kill and restart" in the "Scale scoreboard" workflow (40 rounds, seed 1 on PRs; 100 rounds nightly). The scoreboard's server handling moves to examples/support/server.rs.

Fixed

  • A save killed half way left its temp file (as large as the index) for good; the server now removes them when it takes the index lock.
  • A failed save (disk full) reset the unsaved-update count, so the shutdown skipped its save.
  • A failed rename deleted the old index before retrying; it is now moved aside and put back if the retry fails.
  • A panic while reading or parsing a file during the build left it out without a word; it is now retried and reported like an unreadable file.
  • The index folder is fsynced after the rename on Unix.

Docs: testing page section "Killed at any moment", persistence page, index format guarantees, CHANGELOG.


Generated by Claude Code

Fault points (src/faults.rs, `failpoints` feature): parse.read now also
takes OS errors (EMFILE) and delays (slow disk); new parse.panic,
parse.symbols, watch.overflow, save.write and save.rename. Ten new tests
in tests/fault_injection.rs.

examples/chaos.rs kills the server binary with SIGKILL during a build, a
load, a stream of edits, a periodic save, a burst and the shutdown save,
restarts it and checks it loads the index, leaves no temp file, catches
up and passes /api/check. New job "Kill and restart" in the scoreboard
workflow. The scoreboard's server handling moves to
examples/support/server.rs, shared by both.

Fixed:
- a killed save left its temp file (as large as the index) for good; the
  server now removes them when it takes the index lock;
- a failed save (disk full) reset the unsaved-update count, so the
  shutdown skipped its save;
- a failed rename deleted the old index before retrying; it is now moved
  aside and put back on a second failure;
- a panic while reading or parsing a file during the build left it out
  silently; it is now retried and reported like an unreadable file.
Also fsyncs the index folder after the rename on Unix.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EkCcM7SF9X6MgaCjoysdAw
@coderabbitai

coderabbitai Bot commented Oct 23, 2026

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: ba308c9e-e109-4361-bcff-99ca88b63839
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@jburrow
jburrow merged commit aed93a5 into main Oct 23, 2026
12 checks passed
@jburrow
jburrow deleted the claude/project-thread-bssnnv branch October 23, 2026 21:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants