Verification failed in Antithesis: backend.ConsistentIndex (1049) must be <= WAL.HardState.commit (1047)

[serathius]

Thanks to @nwnt work on #21972 we found another issue! Yey

This is not a new issue, was just masked as exit 2 code until we fixed it. Context: #21775

https://linuxfoundation.antithesis.com/search?search=v5veyJxIjp7Im4iOnsiciI6eyJoIjpbeyJoIjpbeyJjIjp0cnVlLCJmIjoibW9tZW50LnZ0aW1lIiwibyI6Im1hdGNoZXMiLCJ2IjoiMTIzLjE3ODY3MjEzOTk5ODUzIn1dLCJvIjoib3IifSx7ImgiOlt7ImMiOnRydWUsImYiOiJtb21lbnQuaW5wdXRfaGFzaCIsIm8iOiJtYXRjaGVzIiwidiI6Ii03OTgyNTUyMTI2MDg5MjA5NjU1In1dLCJvIjoib3IifV0sIm8iOiJhbmQifSwidCI6eyJnIjpmYWxzZSwibSI6IiJ9LCJ5Ijoibm9uZSJ9fSwicyI6IjM0MzA3MzNkZjRkNTdjOGI1MDYxZGEwZDUzNWJiZWMyLTU2LTE3In0

[ahrtr]

Is the data directory of each etcd member saved somewhere?

[ahrtr]

Also if Antithesis's test is deterministic, can we check whether we can reproduce this issue in 3.7, 3.6 and 3.5 respectively? So that we can tell whether it's a regression in new version, or just a long-standing legacy issue.

[om7057]

Hi @serathius @ahrtr, I spent some time tracing through this path and wanted to share my current understanding before working on a fix. Happy to help with this if the approach makes sense.

From what I can tell, this seems to come from an ordering window between the raft loop and the apply/backend commit flow.

The raft loop sends entries to the apply path before the WAL persistence step completes:

r.applyc <- ap
r.storage.Save(rd.HardState, rd.Entries)
notifyc <- struct{}{}

Meanwhile, the apply side can advance the in-memory consistent index, and the backend commit path can persist that value independently.

The sequence I’m looking at is roughly:

1. Current WAL state has HardState.Commit = 1047.
2. A later Ready batch containing entries 1048–1049 is sent to the apply goroutine.
3. Apply processing advances ConsistentIndex to 1049.
4. Backend commit persists that updated consistent index.
5. A crash happens before the corresponding HardState.Commit = 1049 update is durably reflected.

After restart, verification sees:

backend consistent index (1049) > WAL HardState commit (1047)

and treats it as an invalid state.

This looks related to the case already mentioned in bootstrap.go:

// If only `HardState.Commit` increases, HardState won't be persisted
// to disk, even though the committed entries might have already been
// applied. This can result in consistent_index > CommitIndex.

My understanding is that this does not necessarily indicate missing data: the entries may already exist in the WAL and can be replayed, while the persisted commit checkpoint is what lagged behind.

For fixing this, I see two possible directions:

Option A: Adjust the verification logic to account for this recovery case by validating against WAL entries, not only HardState.Commit.

The invariant would become closer to:

ConsistentIndex must correspond to entries available in the WAL

Rather than requiring it to always be below the last persisted commit checkpoint.

Option B: Prevent the backend from persisting a consistent index newer than the last known WAL commit state.

This keeps the current verification assumption but introduces tighter coupling between backend persistence and raft/WAL progress.

I’m leaning toward exploring Option A because it keeps the apply path unchanged and treats this as a recovery-time consistency check, but I’d appreciate feedback on whether that matches the intended invariant.

For reproducing this, I think the existing BackendAfterPreCommitHookPanic failpoint could be useful since it targets the window after backend state is committed. I can try adding a focused robustness test around this scenario first.

Does this direction make sense? I’d be happy to investigate further and work on the fix.