test(longhaul): add restart-resilience gate to the smoke workflow - #462
Draft
WentingWu666666 wants to merge 1 commit into
Draft
WentingWu666666 wants to merge 1 commit into
WentingWu666666 wants to merge 1 commit into
Conversation
Collaborator
Author
|
Superseded by an in-repo branch PR so the Long-Haul Smoke Gate runs automatically (fork PRs require manual workflow approval). Reopening from developer/wentingwu-longhaul-verifier-seed-fix. |
WentingWu666666
deleted the
developer/wentingwu/longhaul-verifier-seed-fix
branch
September 22, 2026 15:22
WentingWu666666
restored the
developer/wentingwu/longhaul-verifier-seed-fix
branch
September 22, 2026 15:23
WentingWu666666
marked this pull request as ready for review
September 22, 2026 15:32
WentingWu666666
requested review from
alaye-ms,
hossain-rayhan and
xgerman
as code owners
September 22, 2026 15:32
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Critical verifier correctness and availability issues remain, and regression coverage does not fully prove the injected failure path.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 2
Open (4)
What changed in this PR
Fixes false verifier data-loss reports after restart over retention-pruned prefixes by guarding transient sequence seeding.
Changes:
- Adds guarded seeding and fault injection.
- Adds unit, integration, and smoke-gate regression coverage.
- Wires configuration, CLI handling, and documentation.
| File | Summary |
|---|---|
test/longhaul/workload/verifier.go |
Adds guarded seeding and fault-injection handling. |
test/longhaul/workload/verifier_test.go |
Adds seed-guard unit tests. |
test/longhaul/workload/bugb_repro_test.go |
Adds the opt-in restart regression test. |
test/longhaul/README.md |
Documents fault injection. |
test/longhaul/config/config.go |
Parses and validates fault-injection configuration. |
test/longhaul/cmd/longhaul/main.go |
Passes configuration to the verifier. |
.github/workflows/longhaul-smoke.yml |
Adds restart-over-pruned-prefix coverage. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| expectedSeq, seeded, proceed, seedErr := seedGate(v.seeded[writerID], expectedSeq, func() (int64, error) { | ||
| return v.minSeq(ctx, writerID) | ||
| }) | ||
| v.seeded[writerID] = seeded |
Comment on lines
+272
to
+276
| if !proceed { | ||
| v.journal.Warn("verifier", fmt.Sprintf( | ||
| "min-seq seed not established for writer %s; skipping scan this cycle (retry next tick): %v", | ||
| writerID, seedErr)) | ||
| return |
Comment on lines
+78
to
+83
| if _, err := db.Collection(CollectionName).DeleteMany(ctx, bson.D{ | ||
| {Key: "writer_id", Value: "w000"}, | ||
| {Key: "seq", Value: bson.D{{Key: "$lte", Value: int64(prunedThrough)}}}, | ||
| }); err != nil { | ||
| t.Fatalf("prune prefix: %v", err) | ||
| } |
Comment on lines
+275
to
+281
| if v := os.Getenv(EnvFaultSeedMinSeqFailures); v != "" { | ||
| n, err := strconv.Atoi(v) | ||
| if err != nil { | ||
| return cfg, fmt.Errorf("invalid %s=%q: %w", EnvFaultSeedMinSeqFailures, v, err) | ||
| } | ||
| cfg.FaultSeedMinSeqFailures = n | ||
| } |
documentdb-triage-tool
Bot
added
bug
|
🤖 Auto-triaged by documentdb-triage-tool. Applied: Reasoningcomponent from path globs (test, ci, docs); P0 keyword "data loss" in body; effort from diff stats (359+35 LOC, 7 files) If a label is wrong, remove it manually and ping |
WentingWu666666
marked this pull request as draft
September 22, 2026 17:30
After the bounded Phase-A run has exercised the operations and retention pruning has deleted a prefix of the collection, restart the driver over the surviving data (RESET_DATA=false) with operations and backups disabled, and require it to resume cleanly and still report PASS. A pod restart wipes all in-memory driver state (writer cursors, verifier sequence tracking, pruner floors) while the on-disk collection survives, so this is a general guard that every piece of state a restart must rebuild from the database is reconstructed correctly. It catches a whole class of restart-only regressions in the PR gate instead of only on the live long-haul cluster after a real restart. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: badfdbf1-0fe9-43da-9921-854304654217 Signed-off-by: Wenting Wu <wentingwu@microsoft.com>
WentingWu666666
force-pushed
the
developer/wentingwu/longhaul-verifier-seed-fix
branch
from
September 22, 2026 19:32
68d7029 to
725941f
Compare
WentingWu666666
changed the title
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
Adds a restart-resilience gate to the Long-Haul Smoke workflow.
After the existing bounded Phase-A run exercises the operation sequence and
retention pruning deletes a prefix of the collection (already asserted
docs-pruned > 0), the gate:longhaul-reportso the next verdict is read fresh.RESET_DATA=false), withoperations and backups disabled so the restart itself is the only thing
under test.
Why
A pod restart wipes every piece of in-memory driver state — writer cursors,
verifier sequence tracking, pruner floors — while the on-disk collection,
including any retention-pruned prefix, survives. This gate is a general guard
that all state a restart must rebuild purely from the database is reconstructed
correctly, so an entire class of restart-only regressions is caught by the PR
gate instead of only surfacing on the live long-haul cluster after a real
restart.
Scope: workflow-only — no driver/source changes.