[nnastonen]

Since _PRERELEASE has been enabled, it exposed problems with replicas and synchronisation between them. This is an attempt to fix them.

Why

The Debug CI jobs (ASan, TSan, Coverage) intermittently hung until GitHub killed them after 6 hours, or crashed, with nothing in the log to diagnose. Release builds passed. The causes were real races in HomeStore plus bugs in tests and the test harness. Some were hidden because the "ASan" job wasn't actually built with ASan.

Changes

Product fixes

• RaftReplDev: skip data fetches for a repl dev that is no longer active, and drain in-flight fetches before leave_group() frees the raft server. A late fetch crashed the follower, and the other replicas waited forever;
• BlkReadTracker: guard the waiter list. wait_on() modified it under the bucket's shared lock only;
• home_log_store::truncate(): serialize concurrent truncates of a store;
• JournalVirtualDev: lock the descriptor map in open(), destroy() and load;
• CPWatchdog: it never ran (!= a || != b), and its checks were inverted. It now crashes the node on a stuck CP.

Tests

• Raft repl harness:
• Waits on other replicas are bounded (--wait_timeout_sec, default 600s) and report what they were waiting for;
• The IPC barrier is generation-based; consecutive barriers of the same phase used to let replicas through early;
• Spare replicas are counted in the register_listener() barrier;
• SoloReplDev: the replay check accounts for journal truncation by CPs and verifies the exact count;
• LogStore: keep flushing while waiting for inserts. The test's explicit-flush log devices could lose a flush to a parallel truncate;
• IndexCrashRecovery: don't trigger a CP into a simulated crash that has already started.

CI

• The ASan job now builds with ASan and UBSan; CMake ignored ADDRESS_SANITIZER_ON before. It uses 2 parallel compile jobs to fit in runner memory;
• Failing tests print their output, and each test has a 30-minute timeout;
• Hung or crashed tests print all thread stacks (stack_dump_launcher.sh with gdb), and test logs are uploaded as an artifact.

SDSTOR-26170

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 20.83333% with 38 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (dev/v8.x@1fc342e). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
src/lib/checkpoint/cp_mgr.cpp	13.63%	19 Missing ⚠️
src/lib/blkdata_svc/blk_read_tracker.cpp	0.00%	0 Missing and 4 partials ⚠️
src/lib/device/journal_vdev.cpp	0.00%	0 Missing and 4 partials ⚠️
src/lib/logstore/log_store.cpp	0.00%	2 Missing and 2 partials ⚠️
src/lib/replication/repl_dev/raft_repl_dev.cpp	57.14%	2 Missing and 1 partial ⚠️
src/lib/replication/service/raft_repl_service.cpp	40.00%	0 Missing and 3 partials ⚠️
src/include/homestore/replication/repl_dev.hpp	50.00%	1 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@             Coverage Diff             @@
##             dev/v8.x     #940   +/-   ##
===========================================
  Coverage            ?   49.18%           
===========================================
  Files               ?      112           
  Lines               ?    12790           
  Branches            ?     6061           
===========================================
  Hits                ?     6291           
  Misses              ?     2468           
  Partials            ?     4031           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[raakella1]

LG!

[raakella1]

LG!