Skip to content

Fix issues found reviewing the code-health stack - #140

Merged
austin-barrington merged 6 commits into
refactor/lint-cleanupfrom
fix/stack-review-2026-11
Nov 20, 2026
Merged

austin-barrington merged 6 commits into
refactor/lint-cleanupfrom
fix/stack-review-2026-11

Conversation

@austin-barrington

Copy link
Copy Markdown
Member

Fixes from a review of the code-health stack (#133–#138). Based on #138.

Each fix has a test that failed before the change and passes after it.

Fixed

  • Drain ignored a refused handoff. consensus.propose returns Ok for a committed op that the apply refused (for example, a stale epoch). Drain never checked ClusterResponse.ok, so a refused TransferPrimary counted as handed off. It now fails, as the shard scheduler does. This predates the stack. Test: a_refused_primary_handoff_is_an_error.
  • Every DELETE on a restarted node waited 15 s. A reopened WAL set flushed_through to 0 but restored last_sequence from disk. Only the flush truncates the WAL, so everything below the oldest remaining entry has been flushed, and the mark now starts there. Before the fix, every DELETE and DROP SERIES on a restarted, idle node waited the full 15 s, and on replicated nodes that wait happened inside the Raft apply. Test: a_reopened_wal_reports_truncated_entries_flushed.
  • A refused DROP SERIES still dropped the series. It deleted every series of the measurement before the condition was checked, so after the error SHOW SERIES came back empty. The predicates are now built and checked first. Test: a_refused_drop_series_keeps_the_series.
  • DROP SERIES without FROM kept the rows on a single node. It dropped the series and deleted no rows, and tombstones no longer hide those rows. It now deletes them, as the sharded path already did. Test: drop_series_without_from_deletes_the_rows.
  • A sharded DELETE reported a delete that went through as failed. A peer waits up to 15 s for its WAL to flush and then deletes from its whole table. The coordinator sent one request per region, each with the 5 s peer timeout. A peer leading K regions therefore ran K full-table deletes, and under ingest the coordinator timed out and reported a failure while the peer still deleted the rows. Each primary now gets one request, with a timeout of the flush wait plus the peer timeout. Test: a_delete_reaches_each_primary_once_and_waits_for_it.
  • chDB error codes were matched by prefix. "Code: 60" also matched codes 600–609, so a delete that failed with one of them reported success. "Code: 47" also matched 470–479, so a replica stored that failure as a tombstone. Both now match the exact code. Tests: only_code_60_is_an_unknown_table, only_code_47_is_an_unknown_column.
  • Docs and doc attributes. The developer docs named the deleted sync_client.rs; they now name CatchUpService in application/cluster/catch_up.rs. Also removed a duplicate #[doc(hidden)] and a repeated doc line, and moved the apply_shard_map_op doc comment off mod apply;.

Not fixed

  • Forwarded proposals are retried after a timeout. propose_via retries a forward up to 3 times with a 10 s timeout, including after the leader may already have committed it. Schema mutations did this before the stack too. With the PeerRpc change, the scheduler and drain now use the same path. A DELETE's Raft apply can still wait up to one flush interval (10 s by default) under ingest, so a slow apply can be proposed again. The right fix is a choice between telling a timeout apart from a refused connection and not waiting for the flush inside the apply, so it is left for a decision.
  • Rolling upgrade. An old node treats a new coordinator's delete predicate (1 for a DELETE with no WHERE, or series_id IN (...)) as a tombstone, and that tombstone hides the matching rows permanently, including later writes. Don't run DELETE or DROP SERIES while versions are mixed.
  • Name-guard cache clearing. Every DELETE and DROP SERIES bumps drop_generation, which clears the physical-name guard and the ingest schema cache. The next write to each measurement then rescans the catalog. The swap-then-clear is also not atomic with concurrent inserts. Both are minor.
  • delete_series_matching ignores its predicate. It drops every series of the measurement, or of the whole RP when there is no FROM. This predates the stack and is already tracked as an open bug.

Verification

.superpowers/gate.sh: fmt, clippy -D warnings, lib, integration and CLI tests: GATE-OK on 1b80236.

consensus.propose returns Ok for a committed op that the apply refused
(a stale epoch). Drain ignored ClusterResponse.ok, so a refused
TransferPrimary counted as handed off. Check it, as the shard scheduler
does.
flushed_through started at 0 on open, while last_sequence is restored
from disk. Only the flush truncates, so every entry below the oldest one
left was flushed. With the mark at 0, every DELETE and DROP SERIES on a
restarted node that had nothing left to flush waited the full 15 s for
it, inside the Raft apply on replicated nodes.
drop_series wiped the measurement's series metadata, then built the
delete predicate, which refuses a field or an unknown tag. A refused
DROP SERIES returned an error and left SHOW SERIES empty. Build every
predicate first.

DROP SERIES with no FROM deleted rows only on a sharded node; on a single
node it dropped the series and kept every row, which tombstones no longer
hide. It now deletes them everywhere.
A peer deletes a DELETE's rows from its whole table after waiting up to
15 s for its WAL to flush. The coordinator sent one request per region on
the 5 s peer timeout: a peer leading K regions ran K full-table deletes,
and under steady ingest the coordinator timed out and reported a failed
delete that the peer then carried out. Send each primary one request, with
the flush wait plus the peer timeout to answer in.
is_unknown_table matched "Code: 60", which codes 600-609 also start
with, and a delete that failed with one reported success. is_unknown_column
matched "Code: 47" (470-479), and a replica stored such a failed delete
as a tombstone. Match "Code: 60." and "Code: 47.".
The deep-dive and developer guides still pointed at
adapters/cluster/sync_client.rs, which became CatchUpService in
application/cluster/catch_up.rs. Also drop a repeated #[doc(hidden)] and
a repeated doc line, and move apply_shard_map_op's doc comment from
`mod apply;` onto the function.
@cursor

cursor Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

PR Summary

Medium Risk
Touches WAL flush state, cluster drain/handoff, sharded delete fan-out, and DELETE/DROP SERIES paths—operational cluster and data-mutation behavior, though changes are narrowly scoped with tests.

Overview
Addresses several correctness bugs from a code-health review, plus minor doc alignment.

Drain / sharding: Graceful drain now treats a committed but refused TransferPrimary (e.g. stale epoch) as failure via shared propose_handoff, matching the shard scheduler. Sharded DELETE scatter sends one /internal/shard/delete per region primary (not per region) and uses a timeout of DELETE_FLUSH_WAIT (15s) plus the peer RPC budget so coordinators don’t report failure while peers are still flushing and deleting.

WAL / DELETE latency: On WAL reopen, flushed_through starts at one below the oldest retained sequence so truncated entries aren’t mistaken as unflushed—fixing idle restarts that forced every DELETE to wait the full 15s flush wait.

DROP SERIES / DELETE semantics: DROP SERIES builds and validates predicates before delete_series_matching, so refused statements no longer wipe series metadata. Without FROM, matching rows are deleted on single-node paths (aligned with sharded behavior).

chDB error classification: is_unknown_table and is_unknown_column match exact ClickHouse codes (Code: 60. / Code: 47.) so unrelated errors (e.g. 603, 473) aren’t treated as missing table/column.

Docs: Developer and deep-dive docs reference CatchUpService and http_peer_rpc instead of removed sync_client.rs.

Each behavioral fix includes a regression test.

Reviewed by Cursor Bugbot for commit 1b80236. Configure here.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 1b80236. Configure here.

counter!("hyperbytedb_shard_delete_applied_total").increment(1);
outcome.applied += 1;
continue;
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Primary skip misses transferred regions

Medium Severity

After one successful shard_delete to a primary, later snapshot regions with that same primary are counted applied and never sent. A transfer that finishes during that first RPC (now up to the 15s flush wait plus peer timeout) can move a remaining region to another node, so the coordinator reports a full delete while the new primary is never contacted.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 1b80236. Configure here.

@austin-barrington
austin-barrington merged commit 1b80236 into refactor/lint-cleanup Nov 20, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant