Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/deep-dive/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,6 @@ Subsystem walkthroughs with pointers to source files. When a deep dive disagrees
| [Read path](deep-dive-read-path.md) | TimeseriesQL → ClickHouse SQL → chDB | `timeseriesql/`, `application/query_service.rs`, `adapters/chdb/` |
| [Clustering](deep-dive-clustering.md) | Replication, Raft, sync, hinted handoff, **series sharding** | `domain/cluster/`, `application/cluster/`, `application/shard_*.rs`, `adapters/cluster/` |
| [Compaction](deep-dive-compaction.md) | MergeTree background merges | chDB / ClickHouse engine |
| [Self-repair](deep-dive-self-repair.md) | Peer convergence via replication and sync | `adapters/cluster/sync_client.rs`, `replication_log.rs` |
| [Self-repair](deep-dive-self-repair.md) | Peer convergence via replication and sync | `application/cluster/catch_up.rs`, `replication_log.rs` |

Replication peer sends use exponential backoff starting at 1s, capped at 30s, up to `replication_max_retries` attempts.
12 changes: 6 additions & 6 deletions docs/deep-dive/deep-dive-clustering.md
Original file line number Diff line number Diff line change
Expand Up @@ -145,10 +145,10 @@ Called during application startup when `cluster.enabled = true`.
### `run_startup_sync(config, metadata, wal, data_dir)`

1. Set node state to `Syncing`.
2. Build a `SyncClient`.
2. Build a `CatchUpService`.
3. Determine if this is a new node (no existing data) or reconnect:
- **New node:** Call `sync_client.join_and_sync()`.
- **Existing node:** Call `sync_client.reconnect_sync()`.
- **New node:** Call `catch_up.join_and_sync()`.
- **Existing node:** Call `catch_up.reconnect_sync()`.
4. Retry up to 5 times on failure.
5. On success: set state to `Active`.
6. On failure after retries: set state to `Active` with `needs_sync = true` (allows the node to operate but flags it for later re-sync).
Expand All @@ -157,7 +157,7 @@ Called during application startup when `cluster.enabled = true`.

## 4. Startup Sync

**File:** `src/adapters/cluster/sync_client.rs`
**File:** `src/application/cluster/catch_up.rs` (`CatchUpService`; it reaches peers through the `PeerRpc` port, `src/adapters/cluster/http_peer_rpc.rs`)

### `join_and_sync()` -- Full join flow for new nodes

Expand All @@ -178,7 +178,7 @@ Called during application startup when `cluster.enabled = true`.

## 5. Reconnect Sync

**File:** `src/adapters/cluster/sync_client.rs`
**File:** `src/application/cluster/catch_up.rs` (`CatchUpService`; it reaches peers through the `PeerRpc` port, `src/adapters/cluster/http_peer_rpc.rs`)

### `reconnect_sync()` -- For existing nodes reconnecting after downtime

Expand Down Expand Up @@ -457,7 +457,7 @@ This ensures peers that are catching up can still read needed WAL entries.
Peers stay aligned through three mechanisms:

1. **Write replication** — after local WAL append, the coordinator fans out line protocol to peers (`ReplicationPort` / `PeerClient`).
2. **Startup and reconnect sync** — `SyncClient` exchanges metadata snapshots and WAL tail entries via `/internal/sync/{manifest,metadata,wal}`.
2. **Startup and reconnect sync** — `CatchUpService` exchanges metadata snapshots and WAL tail entries via `/internal/sync/{manifest,metadata,wal}`.
3. **Local flush** — each peer replays its WAL (including replicated entries) into local MergeTree tables via `PointsSinkPort`.

There is no separate file-repair loop. chDB storage on each node is derived from the shared WAL + metadata contract.
Expand Down
2 changes: 1 addition & 1 deletion docs/deep-dive/deep-dive-self-repair.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
In clustered deployments, replicas converge through:

1. **Write replication** — line protocol fan-out after each local WAL append.
2. **Metadata + WAL sync** — startup and reconnect paths in `SyncClient` (`/internal/sync/manifest`, `/internal/sync/metadata`, `/internal/sync/wal`).
2. **Metadata + WAL sync** — startup and reconnect paths in `CatchUpService` (`/internal/sync/manifest`, `/internal/sync/metadata`, `/internal/sync/wal`).
3. **Local flush** — each node replays WAL entries (including replicated writes) into its own MergeTree tables.

Peer ack tracking in `ReplicationLog` informs safe WAL truncation so lagging nodes can catch up.
Expand Down
2 changes: 1 addition & 1 deletion docs/developer-guide/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ graph TB
RocksWAL["RocksDB WAL"]
RocksMeta["RocksDB Metadata"]
ChDBAdapter["chDB query + native sink"]
ClusterIO["cluster/ peer_client, sync_client, raft"]
ClusterIO["cluster/ peer_client, http_peer_rpc, raft"]
end

InboundAdapters --> App
Expand Down
2 changes: 1 addition & 1 deletion docs/developer-guide/internals/core-modules.md
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,7 @@ Axum handlers, router (`AppState`), middleware, auth, internal cluster/Raft endp
| File / dir | Description |
|------------|-------------|
| `peer_client.rs` | Outbound replication HTTP client. Implements `ReplicationPort`. |
| `sync_client.rs` | Join/reconnect sync: metadata + WAL catch-up. |
| `http_peer_rpc.rs` | `PeerRpc` over HTTP. Join/reconnect sync itself is `CatchUpService` in `application/cluster/catch_up.rs`. |
| `replication_log.rs` | RocksDB-backed per-peer WAL/mutation ack tracking. |
| `hinted_handoff.rs` | Per-peer queued hints for unreachable peers. |
| `transport.rs` | Shared HTTP transport abstraction for cluster I/O. |
Expand Down
2 changes: 1 addition & 1 deletion docs/developer-guide/system-architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -152,7 +152,7 @@ src/
├── http/ Axum handlers + internal cluster endpoints
├── chdb/ QueryPort + PointsSinkPort (native adapter)
├── wal/, metadata/, auth.rs
└── cluster/ peer_client, sync_client, replication_log, raft/
└── cluster/ peer_client, http_peer_rpc, replication_log, raft/
```

---
Expand Down
15 changes: 13 additions & 2 deletions hyperbytedb/src/adapters/chdb/native_adapter.rs
Original file line number Diff line number Diff line change
Expand Up @@ -141,7 +141,7 @@ fn table_schema_from_measurement_meta(

/// A chDB error for a table that does not exist.
fn is_unknown_table(msg: &str) -> bool {
msg.contains("UNKNOWN_TABLE") || msg.contains("Code: 60") || msg.contains("doesn't exist")
msg.contains("UNKNOWN_TABLE") || msg.contains("Code: 60.") || msg.contains("doesn't exist")
}

fn tag_column_kind(distinct_values: usize, threshold: usize) -> ColumnKind {
Expand Down Expand Up @@ -480,7 +480,6 @@ impl ChdbNativeAdapter {
///
/// Test support.
#[doc(hidden)]
#[doc(hidden)]
pub fn engine_probe_count(&self) -> u64 {
self.engine_probes
.load(std::sync::atomic::Ordering::Relaxed)
Expand Down Expand Up @@ -2688,6 +2687,18 @@ mod tests {
use arrow::array::Float64Array;
use std::collections::BTreeMap;

/// A delete that fails for another reason must not read as "no table,
/// nothing to delete": codes 600-609 start with "Code: 60" too.
#[test]
fn only_code_60_is_an_unknown_table() {
assert!(is_unknown_table(
"Code: 60. DB::Exception: Unknown table expression identifier 'x'. (UNKNOWN_TABLE)"
));
assert!(!is_unknown_table(
"Code: 603. DB::Exception: Cannot open file. (CANNOT_OPEN_FILE)"
));
}

fn make_point(ts: i64, tags: &[(&str, &str)], fields: &[(&str, FieldValue)]) -> Point {
let mut tag_map = BTreeMap::new();
for (k, v) in tags {
Expand Down
28 changes: 27 additions & 1 deletion hyperbytedb/src/adapters/wal/rocksdb_wal.rs
Original file line number Diff line number Diff line change
Expand Up @@ -243,7 +243,9 @@ impl RocksDbWal {
Ok(Self {
db,
seq: Arc::new(AtomicU64::new(seq)),
flushed_through: AtomicU64::new(0),
// Only the flush truncates, so every entry below the oldest one
// left was flushed.
flushed_through: AtomicU64::new(first.saturating_sub(1)),
truncated_through: Arc::new(AtomicU64::new(first)),
range_deletes_unflushed: Arc::new(AtomicU64::new(0)),
sizes: Arc::new(std::sync::Mutex::new(std::collections::VecDeque::new())),
Expand Down Expand Up @@ -800,6 +802,30 @@ mod tests {
}
}

/// Only the flush truncates the WAL, so every entry below the oldest one
/// left was flushed. A reopened WAL says so: a DELETE on a restarted
/// node waits for `flushed_through` to reach `last_sequence`, and with
/// nothing left to flush it waited its full 15 s every time.
#[tokio::test]
async fn a_reopened_wal_reports_truncated_entries_flushed() {
let dir = TempDir::new().unwrap();
{
let wal = RocksDbWal::open(dir.path()).unwrap();
for _ in 0..5 {
wal.append(cpu_entry(1)).await.unwrap();
}
wal.truncate_before(4).await.unwrap();
}
let wal = RocksDbWal::open(dir.path()).unwrap();
assert_eq!(wal.flushed_through(), 3);
// All flushed and truncated: caught up with the last sequence.
wal.truncate_before(6).await.unwrap();
drop(wal);
let wal = RocksDbWal::open(dir.path()).unwrap();
assert_eq!(wal.flushed_through(), wal.last_sequence().await.unwrap());
assert_eq!(wal.flushed_through(), 5);
}

fn memtable_counts(wal: &RocksDbWal) -> (u64, u64) {
let cf = wal.db.cf_handle(WAL_CF).unwrap();
let get = |p: &str| wal.db.property_int_value_cf(&cf, p).unwrap().unwrap();
Expand Down
66 changes: 61 additions & 5 deletions hyperbytedb/src/application/cluster/drain.rs
Original file line number Diff line number Diff line change
Expand Up @@ -233,17 +233,13 @@ impl DrainService {
.await?;
}

use crate::domain::cluster::raft_request::ClusterRequest;
let tp = ShardMapOp::TransferPrimary {
key: key.clone(),
region_id: region.region_id,
new_primary,
epoch: region.epoch,
};
consensus
.propose(ClusterRequest::ShardMapMutation(Box::new(tp)))
.await
.map_err(|e| HyperbytedbError::ShardMap(e.to_string().into()))?;
propose_handoff(consensus.as_ref(), tp).await?;
}
Ok(())
}
Expand Down Expand Up @@ -294,6 +290,27 @@ impl DrainService {
}

/// The measurements materialized views write into.
/// Propose one primary handoff. A committed op the apply refused (a stale
/// epoch) moved nothing, so it is an error, as in the shard scheduler.
async fn propose_handoff(
consensus: &dyn ConsensusPort,
op: ShardMapOp,
) -> Result<(), HyperbytedbError> {
use crate::domain::cluster::raft_request::ClusterRequest;
let resp = consensus
.propose(ClusterRequest::ShardMapMutation(Box::new(op)))
.await
.map_err(|e| HyperbytedbError::ShardMap(e.to_string().into()))?;
if !resp.ok {
return Err(HyperbytedbError::ShardMap(
resp.message
.unwrap_or_else(|| "primary handoff refused".to_string())
.into(),
));
}
Ok(())
}

async fn view_destinations(
metadata: &dyn MetadataPort,
) -> Result<std::collections::HashSet<MeasurementKey>, HyperbytedbError> {
Expand Down Expand Up @@ -399,6 +416,45 @@ mod tests {
);
}

/// A consensus whose apply refuses every op, as a stale epoch does.
struct RefusingConsensus;

impl crate::ports::leadership::LeadershipPort for RefusingConsensus {
fn current_leader(&self) -> Option<u64> {
Some(1)
}
}

#[async_trait]
impl ConsensusPort for RefusingConsensus {
async fn propose(
&self,
_req: crate::domain::cluster::raft_request::ClusterRequest,
) -> Result<crate::domain::cluster::raft_request::ClusterResponse, HyperbytedbError>
{
Ok(crate::domain::cluster::raft_request::ClusterResponse {
ok: false,
message: Some("stale epoch".into()),
})
}
}

/// A TransferPrimary the apply refused moved nothing: the handoff fails
/// instead of counting the region as handed off.
#[tokio::test]
async fn a_refused_primary_handoff_is_an_error() {
let op = ShardMapOp::TransferPrimary {
key: MeasurementKey::new("db", "autogen", "cpu"),
region_id: 1,
new_primary: 2,
epoch: Default::default(),
};
let err = propose_handoff(&RefusingConsensus, op)
.await
.expect_err("a refused handoff must fail");
assert!(err.to_string().contains("stale epoch"), "{err}");
}

fn make_node(id: u64, state: NodeState) -> NodeInfo {
NodeInfo {
node_id: id,
Expand Down
2 changes: 1 addition & 1 deletion hyperbytedb/src/application/predicate_sql.rs
Original file line number Diff line number Diff line change
Expand Up @@ -194,7 +194,7 @@ pub async fn delete_rows(
/// Longest wait for the flush service to write the WAL's current entries:
/// past one flush interval (10 s by default), well inside the query
/// timeout (30 s by default).
const DELETE_FLUSH_WAIT: std::time::Duration = std::time::Duration::from_secs(15);
pub(crate) const DELETE_FLUSH_WAIT: std::time::Duration = std::time::Duration::from_secs(15);

async fn wait_for_flush(wal: &dyn crate::ports::wal::WalPort) -> Result<(), HyperbytedbError> {
let target = wal.last_sequence().await?;
Expand Down
11 changes: 7 additions & 4 deletions hyperbytedb/src/application/query_service/statements/delete.rs
Original file line number Diff line number Diff line change
Expand Up @@ -100,11 +100,11 @@ pub(in crate::application::query_service) async fn drop_series(
String::new()
};
if measurement.is_some() || !predicate_sql.is_empty() {
svc.metadata
.delete_series_matching(target_db, &rp, measurement.as_deref(), &predicate_sql)
.await?;
// The series' rows are deleted; points written to them later
// stay. A tombstone used to hide those too.
//
// The predicates are built (and a field or unknown tag refused)
// before anything is dropped: an error must leave the series.
let mut targets: Vec<(String, String)> = Vec::new();
if let Some(ref meas) = measurement {
let pred = crate::application::predicate_sql::build_delete_predicate(
Expand All @@ -116,7 +116,7 @@ pub(in crate::application::query_service) async fn drop_series(
)
.await?;
targets.push((meas.clone(), pred));
} else if svc.shard_routing.is_some() {
} else {
// Every measurement that has the condition's tags.
for meas in svc.metadata.list_measurements(target_db).await? {
if let Ok(pred) = crate::application::predicate_sql::build_delete_predicate(
Expand All @@ -132,6 +132,9 @@ pub(in crate::application::query_service) async fn drop_series(
}
}
}
svc.metadata
.delete_series_matching(target_db, &rp, measurement.as_deref(), &predicate_sql)
.await?;
for (meas, pred) in targets {
if let Some(ctx) = svc.shard_routing.as_ref() {
scatter_delete_to_regions(
Expand Down
19 changes: 18 additions & 1 deletion hyperbytedb/src/application/schema_mutation_apply.rs
Original file line number Diff line number Diff line change
Expand Up @@ -165,7 +165,7 @@ pub(crate) fn is_unknown_column(msg: &str) -> bool {
msg.contains("UNKNOWN_IDENTIFIER")
|| msg.contains("Unknown identifier")
|| msg.contains("Missing columns")
|| msg.contains("Code: 47")
|| msg.contains("Code: 47.")
}

async fn apply_schema_mutation_inner(
Expand Down Expand Up @@ -401,6 +401,23 @@ pub async fn revoke_privilege(
}
}

#[cfg(test)]
mod unknown_column_tests {
use super::is_unknown_column;

/// Only code 47 is a missing column; 470-479 are other failures and a
/// replica must report them, not store the delete as a tombstone.
#[test]
fn only_code_47_is_an_unknown_column() {
assert!(is_unknown_column(
"Code: 47. DB::Exception: Unknown expression identifier `host`. (UNKNOWN_IDENTIFIER)"
));
assert!(!is_unknown_column(
"Code: 473. DB::Exception: Possible deadlock avoided. (DEADLOCK_AVOIDED)"
));
}
}

#[cfg(test)]
mod drop_space_tests {
use super::drop_space_for;
Expand Down
Loading