Skip to content
Open
2 changes: 1 addition & 1 deletion docs/deep-dive/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,6 @@ Subsystem walkthroughs with pointers to source files. When a deep dive disagrees
| [Read path](deep-dive-read-path.md) | TimeseriesQL → ClickHouse SQL → chDB | `timeseriesql/`, `application/query_service.rs`, `adapters/chdb/` |
| [Clustering](deep-dive-clustering.md) | Replication, Raft, sync, hinted handoff, **series sharding** | `domain/cluster/`, `application/cluster/`, `application/shard_*.rs`, `adapters/cluster/` |
| [Compaction](deep-dive-compaction.md) | MergeTree background merges | chDB / ClickHouse engine |
| [Self-repair](deep-dive-self-repair.md) | Peer convergence via replication and sync | `adapters/cluster/sync_client.rs`, `replication_log.rs` |
| [Self-repair](deep-dive-self-repair.md) | Peer convergence via replication and sync | `application/cluster/catch_up.rs`, `replication_log.rs` |

Replication peer sends use exponential backoff starting at 1s, capped at 30s, up to `replication_max_retries` attempts.
12 changes: 6 additions & 6 deletions docs/deep-dive/deep-dive-clustering.md
Original file line number Diff line number Diff line change
Expand Up @@ -145,10 +145,10 @@ Called during application startup when `cluster.enabled = true`.
### `run_startup_sync(config, metadata, wal, data_dir)`

1. Set node state to `Syncing`.
2. Build a `SyncClient`.
2. Build a `CatchUpService`.
3. Determine if this is a new node (no existing data) or reconnect:
- **New node:** Call `sync_client.join_and_sync()`.
- **Existing node:** Call `sync_client.reconnect_sync()`.
- **New node:** Call `catch_up.join_and_sync()`.
- **Existing node:** Call `catch_up.reconnect_sync()`.
4. Retry up to 5 times on failure.
5. On success: set state to `Active`.
6. On failure after retries: set state to `Active` with `needs_sync = true` (allows the node to operate but flags it for later re-sync).
Expand All @@ -157,7 +157,7 @@ Called during application startup when `cluster.enabled = true`.

## 4. Startup Sync

**File:** `src/adapters/cluster/sync_client.rs`
**File:** `src/application/cluster/catch_up.rs` (`CatchUpService`; it reaches peers through the `PeerRpc` port, `src/adapters/cluster/http_peer_rpc.rs`)

### `join_and_sync()` -- Full join flow for new nodes

Expand All @@ -178,7 +178,7 @@ Called during application startup when `cluster.enabled = true`.

## 5. Reconnect Sync

**File:** `src/adapters/cluster/sync_client.rs`
**File:** `src/application/cluster/catch_up.rs` (`CatchUpService`; it reaches peers through the `PeerRpc` port, `src/adapters/cluster/http_peer_rpc.rs`)

### `reconnect_sync()` -- For existing nodes reconnecting after downtime

Expand Down Expand Up @@ -457,7 +457,7 @@ This ensures peers that are catching up can still read needed WAL entries.
Peers stay aligned through three mechanisms:

1. **Write replication** — after local WAL append, the coordinator fans out line protocol to peers (`ReplicationPort` / `PeerClient`).
2. **Startup and reconnect sync** — `SyncClient` exchanges metadata snapshots and WAL tail entries via `/internal/sync/{manifest,metadata,wal}`.
2. **Startup and reconnect sync** — `CatchUpService` exchanges metadata snapshots and WAL tail entries via `/internal/sync/{manifest,metadata,wal}`.
3. **Local flush** — each peer replays its WAL (including replicated entries) into local MergeTree tables via `PointsSinkPort`.

There is no separate file-repair loop. chDB storage on each node is derived from the shared WAL + metadata contract.
Expand Down
2 changes: 1 addition & 1 deletion docs/deep-dive/deep-dive-self-repair.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
In clustered deployments, replicas converge through:

1. **Write replication** — line protocol fan-out after each local WAL append.
2. **Metadata + WAL sync** — startup and reconnect paths in `SyncClient` (`/internal/sync/manifest`, `/internal/sync/metadata`, `/internal/sync/wal`).
2. **Metadata + WAL sync** — startup and reconnect paths in `CatchUpService` (`/internal/sync/manifest`, `/internal/sync/metadata`, `/internal/sync/wal`).
3. **Local flush** — each node replays WAL entries (including replicated writes) into its own MergeTree tables.

Peer ack tracking in `ReplicationLog` informs safe WAL truncation so lagging nodes can catch up.
Expand Down
2 changes: 1 addition & 1 deletion docs/developer-guide/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ graph TB
RocksWAL["RocksDB WAL"]
RocksMeta["RocksDB Metadata"]
ChDBAdapter["chDB query + native sink"]
ClusterIO["cluster/ peer_client, sync_client, raft"]
ClusterIO["cluster/ peer_client, http_peer_rpc, raft"]
end

InboundAdapters --> App
Expand Down
2 changes: 1 addition & 1 deletion docs/developer-guide/internals/core-modules.md
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,7 @@ Axum handlers, router (`AppState`), middleware, auth, internal cluster/Raft endp
| File / dir | Description |
|------------|-------------|
| `peer_client.rs` | Outbound replication HTTP client. Implements `ReplicationPort`. |
| `sync_client.rs` | Join/reconnect sync: metadata + WAL catch-up. |
| `http_peer_rpc.rs` | `PeerRpc` over HTTP. Join/reconnect sync itself is `CatchUpService` in `application/cluster/catch_up.rs`. |
| `replication_log.rs` | RocksDB-backed per-peer WAL/mutation ack tracking. |
| `hinted_handoff.rs` | Per-peer queued hints for unreachable peers. |
| `transport.rs` | Shared HTTP transport abstraction for cluster I/O. |
Expand Down
2 changes: 1 addition & 1 deletion docs/developer-guide/system-architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -152,7 +152,7 @@ src/
├── http/ Axum handlers + internal cluster endpoints
├── chdb/ QueryPort + PointsSinkPort (native adapter)
├── wal/, metadata/, auth.rs
└── cluster/ peer_client, sync_client, replication_log, raft/
└── cluster/ peer_client, http_peer_rpc, replication_log, raft/
```

---
Expand Down
9 changes: 9 additions & 0 deletions hyperbytedb-cli/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -39,3 +39,12 @@ axum = "0.8"
serial_test = "3"
tempfile = "3"
tokio = { version = "1", features = ["macros", "rt-multi-thread", "net"] }

[lints.clippy]
# Performance lints CI enforces through -D warnings. significant_drop_tightening
# stays off: too many false positives.
redundant_clone = "warn"
large_futures = "warn"
needless_collect = "warn"
iter_with_drain = "warn"
or_fun_call = "warn"
9 changes: 9 additions & 0 deletions hyperbytedb-proxy/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -43,3 +43,12 @@ futures = "0.3"
criterion = { version = "0.5", features = ["html_reports"] }
tokio = { version = "1", features = ["rt-multi-thread", "macros"] }


[lints.clippy]
# Performance lints CI enforces through -D warnings. significant_drop_tightening
# stays off: too many false positives.
redundant_clone = "warn"
large_futures = "warn"
needless_collect = "warn"
iter_with_drain = "warn"
or_fun_call = "warn"
3 changes: 3 additions & 0 deletions hyperbytedb-proxy/src/pool.rs
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,9 @@ impl BackendPool {
}

/// Replace the backend snapshot (tests and benchmarks only).
///
/// Test support.
#[doc(hidden)]
#[doc(hidden)]
pub async fn set_backends_for_test(&self, backends: Vec<Arc<Backend>>) {
let mut guard = self.backends.write().await;
Expand Down
9 changes: 9 additions & 0 deletions hyperbytedb/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -149,3 +149,12 @@ harness = false
[dev-dependencies]
criterion = { version = "0.5", features = ["html_reports"] }
serial_test = "3"

[lints.clippy]
# Performance lints CI enforces through -D warnings. significant_drop_tightening
# stays off: too many false positives.
redundant_clone = "warn"
large_futures = "warn"
needless_collect = "warn"
iter_with_drain = "warn"
or_fun_call = "warn"
14 changes: 7 additions & 7 deletions hyperbytedb/benches/ingestion_columnar.rs
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,7 @@ fn bench_metadata(c: &mut Criterion) {
std::fs::create_dir_all(&meta_dir).unwrap();
let metadata: Arc<RocksDbMetadata> = Arc::new(RocksDbMetadata::open(&meta_dir).unwrap());
rt.block_on(metadata.create_database("benchdb")).unwrap();
let meta_port: Arc<dyn MetadataPort> = metadata.clone();
let meta_port: Arc<dyn MetadataPort> = metadata;

let limits = IngestCardinalityLimits::default();
let cache = IngestSchemaCache::new();
Expand Down Expand Up @@ -139,7 +139,7 @@ fn bench_wal(c: &mut Criterion) {
let metadata: Arc<RocksDbMetadata> = Arc::new(RocksDbMetadata::open(&meta_dir).unwrap());
let wal: Arc<RocksDbWal> = Arc::new(RocksDbWal::open(&wal_dir).unwrap());
rt.block_on(metadata.create_database("benchdb")).unwrap();
let meta_port: Arc<dyn MetadataPort> = metadata.clone();
let meta_port: Arc<dyn MetadataPort> = metadata;

let limits = IngestCardinalityLimits::default();
let cache = IngestSchemaCache::new();
Expand Down Expand Up @@ -210,7 +210,7 @@ fn bench_metadata_fast(c: &mut Criterion) {
std::fs::create_dir_all(&meta_dir).unwrap();
let metadata: Arc<RocksDbMetadata> = Arc::new(RocksDbMetadata::open(&meta_dir).unwrap());
rt.block_on(metadata.create_database("benchdb")).unwrap();
let meta_port: Arc<dyn MetadataPort> = metadata.clone();
let meta_port: Arc<dyn MetadataPort> = metadata;

let limits = IngestCardinalityLimits::default();
let cache = IngestSchemaCache::new();
Expand Down Expand Up @@ -244,7 +244,7 @@ fn bench_wal_fast(c: &mut Criterion) {
let metadata: Arc<RocksDbMetadata> = Arc::new(RocksDbMetadata::open(&meta_dir).unwrap());
let wal: Arc<RocksDbWal> = Arc::new(RocksDbWal::open(&wal_dir).unwrap());
rt.block_on(metadata.create_database("benchdb")).unwrap();
let meta_port: Arc<dyn MetadataPort> = metadata.clone();
let meta_port: Arc<dyn MetadataPort> = metadata;

let limits = IngestCardinalityLimits::default();
let cache = IngestSchemaCache::new();
Expand Down Expand Up @@ -323,7 +323,7 @@ fn bench_metadata_concurrent(c: &mut Criterion) {
std::fs::create_dir_all(&meta_dir).unwrap();
let metadata: Arc<RocksDbMetadata> = Arc::new(RocksDbMetadata::open(&meta_dir).unwrap());
rt.block_on(metadata.create_database("benchdb")).unwrap();
let meta_port: Arc<dyn MetadataPort> = metadata.clone();
let meta_port: Arc<dyn MetadataPort> = metadata;
let cache = Arc::new(IngestSchemaCache::new());
let limits = IngestCardinalityLimits::default();

Expand Down Expand Up @@ -377,7 +377,7 @@ fn bench_wal_concurrent(c: &mut Criterion) {
let metadata: Arc<RocksDbMetadata> = Arc::new(RocksDbMetadata::open(&meta_dir).unwrap());
let wal: Arc<RocksDbWal> = Arc::new(RocksDbWal::open(&wal_dir).unwrap());
rt.block_on(metadata.create_database("benchdb")).unwrap();
let meta_port: Arc<dyn MetadataPort> = metadata.clone();
let meta_port: Arc<dyn MetadataPort> = metadata;
let cache = Arc::new(IngestSchemaCache::new());
let limits = IngestCardinalityLimits::default();

Expand Down Expand Up @@ -452,7 +452,7 @@ fn bench_wal_batched_concurrent(c: &mut Criterion) {
let metadata: Arc<RocksDbMetadata> = Arc::new(RocksDbMetadata::open(&meta_dir).unwrap());
let raw_wal: Arc<RocksDbWal> = Arc::new(RocksDbWal::open(&wal_dir).unwrap());
rt.block_on(metadata.create_database("benchdb")).unwrap();
let meta_port: Arc<dyn MetadataPort> = metadata.clone();
let meta_port: Arc<dyn MetadataPort> = metadata;
// `BatchingWal::new` spawns a background `batcher_loop` task; it
// must be created inside a tokio runtime context.
let wal: Arc<BatchingWal> =
Expand Down
10 changes: 5 additions & 5 deletions hyperbytedb/benches/ingestion_line_protocol.rs
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,7 @@ fn bench_metadata(c: &mut Criterion) {
std::fs::create_dir_all(&meta_dir).unwrap();
let metadata: Arc<RocksDbMetadata> = Arc::new(RocksDbMetadata::open(&meta_dir).unwrap());
rt.block_on(metadata.create_database("benchdb")).unwrap();
let meta_port: Arc<dyn MetadataPort> = metadata.clone();
let meta_port: Arc<dyn MetadataPort> = metadata;

let limits = IngestCardinalityLimits::default();
let cache = IngestSchemaCache::new();
Expand Down Expand Up @@ -118,7 +118,7 @@ fn bench_wal(c: &mut Criterion) {
let metadata: Arc<RocksDbMetadata> = Arc::new(RocksDbMetadata::open(&meta_dir).unwrap());
let wal: Arc<RocksDbWal> = Arc::new(RocksDbWal::open(&wal_dir).unwrap());
rt.block_on(metadata.create_database("benchdb")).unwrap();
let meta_port: Arc<dyn MetadataPort> = metadata.clone();
let meta_port: Arc<dyn MetadataPort> = metadata;

let limits = IngestCardinalityLimits::default();
let cache = IngestSchemaCache::new();
Expand Down Expand Up @@ -198,7 +198,7 @@ fn bench_metadata_concurrent(c: &mut Criterion) {
std::fs::create_dir_all(&meta_dir).unwrap();
let metadata: Arc<RocksDbMetadata> = Arc::new(RocksDbMetadata::open(&meta_dir).unwrap());
rt.block_on(metadata.create_database("benchdb")).unwrap();
let meta_port: Arc<dyn MetadataPort> = metadata.clone();
let meta_port: Arc<dyn MetadataPort> = metadata;
let cache = Arc::new(IngestSchemaCache::new());
let limits = IngestCardinalityLimits::default();

Expand Down Expand Up @@ -254,7 +254,7 @@ fn bench_wal_concurrent(c: &mut Criterion) {
let metadata: Arc<RocksDbMetadata> = Arc::new(RocksDbMetadata::open(&meta_dir).unwrap());
let wal: Arc<RocksDbWal> = Arc::new(RocksDbWal::open(&wal_dir).unwrap());
rt.block_on(metadata.create_database("benchdb")).unwrap();
let meta_port: Arc<dyn MetadataPort> = metadata.clone();
let meta_port: Arc<dyn MetadataPort> = metadata;
let cache = Arc::new(IngestSchemaCache::new());
let limits = IngestCardinalityLimits::default();

Expand Down Expand Up @@ -329,7 +329,7 @@ fn bench_wal_batched_concurrent(c: &mut Criterion) {
let metadata: Arc<RocksDbMetadata> = Arc::new(RocksDbMetadata::open(&meta_dir).unwrap());
let raw_wal: Arc<RocksDbWal> = Arc::new(RocksDbWal::open(&wal_dir).unwrap());
rt.block_on(metadata.create_database("benchdb")).unwrap();
let meta_port: Arc<dyn MetadataPort> = metadata.clone();
let meta_port: Arc<dyn MetadataPort> = metadata;
let wal: Arc<BatchingWal> =
rt.block_on(async { BatchingWal::new(raw_wal.clone(), 2048, 512, Duration::ZERO, 0) });
let cache = Arc::new(IngestSchemaCache::new());
Expand Down
5 changes: 2 additions & 3 deletions hyperbytedb/benches/support/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,7 @@ pub fn setup_flush(profile: DatasetProfile) -> FlushBenchEnv {
let shared = SharedSession::new_eager(chdb_dir.to_str().unwrap(), 1).unwrap();
let points_sink: Arc<dyn PointsSinkPort> = Arc::new(ChdbNativeAdapter::new(shared));

let ingestion = IngestionServiceImpl::new(wal.clone(), metadata.clone(), 100_000, 10_000, 0);
let ingestion = IngestionServiceImpl::new(wal.clone(), metadata, 100_000, 10_000, 0);
let flush_service = Arc::new(FlushServiceImpl::new(wal, 0, points_sink));

FlushBenchEnv {
Expand Down Expand Up @@ -173,8 +173,7 @@ pub fn setup(profile: DatasetProfile) -> BenchEnv {

let ingestion = IngestionServiceImpl::new(wal.clone(), metadata.clone(), 100_000, 10_000, 0);
let flush_service = FlushServiceImpl::new(wal.clone(), 0, points_sink.clone());
let query_service =
QueryServiceImpl::new(query_port, metadata.clone(), wal.clone(), 120, points_sink);
let query_service = QueryServiceImpl::new(query_port, metadata, wal, 120, points_sink);

seed_dataset(&rt, &ingestion, &flush_service, profile);

Expand Down
32 changes: 21 additions & 11 deletions hyperbytedb/src/adapters/chdb/native_adapter.rs
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ use crate::ports::points_sink::{PointsSinkPort, WriteAck};

/// Default threshold when no `[chdb].tag_low_cardinality_max` or cardinality
/// config is available (e.g. unit tests constructing the adapter directly).
#[cfg(test)]
pub const DEFAULT_TAG_LOW_CARDINALITY_MAX: usize = 100_000;

/// Column kind for tags and fields. Tags use `LowCardinality(String)` only
Expand Down Expand Up @@ -140,7 +141,7 @@ fn table_schema_from_measurement_meta(

/// A chDB error for a table that does not exist.
fn is_unknown_table(msg: &str) -> bool {
msg.contains("UNKNOWN_TABLE") || msg.contains("Code: 60") || msg.contains("doesn't exist")
msg.contains("UNKNOWN_TABLE") || msg.contains("Code: 60.") || msg.contains("doesn't exist")
}

fn tag_column_kind(distinct_values: usize, threshold: usize) -> ColumnKind {
Expand Down Expand Up @@ -476,6 +477,8 @@ impl ChdbNativeAdapter {
/// True if `series_id` is already registered (dimension row inserted +
/// persisted) for this table.
/// How many times `ensure_table` has probed the engine's columns.
///
/// Test support.
#[doc(hidden)]
pub fn engine_probe_count(&self) -> u64 {
self.engine_probes
Expand Down Expand Up @@ -774,16 +777,11 @@ impl ChdbNativeAdapter {
let tag_ddl_needed = tag_phys
.iter()
.any(|(_, phys, kind)| series_cached.columns.get(phys).copied() != Some(*kind));
let missing_fields: Vec<&(String, String, u8)> = field_phys
let fields_missing = field_phys
.iter()
.filter(|(_, phys, d)| !cached.knows(phys, ColumnKind::Field(*d)))
.collect();
.any(|(_, phys, d)| !cached.knows(phys, ColumnKind::Field(*d)));

if cached.materialized
&& series_cached.materialized
&& !tag_ddl_needed
&& missing_fields.is_empty()
{
if cached.materialized && series_cached.materialized && !tag_ddl_needed && !fields_missing {
if probed {
// Nothing missing: the cache already holds every engine column.
let mut guard = self.schemas.write();
Expand Down Expand Up @@ -1353,7 +1351,7 @@ impl ChdbNativeAdapter {
measurements: vec![PreparedMeasurementBatch {
measurement: wire.measurement.clone(),
table_name: ensured.table.clone(),
series_table_name: ensured.series_table.clone(),
series_table_name: ensured.series_table,
batch: Arc::new(batch),
row_count: n,
min_time,
Expand Down Expand Up @@ -2689,6 +2687,18 @@ mod tests {
use arrow::array::Float64Array;
use std::collections::BTreeMap;

/// A delete that fails for another reason must not read as "no table,
/// nothing to delete": codes 600-609 start with "Code: 60" too.
#[test]
fn only_code_60_is_an_unknown_table() {
assert!(is_unknown_table(
"Code: 60. DB::Exception: Unknown table expression identifier 'x'. (UNKNOWN_TABLE)"
));
assert!(!is_unknown_table(
"Code: 603. DB::Exception: Cannot open file. (CANNOT_OPEN_FILE)"
));
}

fn make_point(ts: i64, tags: &[(&str, &str)], fields: &[(&str, FieldValue)]) -> Point {
let mut tag_map = BTreeMap::new();
for (k, v) in tags {
Expand Down Expand Up @@ -3000,7 +3010,7 @@ mod tests {
"DDL must sort field columns by physical name, got: {ddl}"
);

let mut sorted = field_phys.clone();
let mut sorted = field_phys;
sorted.sort_by(|a, b| a.1.cmp(&b.1));
let ensured = EnsuredTable {
table: "`db_rp_cpu`".to_string(),
Expand Down
1 change: 1 addition & 0 deletions hyperbytedb/src/adapters/cluster/hinted_handoff.rs
Original file line number Diff line number Diff line change
Expand Up @@ -216,6 +216,7 @@ impl HintedHandoff {
}

/// Total pending hints across all peers. O(peers).
#[cfg(test)]
pub fn total_pending(&self) -> Result<u64, HyperbytedbError> {
let counts = self.counts.lock().unwrap_or_else(|e| e.into_inner());
Ok(counts.values().sum())
Expand Down
2 changes: 2 additions & 0 deletions hyperbytedb/src/adapters/cluster/replication_log.rs
Original file line number Diff line number Diff line change
Expand Up @@ -184,6 +184,7 @@ impl ReplicationLog {
/// **Note:** Only considers peers that already have a `repl_ack` key. Peers that
/// have never successfully acked are omitted, which can overstate the minimum.
/// Prefer [`Self::min_max_wal_ack_for_peers`] for flush truncation decisions.
#[cfg(test)]
pub fn min_wal_ack(&self) -> Result<Option<u64>, HyperbytedbError> {
let cf = self
.db
Expand Down Expand Up @@ -256,6 +257,7 @@ impl ReplicationLog {
}

/// Read mutation log entries from a given sequence.
#[cfg(test)]
pub fn read_mutations_from(
&self,
from_seq: u64,
Expand Down
2 changes: 1 addition & 1 deletion hyperbytedb/src/adapters/http/router.rs
Original file line number Diff line number Diff line change
Expand Up @@ -202,7 +202,7 @@ pub fn build_router(state: Arc<AppState>) -> Router {
.route(
"/api/v1/chdb",
post(chdb::handle_chdb).layer(axum::middleware::from_fn_with_state(
auth_state.clone(),
auth_state,
auth_middleware::auth_layer,
)),
);
Expand Down
3 changes: 3 additions & 0 deletions hyperbytedb/src/adapters/wal/batching_wal.rs
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,9 @@ impl BatchingWal {
Instant,
oneshot::Sender<Result<(u64, BatchTiming), HyperbytedbError>>,
)>::with_capacity(batch_len);
// `drain`, not `into_iter`: `batch` is reused across loop turns
// and keeps its allocation.
#[allow(clippy::iter_with_drain)]
for req in batch.drain(..) {
bundles.push(req.bundle);
pre_encoded.push(req.pre_encoded);
Expand Down
Loading