Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions .changeset/replicated-engine.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
---
'@hyperdx/otel-collector': minor
---

feat: opt-in Replicated database and table engines for the schema seed

When `HYPERDX_OTEL_EXPORTER_CLICKHOUSE_REPLICATED=true`, the migrate tool
creates the target database with the `Replicated` engine and seeds
`ReplicatedMergeTree` / `ReplicatedSummingMergeTree` tables instead of
`MergeTree` / `SummingMergeTree`. If the database already exists with another
engine, it logs a warning and seeds the plain engines as before. Off by default.
With `ENABLE_PROMQL=true` on ClickHouse 26.10 or later, the inner tables of the
PromQL `metrics_ts` table are Replicated too; older servers keep plain ones.

The database is created only on the node migrate connects to. On a cluster, run
migrate against every replica, or create the `Replicated` database on each node
first (for example with `ON CLUSTER`); otherwise inserts that reach a node
without the database fail.

The migrate tool also warns when it targets a database other than `default` that
has no ClickStack tables yet while `default` still holds them, since those are
not migrated.

With `HYPERDX_OTEL_EXPORTER_CLICKHOUSE_FALLBACK_DATABASE` set (off by default),
a target database that does not exist yet yields to the fallback database when
the fallback already has ClickStack tables. The collector then writes to the
fallback database too.
9 changes: 9 additions & 0 deletions docker-compose.dev.yml
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,15 @@ services:
# HYPERDX_OTEL_EXPORTER_LOGS_TTL / _TRACES_TTL / _METRICS_TTL / _SESSIONS_TTL
# Apply TTL changes to already-existing tables too (default false):
# HYPERDX_OTEL_EXPORTER_RECONCILE_TABLE_TTL: 'true'
# Create the database with the Replicated engine and seed Replicated* table
# engines (default false). The database must not exist yet (or already be
# Replicated) and the server needs {shard}/{replica} macros. Migrate creates
# the database only on the node it connects to, so on a cluster run it
# against every replica or create the database on each node first:
# HYPERDX_OTEL_EXPORTER_CLICKHOUSE_REPLICATED: 'true'
# Keep using this database instead when HYPERDX_OTEL_EXPORTER_CLICKHOUSE_DATABASE
# does not exist yet but this one already has ClickStack tables (default off):
# HYPERDX_OTEL_EXPORTER_CLICKHOUSE_FALLBACK_DATABASE: 'default'
# Leave text index dictionary block sizes alone (default: set 8192 on 26.6+):
# HYPERDX_OTEL_EXPORTER_SET_TEXT_INDEX_BLOCK_SIZE: 'false'
ENABLE_PROMQL: 'true'
Expand Down
10 changes: 9 additions & 1 deletion docker/otel-collector/entrypoint.sh
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,15 @@ if [ "$HYPERDX_OTEL_EXPORTER_CREATE_LEGACY_SCHEMA" != "true" ]; then
# - CLICKHOUSE_TLS_SERVER_NAME_OVERRIDE: Server name for TLS verification
# - CLICKHOUSE_TLS_INSECURE_SKIP_VERIFY: Skip TLS verification (set to "true")
echo "🚀 Using Go-based migrate tool with TLS support 🔐"
migrate /etc/otel/schema/seed
# With HYPERDX_OTEL_EXPORTER_CLICKHOUSE_FALLBACK_DATABASE set, migrate records
# the database it seeded here so the collector writes to the same one.
RESOLVED_DATABASE_PATH="${HYPERDX_OTEL_EXPORTER_RESOLVED_DATABASE_PATH:-/tmp/hyperdx-otel-exporter-database}"
rm -f "$RESOLVED_DATABASE_PATH"
HYPERDX_OTEL_EXPORTER_RESOLVED_DATABASE_PATH="$RESOLVED_DATABASE_PATH" migrate /etc/otel/schema/seed
if [ -s "$RESOLVED_DATABASE_PATH" ]; then
export HYPERDX_OTEL_EXPORTER_CLICKHOUSE_DATABASE="$(cat "$RESOLVED_DATABASE_PATH")"
echo "Using ClickHouse database from migrate: $HYPERDX_OTEL_EXPORTER_CLICKHOUSE_DATABASE"
fi
fi

# Check if OPAMP_SERVER_URL is defined to determine mode
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
-- +goose Up
-- +goose NO TRANSACTION
CREATE DATABASE IF NOT EXISTS ${DATABASE};
-- The engine macro is empty unless HYPERDX_OTEL_EXPORTER_CLICKHOUSE_REPLICATED
-- applies, then it is " ENGINE = Replicated(...)" (see processSchemaDir).
CREATE DATABASE IF NOT EXISTS ${DATABASE}${DATABASE_ENGINE};
Comment thread
SpencerTorres marked this conversation as resolved.
Resolved

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 P2 — Replicated database is only created on the node migrate connects to

A Replicated database has to be created on every replica, either with ON CLUSTER or by running CREATE DATABASE on each node. Migrate runs a plain CREATE DATABASE IF NOT EXISTS over the single CLICKHOUSE_ENDPOINT. If collectors reach ClickHouse through a load balancer or one shared host, only that one node joins the database. Inserts routed to any other node then fail because the database does not exist there. Either document that the database must already exist on every node or be created by connecting to each node, or accept an optional cluster name and emit ON CLUSTER '<cluster>'. Add this to the changeset and to the docker-compose.dev.yml comment as well.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

2 changes: 1 addition & 1 deletion docker/otel-collector/schema/seed/00002_otel_logs.sql
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ CREATE TABLE IF NOT EXISTS ${DATABASE}.otel_logs
INDEX idx_log_attr_items LogAttributeItems TYPE text(tokenizer = 'array'),
INDEX idx_lower_body lower(Body) TYPE text(tokenizer = 'splitByNonAlpha')
)
ENGINE = MergeTree
ENGINE = ${ENGINE_PREFIX}MergeTree
PARTITION BY toDate(Timestamp)
ORDER BY (toStartOfFiveMinutes(Timestamp), ServiceName, Timestamp)
TTL toDateTime(Timestamp) + ${LOGS_TTL}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ CREATE TABLE IF NOT EXISTS ${DATABASE}.otel_logs
INDEX idx_log_attr_value mapValues(LogAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_lower_body lower(Body) TYPE tokenbf_v1(32768, 3, 0) GRANULARITY 8
)
ENGINE = MergeTree
ENGINE = ${ENGINE_PREFIX}MergeTree
PARTITION BY toDate(Timestamp)
ORDER BY (toStartOfFiveMinutes(Timestamp), ServiceName, Timestamp)
TTL toDateTime(Timestamp) + ${LOGS_TTL}
Expand Down
10 changes: 5 additions & 5 deletions docker/otel-collector/schema/seed/00003_otel_metrics.sql
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ CREATE TABLE IF NOT EXISTS ${DATABASE}.otel_metrics_gauge
INDEX idx_attr_value mapValues(Attributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_time_minmax TimeUnix TYPE minmax GRANULARITY 1
)
ENGINE = MergeTree
ENGINE = ${ENGINE_PREFIX}MergeTree
PARTITION BY toDate(TimeUnix)
ORDER BY (ServiceName, MetricName, toStartOfHour(TimeUnix), cityHash64(Attributes), TimeUnix)
TTL toDateTime(TimeUnix) + ${METRICS_TTL}
Expand Down Expand Up @@ -70,7 +70,7 @@ CREATE TABLE IF NOT EXISTS ${DATABASE}.otel_metrics_sum
INDEX idx_attr_value mapValues(Attributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_time_minmax TimeUnix TYPE minmax GRANULARITY 1
)
ENGINE = MergeTree
ENGINE = ${ENGINE_PREFIX}MergeTree
PARTITION BY toDate(TimeUnix)
ORDER BY (ServiceName, MetricName, toStartOfHour(TimeUnix), cityHash64(Attributes), TimeUnix)
TTL toDateTime(TimeUnix) + ${METRICS_TTL}
Expand Down Expand Up @@ -115,7 +115,7 @@ CREATE TABLE IF NOT EXISTS ${DATABASE}.otel_metrics_histogram
INDEX idx_attr_value mapValues(Attributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_time_minmax TimeUnix TYPE minmax GRANULARITY 1
)
ENGINE = MergeTree
ENGINE = ${ENGINE_PREFIX}MergeTree
PARTITION BY toDate(TimeUnix)
ORDER BY (ServiceName, MetricName, toStartOfHour(TimeUnix), cityHash64(Attributes), TimeUnix)
TTL toDateTime(TimeUnix) + ${METRICS_TTL}
Expand Down Expand Up @@ -164,7 +164,7 @@ CREATE TABLE IF NOT EXISTS ${DATABASE}.otel_metrics_exponential_histogram
INDEX idx_attr_value mapValues(Attributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_time_minmax TimeUnix TYPE minmax GRANULARITY 1
)
ENGINE = MergeTree
ENGINE = ${ENGINE_PREFIX}MergeTree
PARTITION BY toDate(TimeUnix)
ORDER BY (ServiceName, MetricName, toStartOfHour(TimeUnix), cityHash64(Attributes), TimeUnix)
TTL toDateTime(TimeUnix) + ${METRICS_TTL}
Expand Down Expand Up @@ -201,7 +201,7 @@ CREATE TABLE IF NOT EXISTS ${DATABASE}.otel_metrics_summary
INDEX idx_attr_value mapValues(Attributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_time_minmax TimeUnix TYPE minmax GRANULARITY 1
)
ENGINE = MergeTree
ENGINE = ${ENGINE_PREFIX}MergeTree
PARTITION BY toDate(TimeUnix)
ORDER BY (ServiceName, MetricName, toStartOfHour(TimeUnix), cityHash64(Attributes), TimeUnix)
TTL toDateTime(TimeUnix) + ${METRICS_TTL}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ CREATE TABLE IF NOT EXISTS ${DATABASE}.hyperdx_sessions
INDEX idx_log_attr_value mapValues(LogAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_lower_body lower(Body) TYPE tokenbf_v1(32768, 3, 0) GRANULARITY 8
)
ENGINE = MergeTree
ENGINE = ${ENGINE_PREFIX}MergeTree
PARTITION BY toDate(TimestampTime)
PRIMARY KEY (ServiceName, TimestampTime)
ORDER BY (ServiceName, TimestampTime, Timestamp)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ CREATE TABLE IF NOT EXISTS ${DATABASE}.hyperdx_sessions
INDEX idx_log_attr_value mapValues(LogAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_lower_body lower(Body) TYPE tokenbf_v1(32768, 3, 0) GRANULARITY 8
)
ENGINE = MergeTree
ENGINE = ${ENGINE_PREFIX}MergeTree
PARTITION BY toDate(TimestampTime)
PRIMARY KEY (ServiceName, TimestampTime)
ORDER BY (ServiceName, TimestampTime, Timestamp)
Expand Down
2 changes: 1 addition & 1 deletion docker/otel-collector/schema/seed/00005_otel_traces.sql
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ CREATE TABLE IF NOT EXISTS ${DATABASE}.otel_traces
INDEX idx_duration Duration TYPE minmax GRANULARITY 1,
INDEX idx_lower_span_name SpanName TYPE text(tokenizer = 'splitByNonAlpha', preprocessor=lower(SpanName))
)
ENGINE = MergeTree
ENGINE = ${ENGINE_PREFIX}MergeTree
PARTITION BY toDate(Timestamp)
ORDER BY (ServiceName, SpanName, toDateTime(Timestamp))
TTL toDate(Timestamp) + ${TRACES_TTL}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ CREATE TABLE IF NOT EXISTS ${DATABASE}.otel_traces
INDEX idx_duration Duration TYPE minmax GRANULARITY 1,
INDEX idx_lower_span_name lower(SpanName) TYPE tokenbf_v1(32768, 3, 0) GRANULARITY 8
)
ENGINE = MergeTree
ENGINE = ${ENGINE_PREFIX}MergeTree
PARTITION BY toDate(Timestamp)
ORDER BY (ServiceName, SpanName, toDateTime(Timestamp))
TTL toDate(Timestamp) + ${TRACES_TTL}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ CREATE TABLE IF NOT EXISTS ${DATABASE}.otel_logs_kv_rollup_15m
INDEX idx_count_minmax count TYPE minmax GRANULARITY 1,
INDEX idx_timestamp_minmax Timestamp TYPE minmax GRANULARITY 1
)
ENGINE = SummingMergeTree
ENGINE = ${ENGINE_PREFIX}SummingMergeTree
PARTITION BY toDate(Timestamp)
ORDER BY (ColumnIdentifier, Key, Timestamp, Value)
TTL Timestamp + ${LOGS_TTL}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ CREATE TABLE IF NOT EXISTS ${DATABASE}.otel_traces_kv_rollup_15m
INDEX idx_count_minmax count TYPE minmax GRANULARITY 1,
INDEX idx_timestamp_minmax Timestamp TYPE minmax GRANULARITY 1
)
ENGINE = SummingMergeTree
ENGINE = ${ENGINE_PREFIX}SummingMergeTree
PARTITION BY toDate(Timestamp)
ORDER BY (ColumnIdentifier, Key, Timestamp, Value)
TTL Timestamp + ${TRACES_TTL}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,7 @@
-- +goose Up
-- The engines macro adds Replicated inner table engines when the seed is
-- replicated. It must follow SETTINGS: placed before it, ClickHouse ignores
-- allow_experimental_time_series_table and the CREATE fails.
CREATE TABLE IF NOT EXISTS ${DATABASE}.metrics_ts
ENGINE = TimeSeries
SETTINGS allow_experimental_time_series_table = 1;
SETTINGS allow_experimental_time_series_table = 1${TIMESERIES_ENGINES};
Loading
Loading