Repository navigation
feat: replicated schema migration #3309
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
f07caf7
465a1ea
5375b6a
72a4163
cfbe598
ca2e4a3
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| --- | ||
| '@hyperdx/otel-collector': minor | ||
| --- | ||
|
|
||
| feat: opt-in Replicated database and table engines for the schema seed | ||
|
|
||
| When `HYPERDX_OTEL_EXPORTER_CLICKHOUSE_REPLICATED=true`, the migrate tool | ||
| creates the target database with the `Replicated` engine and seeds | ||
| `ReplicatedMergeTree` / `ReplicatedSummingMergeTree` tables instead of | ||
| `MergeTree` / `SummingMergeTree`. If the database already exists with another | ||
| engine, it logs a warning and seeds the plain engines as before. Off by default. | ||
| With `ENABLE_PROMQL=true` on ClickHouse 26.10 or later, the inner tables of the | ||
| PromQL `metrics_ts` table are Replicated too; older servers keep plain ones. | ||
|
|
||
| The database is created only on the node migrate connects to. On a cluster, run | ||
| migrate against every replica, or create the `Replicated` database on each node | ||
| first (for example with `ON CLUSTER`); otherwise inserts that reach a node | ||
| without the database fail. | ||
|
|
||
| The migrate tool also warns when it targets a database other than `default` that | ||
| has no ClickStack tables yet while `default` still holds them, since those are | ||
| not migrated. | ||
|
|
||
| With `HYPERDX_OTEL_EXPORTER_CLICKHOUSE_FALLBACK_DATABASE` set (off by default), | ||
| a target database that does not exist yet yields to the fallback database when | ||
| the fallback already has ClickStack tables. The collector then writes to the | ||
| fallback database too. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,3 +1,5 @@ | ||
| -- +goose Up | ||
| -- +goose NO TRANSACTION | ||
| CREATE DATABASE IF NOT EXISTS ${DATABASE}; | ||
| -- The engine macro is empty unless HYPERDX_OTEL_EXPORTER_CLICKHOUSE_REPLICATED | ||
| -- applies, then it is " ENGINE = Replicated(...)" (see processSchemaDir). | ||
| CREATE DATABASE IF NOT EXISTS ${DATABASE}${DATABASE_ENGINE}; | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 P2 — Replicated database is only created on the node migrate connects to A Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond |
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,7 @@ | ||
| -- +goose Up | ||
| -- The engines macro adds Replicated inner table engines when the seed is | ||
| -- replicated. It must follow SETTINGS: placed before it, ClickHouse ignores | ||
| -- allow_experimental_time_series_table and the CREATE fails. | ||
| CREATE TABLE IF NOT EXISTS ${DATABASE}.metrics_ts | ||
| ENGINE = TimeSeries | ||
| SETTINGS allow_experimental_time_series_table = 1; | ||
| SETTINGS allow_experimental_time_series_table = 1${TIMESERIES_ENGINES}; |
Uh oh!
There was an error while loading. Please reload this page.