Skip to content

feat: replicated schema migration - #3309

Open
SpencerTorres wants to merge 6 commits into
mainfrom
spencer/replicated-engine
Open

SpencerTorres wants to merge 6 commits into
mainfrom
spencer/replicated-engine

Conversation

@SpencerTorres

Copy link
Copy Markdown
Member

Summary

Adds opt-in support for ClickHouse's Replicated engines to the schema migration, so the database and tables can replicate across nodes.

  • HYPERDX_OTEL_EXPORTER_CLICKHOUSE_REPLICATED=true creates the database with the Replicated engine and seeds Replicated* tables. An existing non-Replicated database keeps plain engines, with a warning.
  • HYPERDX_OTEL_EXPORTER_CLICKHOUSE_FALLBACK_DATABASE keeps writing to an existing database when the configured one doesn't exist yet. The collector is pointed at whichever database the migration used.

Both are off by default, and nothing changes when unset. Tested with unit tests and against ClickHouse 25.7 and 26.2 with Keeper, including multi-node replication.

References

  • Linear Issue: HDX-4664

@changeset-bot

changeset-bot Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: ca2e4a3

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages
Name Type
@hyperdx/otel-collector Minor
@hyperdx/api Minor
@hyperdx/app Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercel Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

2 Skipped Deployments
Project Deployment Actions Updated
hyperdx-oss Ignored Ignored Preview Nov 20, 2026 11:18pm UTC
hyperdx-storybook Skipped Skipped Nov 20, 2026 11:18pm UTC

Request Review

@github-actions github-actions Bot added the review/tier-4 Critical — deep review + domain expert sign-off label Oct 7, 2026
@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

🔴 Tier 4 — Critical

Touches authentication, tenancy data models, the public API or shipped database config — or substantially changes the query rendering engine, background tasks, the OTel pipeline, image build, or release CI.

Why this tier:

  • Background tasks or delivery pipeline substantially modified — 517 lines (bar: 30):
    • docker/otel-collector/entrypoint.sh
    • docker/otel-collector/schema/seed/00001_create_database.sql
    • docker/otel-collector/schema/seed/00002_otel_logs.sql
    • docker/otel-collector/schema/seed/00002_otel_logs_compat.sql
    • docker/otel-collector/schema/seed/00003_otel_metrics.sql
    • docker/otel-collector/schema/seed/00004_hyperdx_sessions.sql
    • docker/otel-collector/schema/seed/00004_hyperdx_sessions_compat.sql
    • docker/otel-collector/schema/seed/00005_otel_traces.sql
    • docker/otel-collector/schema/seed/00005_otel_traces_compat.sql
    • docker/otel-collector/schema/seed/00006_otel_logs_rollups.sql
    • docker/otel-collector/schema/seed/00007_otel_traces_rollups.sql
    • docker/otel-collector/schema/seed/00008_otel_metrics_timeseries.sql
    • packages/otel-collector/cmd/migrate/main.go
    • packages/otel-collector/cmd/migrate/main_test.go

Review process: Deep review from a domain expert. Synchronous walkthrough may be required.
SLA: Schedule synchronous review within 2 business days.

Stats
  • Production files changed: 15
  • Production lines changed: 526
  • Critical-path lines changed: 517
  • Branch: spencer/replicated-engine
  • Author: SpencerTorres

To override this classification, remove the review/tier-4 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Duplicate code analysis

No new duplicate code introduced by this PR.


jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 108 in total (1.13%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.


// A fallback database keeps the plain engines it was seeded with.
replicated := false
if database == cfg.Database {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — Replicated mode is dropped without a warning when the fallback database is used

With HYPERDX_OTEL_EXPORTER_CLICKHOUSE_REPLICATED=true and a fallback that wins (the target is missing and the fallback has tables), planReplicated never runs. The fallback is seeded with plain engines and nothing says Replicated was not applied. Log a warning in that branch, for example: if cfg.Replicated && database != cfg.Database { log.Printf("WARNING: Replicated engines not applied; staying on fallback %s", database) }.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

PR Review

If you are a coding agent acting for the author, read this first.
Nothing below blocks merge automatically; a maintainer decides. What they will expect:

  • 🔴 P0/P1 in code this PR adds or changes: fix before asking for review.
  • 🟡 P2 in code this PR changes: your call. Fix if small, otherwise reply.
  • 🔵 P3 is collapsed at the bottom: nobody needs to act on it before merge.
  • Anything about surrounding code, or asking you to widen the change (hoist a helper,
    dedupe with another file, fix other call sites): never fix here, whatever the
    priority. Reply with one sentence; if it is P0, say so plainly so a human sees it.

One commit per review round. After two rounds, stop and ask a maintainer to review
scope rather than addressing more automated findings. Full rule: AGENTS.md.

3 finding(s): 🔴 0 P0/P1 · 🟡 0 P2 · 🔵 3 P3

No inline comments. 3 P3 folded at the bottom.

🔵 3 P3 — no action needed before merge
  • 🔵 P3 docker/otel-collector/entrypoint.sh:86 — No test covers the fallback handoff from migrate to the collector, the only path that changes where data is written → Add a scenario to smoke-tests/otel-collector/docker-compose.yaml and a bats test. It should set HYPERDX_OTEL_EXPORTER_CLICKHOUSE_DATABASE to a missing database and HYPERDX_OTEL_EXPORTER_CLICKHOUSE_FALLBACK_DATABASE to a seeded one, then check that rows land in the fallback. The unit tests only cover the pure resolveDatabase and planReplicated functions. Nothing tests the order of checks in main() (dbExists being reused or re-queried for the fallback), the resolved-database file write, or the entrypoint's export.
  • 🔵 P3 packages/otel-collector/cmd/migrate/main.go:767 — Third copy of the same version-comparison logic → supportsReplicatedTimeSeries repeats the body of supportsFullTextSearch and supportsTextIndexTableSettings (major > X || (major == X && minor >= Y)) with only the constants changed. Factor out one atLeastVersion(major, minor, wantMajor, wantMinor int) bool and have all three gates call it.
  • 🔵 P3 packages/otel-collector/cmd/migrate/main.go:280 — The default resolved-database path is defined in two places → /tmp/hyperdx-otel-exporter-database is the default both here and in docker/otel-collector/entrypoint.sh:84, and the entrypoint always passes the variable explicitly, so the Go default never applies there. Keep one source of truth: either let the entrypoint own the path and make migrate skip the write when the variable is empty, or have only the Go side define it.

Priority comes from a separate grading pass over the finished review. P3 is folded here instead of posted inline. No finding blocks merge automatically; a maintainer decides.

@greptile-apps

greptile-apps Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 4/5

[Critical risk] Database schema migration with replication and fallback logic.

Remove the prohibited test section banners before merging; no behavioral defect was established.

Fix All in Claude CodeFindings

  1. P2 Test banners break the guide ▶

Summary

Adds opt-in replicated database and table engines, plus a fallback database that is passed to the collector after seeding.

  • Existing non-replicated target databases keep plain table engines.
  • Seed statements and TTL updates use the selected database.
  • Remove the new test section banners to satisfy the repository’s comment rule.
Diagram
%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Read target and fallback settings] --> B{Target missing and fallback has tables?}
  B -->|Yes| C[Select fallback with plain engines]
  B -->|No| D[Select target]
  D --> E{Replication enabled and target missing or Replicated?}
  E -->|Yes| F[Use replicated engines]
  E -->|No| G[Use plain engines]
  C --> H[Seed selected database]
  F --> H
  G --> H
  H --> I{Fallback setting enabled?}
  I -->|Yes| J[Write selected database to handoff file]
  J --> K[Entrypoint exports database for collector]
  I -->|No| L[Keep configured collector database]
Loading

Reviews (1) · Last reviewed commit: "feat: replicated schema migration" · Reviewed by Greptile

Comment on lines +828 to +830
// ---------------------------------------------------------------------------
// replicated engines
// ---------------------------------------------------------------------------

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Test banners break the guide

The new section dividers violate the AGENTS.md directive: “No banner comments, no ASCII section dividers.” This rule also applies to tests. Remove this banner and the matching fallback banner at lines 974–976 before merging to satisfy the repository requirement.

Context Used: AGENTS.md (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Fix in Claude Code Fix in Conductor Fix in Cursor Fix in Codex

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Deep Review

Scope: PR #3309 — opt-in ClickHouse Replicated database/table engines plus a fallback-database handoff for the otel-collector schema migration. Base d945d7ea. 16 files, ~524 added lines (Go migrate tool, SQL seed macros, Docker entrypoint, changeset).

Intent: HYPERDX_OTEL_EXPORTER_CLICKHOUSE_REPLICATED=true seeds Replicated* engines and a Replicated database; HYPERDX_OTEL_EXPORTER_CLICKHOUSE_FALLBACK_DATABASE keeps an upgrading install writing to its existing database until the new target is seeded, with migrate writing the resolved database name to a file the entrypoint exports to the collector. Both off by default.

This is a carefully written, well-tested change. The resolved-database handoff is robust: set -e in entrypoint.sh plus migrate's os.Exit(1) on seed failure means the collector never starts pointed at an unseeded database, and the fatal-vs-warning split on the resolved-file write (main.go:1108) correctly hard-fails only when the seeded database differs from the configured one. No P0/P1 issues were established.

✅ No critical issues found.

🟡 P2 -- recommended

  • packages/otel-collector/cmd/migrate/main.go:647 -- hasClickStackTables detects an existing install by checking only otel_logs and otel_traces, so a metrics-only or sessions-only deployment reads as empty; with the fallback enabled this routes the collector to the new empty target while the old database still holds the live tables.
    • Fix: include the metrics/sessions tables (e.g. otel_metrics_sum, hyperdx_sessions, metrics_ts) in the existence probe, or document that fallback detection assumes a logs/traces install.
  • packages/otel-collector/cmd/migrate/main.go:766 -- on ClickHouse < 26.10 the PromQL metrics_ts inner tables are seeded without explicit Replicated* engines inside a Replicated database, and the warning asserts they "are not replicated"; confirm whether the Replicated database auto-converts the inner tables, since if it does not, PromQL metrics diverge across replicas and if it does, the warning is misleading.
    • Fix: verify the inner-table engine on a < 26.10 replica and either correct the warning text or document the divergence as an unsupported configuration.
🔵 P3 nitpicks (3)
  • packages/otel-collector/cmd/migrate/main_test.go:841 -- the new // ---- ASCII section-divider banners labeling the "replicated engines" and "fallback database" blocks conflict with the AGENTS.md rule against banner comments / ASCII section dividers.
    • Fix: drop the divider lines and keep the plain function-level comments.
  • packages/otel-collector/cmd/migrate/main.go:569 -- the raw database name is interpolated into the Keeper path Replicated('/clickhouse/databases/%s', ...), so a database name with path-significant or special characters produces a malformed znode path.
    • Fix: restrict/validate the database identifier or document that only plain identifier names are supported with Replicated.
  • packages/otel-collector/cmd/migrate/main.go:997 -- the main() block that chains legacyDatabase → hasClickStackTables → getDatabaseEngine (called twice) → resolveDatabase → planReplicated with a reused warning variable is dense and hard to follow.
    • Fix: extract the database-resolution sequence into a helper returning (database, replicated, warnings) to flatten the branching.

Reviewers (9 dispatched): correctness, data-migrations, reliability, adversarial, testing, maintainability, project-standards, security, previous-comments.

Note on coverage: the parallel reviewer sub-agents had not returned at the point synthesis was required, so these findings reflect the orchestrator's own verified read of the diff rather than a merge of the sub-agent outputs. Severities are graded conservatively; re-running the review will add corroboration and may surface additional lower-confidence items.

Testing gaps: main() wiring, the entrypoint resolved-database handoff, getDatabaseEngine, and hasClickStackTables have no unit coverage (they need a live ClickHouse); the pure decision functions (planReplicated, resolveDatabase) are well covered by table-driven tests.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

E2E Test Results

✅ All tests passed • 474 passed • 1 skipped • 1251s

Status Count
✅ Passed 474
❌ Failed 0
⚠️ Flaky 8
⏭️ Skipped 1

Tests ran across 4 shards in parallel.

View full report →

@wrn14897 wrn14897 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you elaborate on why we need HYPERDX_OTEL_EXPORTER_CLICKHOUSE_FALLBACK_DATABASE?

Seed hyperdx_sessions_compat (new on main) with ${ENGINE_PREFIX}MergeTree,
and set the text index block size on the resolved database rather than the
configured one.
@SpencerTorres
SpencerTorres requested a review from a team as a code owner November 20, 2026 20:55
-- +goose Up
-- +goose NO TRANSACTION
CREATE DATABASE IF NOT EXISTS ${DATABASE};
CREATE DATABASE IF NOT EXISTS ${DATABASE}${DATABASE_ENGINE};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 P2 — Replicated database is only created on the node migrate connects to

A Replicated database has to be created on every replica, either with ON CLUSTER or by running CREATE DATABASE on each node. Migrate runs a plain CREATE DATABASE IF NOT EXISTS over the single CLICKHOUSE_ENDPOINT. If collectors reach ClickHouse through a load balancer or one shared host, only that one node joins the database. Inserts routed to any other node then fail because the database does not exist there. Either document that the database must already exist on every node or be created by connecting to each node, or accept an optional cluster name and emit ON CLUSTER '<cluster>'. Add this to the changeset and to the docker-compose.dev.yml comment as well.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

tempDir, err := processSchemaDir(cfg.SchemaDir, cfg.Database, ttlExprs)
// The legacy tables check is best effort unless the fallback depends on it.
var legacyHasTables bool
if legacy := legacyDatabase(cfg.FallbackDatabase); legacy != cfg.Database {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 P2 — New legacy-tables warning fires on every boot, including with both flags unset

hasClickStackTables(default) now runs whenever the target is not default. When neither flag is set, dbExists is never queried, so resolveDatabase can't tell a deliberate past move from a fresh one. Any deployment with tables in both default and its target logs "not migrated ... HyperDX sources created earlier still point at default" on every start, with no way to silence it. This includes the smoke-test databases on a shared server. The same thing happens forever once a fallback user moves to the target. This contradicts "nothing changes when unset". Fix options: gate the warning on the target not existing (always call getDatabaseEngine), or on the target not yet having ClickStack tables, or show it only on first seed.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

log.Printf("WARNING: %s", warning)
}

// A fallback database keeps the plain engines it was seeded with.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 P2 — REPLICATED=true is silently ignored when the fallback database is chosen

When resolveDatabase returns the fallback, planReplicated is skipped. An operator who set HYPERDX_OTEL_EXPORTER_CLICKHOUSE_REPLICATED=true gets only the fallback warning, not the Replicated-specific one. They also get plain engines, even if the fallback database is itself Replicated. Fix: check the fallback's engine with getDatabaseEngine and pass it to planReplicated, or log a warning when cfg.Replicated is set and the fallback wins.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

@SpencerTorres

Copy link
Copy Markdown
Member Author

Could you elaborate on why we need HYPERDX_OTEL_EXPORTER_CLICKHOUSE_FALLBACK_DATABASE?

Replicated can only be set when a database is created, and our new helm changes use otel as the default install database. Without a fallback, existing installs that upgrade would write into a new, empty database while their sources still read default. With the fallback set, they stay on default until the new database exists. Mainly for our helm PR.

What's the possible value of DATABASE_ENGINE here?

Usually empty, but when replicated is enabled we set it as replicated engine

do we want to check only logs and traces here?

For the purpose of checking for an existing install yes

}

// A fallback already exists, so only its own engine can make it Replicated.
if cfg.Replicated && database != cfg.Database {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 P2 — With both REPLICATED and FALLBACK_DATABASE set, the Replicated warning gives advice the operator has already followed

In this case resolveDatabase has moved the seed to the fallback database, so planReplicated tells the operator to "set HYPERDX_OTEL_EXPORTER_CLICKHOUSE_DATABASE to a new database name". They already did that. The fallback is what blocks replication. When database != cfg.Database, use a warning that tells them to clear HYPERDX_OTEL_EXPORTER_CLICKHOUSE_FALLBACK_DATABASE (or to replicate the fallback database), for example by passing the fallback context into planReplicated.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

@vercel
vercel Bot temporarily deployed to Preview – hyperdx-storybook November 20, 2026 23:18 Inactive

This branch was successfully deployed

1 active (outdated) and 1 inactive deployments
Preview – hyperdx-storybook — ca2e4a31 Deployed Nov 20, 2026 by vercel[bot]
Preview – hyperdx-oss — 465a1ea6 Deployed Nov 20, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review/tier-4 Critical — deep review + domain expert sign-off

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants