Skip to content

[HDX-4664] Use a dedicated Replicated otel database for fresh installs - #286

Open
SpencerTorres wants to merge 2 commits into
mainfrom
spencer/replicated-engine
Open

SpencerTorres wants to merge 2 commits into
mainfrom
spencer/replicated-engine

Conversation

@SpencerTorres

@SpencerTorres SpencerTorres commented Oct 7, 2026 •

Copy link
Copy Markdown
Member

Summary

Fresh installs with the bundled ClickHouse now use a dedicated otel database, created by the collector's migration as Replicated with Replicated* tables. Existing installs keep their current database and engines.

Draft until upgraded HyperDX collector ships (ClickHouse/ClickStack#3309) (adds the replicated migration) and the Update App Version PR for 2.x.x is merged. With the current collector image, otel would be created with plain engines.

How it works

Install Database Engines HyperDX sources
Fresh install otel Replicated otel
helm upgrade Unchanged (default) Unchanged Unchanged
ArgoCD / helm template upgrade Collector stays on default via fallback Unchanged Existing unchanged
External ClickHouse default Unchanged default

Upgrades are detected with a lookup of the existing clickstack-config ConfigMap. Renders without cluster access can't see it, so the collector falls back to default when it finds existing ClickStack tables there.

Known limitations

  • After an ArgoCD / helm template upgrade, teams created later get default sources on otel. Pin HYPERDX_OTEL_EXPORTER_CLICKHOUSE_DATABASE: "default" to avoid this (documented in UPGRADE.md).
  • Existing installs stay non-replicated. Moving them is a manual step.

Test Plan

  • helm unittest (270 tests) and helm lint
  • kind with the ClickHouse operator, standalone and supervisor mode: upgrade from 3.4.0, ArgoCD-style render, fresh install, scale to 2 replicas

@changeset-bot

changeset-bot Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 23828af

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
helm-charts Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@SpencerTorres
SpencerTorres marked this pull request as ready for review November 18, 2026 19:25
@SpencerTorres
SpencerTorres requested a review from a team as a code owner November 18, 2026 19:25
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Deep Review

Scope: PR #286 spencer/replicated-engine vs base 0c5ba2f — 14 files, +592/−30. Helm templates, values.yaml, helm-unittest tests, docs, CI workflow, and integration/smoke shell scripts.
Intent: Make fresh installs with the bundled ClickHouse use a dedicated Replicated otel database while helm upgrade keeps existing installs on their current database/engines, detected via a lookup of the live clickstack-config ConfigMap.
Mode: report-only (read-only; no edits/artifacts).

✅ No critical issues found. The template logic is sound: the ClickHouse spec is tpl-rendered (clickhouse/cluster.yaml:19) so the new databaseGrants/database helper expansions resolve correctly, grants de-duplicate default.*/otel.*, bool-and-string clickhouse.enabled are both handled, and missing/empty ConfigMap keys fall back to default/false. New behavior is well covered (new 286-line clickhouse-database_test.yaml, 99.88% line / 91.6% branch coverage), the changeset and docs/UPGRADE.md "## Unreleased" note are present, and the regenerated README.md matches values.yaml.

🟡 P2 -- recommended

  • charts/clickstack/templates/_helpers.tpl:140 -- On a GitOps/helm template upgrade of an existing install, lookup returns nothing so clickstack.clickhouse.database resolves to otel, which flips the rendered PromQL remote-write handler to otel.metrics_ts (a table the collector never creates there) and silently stops PromQL ingestion unless the operator applies the documented HYPERDX_OTEL_EXPORTER_CLICKHOUSE_DATABASE: "default" pin.
    • Fix: Make the required pin impossible to miss for GitOps upgraders — surface it in NOTES.txt/install notes (not only docs/UPGRADE.md) so an unpinned render warns before PromQL breaks.
  • charts/clickstack/tests/clickhouse-database_test.yaml:356 -- The per-scenario cases assert the config values, default sources, and user grants but never assert spec.settings.extraConfig.prometheus.handlers.remote_write_handler.handler.database, so the exact divergence that breaks PromQL on an upgrade/lookup render is untested.
    • Fix: Add an assertion on the rendered PromQL remote-write handler database for the upgrade (existing-ConfigMap) and external-ClickHouse scenarios, mirroring the grant/source assertions already present.

Reviewers (6): correctness, testing, maintainability, project-standards, reliability, adversarial.

Testing gaps:

  • PromQL remote-write handler database is asserted only for the fresh-install (otel) default; the upgrade/lookup path that resolves it to default is not pinned by a test.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR.

@SpencerTorres
SpencerTorres force-pushed the spencer/replicated-engine branch from 388fd75 to 0db83f3 Compare November 20, 2026 22:40
@github-actions

Copy link
Copy Markdown
Contributor

Helm template coverage

Metric Value
Line coverage 99.88%
Branch coverage 91.60%
Line threshold 99% (met)
Branch threshold 90% (met)
Uncovered details

charts/clickstack/templates/NOTES.txt

Uncovered lines: none

Uncovered branches: 38:default:fallback

charts/clickstack/templates/_helpers.tpl

Uncovered lines: 17

Uncovered branches: 163:range:empty, 165:range:empty, 16:if:true, 203:and(2):false, 234:or(1):true, 287:default:fallback, 297:default:fallback, 298:or(1):true, 314:if:false, 444:range:empty, 465:range:empty, 46:range:empty, 492:default:fallback, 528:default:fallback

charts/clickstack/templates/additional-manifests.yaml

Uncovered lines: none

Uncovered branches: none

charts/clickstack/templates/clickhouse/cluster.yaml

Uncovered lines: none

Uncovered branches: none

charts/clickstack/templates/clickhouse/keeper.yaml

Uncovered lines: none

Uncovered branches: none

charts/clickstack/templates/hyperdx/configmap.yaml

Uncovered lines: none

Uncovered branches: none

charts/clickstack/templates/hyperdx/cronjob-check-alerts.yaml

Uncovered lines: none

Uncovered branches: none

charts/clickstack/templates/hyperdx/dashboard-configmap.yaml

Uncovered lines: none

Uncovered branches: none

charts/clickstack/templates/hyperdx/dashboard-provisioner-rbac.yaml

Uncovered lines: none

Uncovered branches: 41:range:empty, 61:range:empty

charts/clickstack/templates/hyperdx/deployment.yaml

Uncovered lines: none

Uncovered branches: 122:default:fallback, 132:default:fallback, 197:default:fallback, 202:with:empty, 21:with:empty, 27:default:fallback, 42:if:false, 43:with:empty

charts/clickstack/templates/hyperdx/hpa.yaml

Uncovered lines: none

Uncovered branches: 17:with:empty

charts/clickstack/templates/hyperdx/ingress.yaml

Uncovered lines: none

Uncovered branches: 16:with:empty, 36:if:false, 41:range:empty, 46:range:empty, 49:with:empty, 55:range:empty, 59:range:empty, 60:or(2):true, 60:or(3):true

charts/clickstack/templates/hyperdx/networkpolicy.yaml

Uncovered lines: none

Uncovered branches: 12:with:empty

charts/clickstack/templates/hyperdx/pdb.yaml

Uncovered lines: none

Uncovered branches: 11:with:empty

charts/clickstack/templates/hyperdx/secret.yaml

Uncovered lines: none

Uncovered branches: 2:or(2):true, 2:or(3):true

charts/clickstack/templates/hyperdx/service.yaml

Uncovered lines: none

Uncovered branches: 10:with:empty, 15:default:fallback

charts/clickstack/templates/hyperdx/serviceaccount.yaml

Uncovered lines: none

Uncovered branches: none

charts/clickstack/templates/mongodb/community.yaml

Uncovered lines: none

Uncovered branches: none

charts/clickstack/templates/mongodb/password-secret.yaml

Uncovered lines: none

Uncovered branches: none

charts/clickstack/templates/otel-collector/custom-config.yaml

Uncovered lines: none

Uncovered branches: none

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant