Skip to content

feat: let the dashboard watcher skip the release namespace - #315

Merged
jordan-simonovski merged 5 commits into
ClickHouse:mainfrom
braghettos:fix/dashboards-include-release-namespace
Nov 19, 2026
Merged

jordan-simonovski merged 5 commits into
ClickHouse:mainfrom
braghettos:fix/dashboards-include-release-namespace

Conversation

@braghettos

Copy link
Copy Markdown
Contributor

Problem

The dashboard provisioner always watches the release namespace: clickstack.hyperdx.effectiveNamespaces is concat (list .Release.Namespace) $namespaces, and there is no value to turn that off. With hyperdx.dashboards.namespaces: [dashboards], the chart renders a ClusterRole (get/list/watch on ConfigMaps), a RoleBinding in both the release namespace and dashboards, and the watcher's NAMESPACE is <release-ns>,dashboards.

So you can't confine HyperDX's ConfigMap read access to a dedicated namespace. In our case the install namespace is shared with other components and holds hundreds of unrelated ConfigMaps. The HyperDX pod's ServiceAccount, which the user-facing app container also uses, can read all of them. Working around it with rbac.create: false and narrower RBAC doesn't help: the watcher gets a 403 on the release namespace during its startup sync, exits 1 and crash-loops, and the HyperDX pod goes NotReady.

Change

  • New value hyperdx.dashboards.includeReleaseNamespace, default true. effectiveNamespaces honours it, so false plus namespaces: [x] watches only x. A missing key, e.g. --reuse-values from an older chart, counts as true, like dashboardsRbacCreate.
  • RBAC when the effective set doesn't include the release namespace: the chart renders a namespaced Role and RoleBinding in each target namespace, with no ClusterRole and nothing in the release namespace. Outside the release namespace these objects get the same namespace-qualified name as the cluster-scoped objects (<fullname>-<release-ns>-dashboard-provisioner), so two same-named releases can't collide in a shared target namespace. dashboardsCrossNamespace now selects the ClusterRole path only when the set has more than one namespace and includes the release namespace. The existing Role branch now loops over the effective set, so the release-namespace-only case renders exactly as before.
  • Guards in validateDashboards:
    • includeReleaseNamespace: false with no namespaces fails the render: the watcher would have nothing to watch.
    • Inline configMaps also fail the render when the release namespace isn't watched. They are created in the release namespace, so they would never be discovered.
  • values.yaml comments and the regenerated README; minor changeset. No docs/UPGRADE.md entry, because nothing changes on upgrade unless you opt in.

Backward compatibility

With the default true, output is unchanged. I compared helm template output before and after this change for: dashboards disabled; enabled with defaults; [team-a, team-b]; [<release-ns>]; [<release-ns>, team-a]; [ALL]; rbac.create: false; and a single extra namespace with a custom ServiceAccount. All of them are byte-identical. [ALL] ignores the new value.

Tests

  • make test: 310/310 pass, including 11 new cases in tests/dashboard-provisioner_test.yaml. They cover:

    • explicit true, and a missing key, both keeping today's scope
    • false + one namespace: Role and RoleBinding only in that namespace, the watcher's NAMESPACE set to it, no ClusterRole, and nothing in the release namespace
    • false + two namespaces: a Role and RoleBinding per namespace
    • false with the release namespace listed: the ClusterRole path as today
    • false + [ALL]
    • false + rbac.create: false
    • the empty-namespaces guard, for both an empty list and null
    • the inline configMaps guard, failing when the release namespace isn't watched and allowed when it is listed
    • guards skipped when dashboards are disabled

    Run with helm-unittest v1.0.3 as pinned in scripts/tool-versions.env.

  • With the old RBAC template, the new RBAC cases fail.

  • make coverage: 99.85% line, 91.27% branch.

  • make docs-check: clean.

  • Manual kind check, arm64. The base is integration-tests/api-only (standalone Mongo, no operators). I installed into namespace clickstack with includeReleaseNamespace: false, namespaces: [dashboards], with labelled ConfigMaps in both namespaces, created before install and after startup:

    • Role and RoleBinding only in dashboards; no ClusterRole or ClusterRoleBinding; nothing in clickstack
    • the watcher's NAMESPACE is dashboards
    • kubectl auth can-i list configmaps as the HyperDX ServiceAccount returns no in clickstack, yes in dashboards, and no with -A
    • ConfigMaps in dashboards are copied into /dashboards by the initial list sync and by the watch; the ones in clickstack never are
    • no 403s in the watcher logs; the pod is 2/2 Ready with 0 watcher restarts (with the limit raised, see below)

Separate observation (not changed here)

In this kind run, the watcher's default 64Mi memory limit got it OOMKilled about once a minute: 3 restarts in 4 minutes, the pod not Ready, and the initial sync never finished. Raising the limit to 128Mi fixed that: the working set stayed at about 72Mi with a 95Mi peak, and there were 0 restarts. That was with 4 small ConfigMaps in one namespace, so it looks like baseline usage of kiwigrid/k8s-sidecar:2.10.1 rather than load. I kept the default out of this PR so the change stays focused; happy to open a follow-up if you'd like the limit raised.

🤖 Generated with Claude Code

https://claude.ai/code/session_01TLFnUd6g7xhDXDvaho4rDW

Add hyperdx.dashboards.includeReleaseNamespace (default true, unchanged
output). With false, the watcher's NAMESPACE and the RBAC cover only
hyperdx.dashboards.namespaces: a namespaced Role and RoleBinding in each
listed namespace (namespace-qualified names), no ClusterRole, and nothing
in the release namespace. Without it, the release namespace is always
watched and readable, and granting narrower RBAC out-of-band makes the
watcher 403 on the release namespace and crash-loop.

The render fails when includeReleaseNamespace is false with no
namespaces, or with inline configMaps while the release namespace is not
watched (they would never be discovered).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TLFnUd6g7xhDXDvaho4rDW
@braghettos
braghettos requested a review from a team as a code owner November 16, 2026 16:09
@changeset-bot

changeset-bot Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 12561d2

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
helm-charts Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@jordan-simonovski jordan-simonovski left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey, thanks for your contribution! 🙌

Change looks good apart from one thing. Flipping includeReleaseNamespace on an existing install keeps the RoleBinding name in the target namespace but changes its roleRef from the ClusterRole to a Role. Since roleRef can't be changed, I think the upgrade would fail. I've left details and a possible fix inline.

Happy to approve once that's sorted. And yes please to the follow up PR raising the watcher's memory limit, thanks for digging into that.

{{- /* Default: discovery scoped to the release namespace only — a namespaced Role, no cluster-scoped objects. */}}
{{- /* Default: discovery scoped to the release namespace only — a namespaced Role, no cluster-scoped objects. With includeReleaseNamespace: false, the same namespaced Role + RoleBinding pair is rendered in each target namespace instead, and nothing in the release namespace. Outside the release namespace the names are namespace-qualified like the cluster-scoped objects above, so two same-named releases can't collide in a shared target namespace. */}}
{{- range $i, $ns := (include "clickstack.hyperdx.effectiveNamespaces" . | splitList ",") }}
{{- $name := ternary (printf "%s-dashboard-provisioner" $fullname) $clusterName (eq $ns $.Release.Namespace) }}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Flipping includeReleaseNamespace on an existing install changes a RoleBinding's roleRef, which the API server rejects.

Setting includeReleaseNamespace: false renders a RoleBinding with the same name in the same namespace that points at a Role. roleRef is immutable, so helm upgrade would fail with "cannot change roleRef" for the people this flag is for, and again if they flip it back.

A different name for the Role path's objects outside the release namespace should avoid that:

{{- $name := ternary (printf "%s-dashboard-provisioner" $fullname) (printf "%s-scoped" $clusterName) (eq $ns $.Release.Namespace) }}

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Deep Review

A multi-agent review was dispatched (correctness, security, adversarial, testing, maintainability, project-standards, reliability, previous-comments) against the diff at base 04000077a72ada33adb591d1d52405143c0eb9ea. The synthesized report below reflects direct static verification of the four changed templates/values plus the prior-review thread; it focuses on the RBAC rendering paths, the new validation guards, and the immutable-roleRef flip concern raised in prior review.

✅ No critical issues found.

The change is well-contained and the core correctness claims hold up under inspection:

  • The includeReleaseNamespace flag threads consistently through effectiveNamespaces, which is the single source for both the watcher's NAMESPACE env (charts/clickstack/templates/hyperdx/deployment.yaml:223) and the per-namespace RBAC, so granted scope and watched scope cannot diverge.
  • The prior CHANGES_REQUESTED feedback (flipping the flag would mutate a RoleBinding's immutable roleRef) is addressed: outside the release namespace the Role-path objects are named $clusterName-scoped (charts/clickstack/templates/hyperdx/dashboard-provisioner-rbac.yaml:61), distinct from the cross-namespace path's $clusterName RoleBinding in the same target namespace, so no same-named object changes its roleRef kind on a flip. A unit test pins both names in the same namespace.
  • Both new validateDashboards guards (empty namespaces when the release namespace is excluded; inline configMaps when the release namespace is unwatched) fail the render at template time rather than letting a misconfigured watcher crash-loop, and both have passing/failing/counter-case tests.

🟡 P2 — recommended

  • charts/clickstack/templates/hyperdx/dashboard-provisioner-rbac.yaml:61 — The -scoped suffix is appended after $clusterName is already trunc 253, so printf "%s-scoped" $clusterName can in principle exceed the 253-char RBAC name limit; it is safe only because fullname and namespace are each ≤63, a bound that is relied upon in a comment rather than enforced in code.
    • Fix: Re-apply trunc 253 | trimSuffix "-" to the final $name after the -scoped append so the invariant is enforced rather than assumed.
🔵 P3 nitpicks (1)
  • charts/clickstack/templates/hyperdx/dashboard-provisioner-rbac.yaml:59 — The else branch now serves two distinct cases (default release-namespace-only Role, and scoped exclude-release Roles) behind one range, and its intent lives in a single long comment; a future maintainer could misread the ternary ... (eq $ns $.Release.Namespace) name selection.
    • Fix: Consider splitting the comment into a short per-case note at the ternary line clarifying which name applies to which namespace.

Reviewers (8): correctness, security, adversarial, testing, maintainability, project-standards, reliability, previous-comments.

Testing gaps: None blocking — the 11 new helm-unittest cases cover the new RBAC paths, both guards (including the null namespaces case), the [ALL] and rbac.create: false combinations, the roleRef-flip name distinction, and the dashboards-disabled guard skip.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR.

braghettos and others added 2 commits November 18, 2026 12:47
charts/clickstack/README.md: the generated hyperdx.config and
hyperdx.deployment rows are main's; the hyperdx.dashboards row is main's
plus this branch's includeReleaseNamespace default, its only difference.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016RAxvjuD1TdxneAHTCFpW8
With hyperdx.dashboards.namespaces: [ns], flipping includeReleaseNamespace on
an existing install switched the RoleBinding in ns between the cross-namespace
path (roleRef ClusterRole) and the Role path (roleRef Role) under the same
name. roleRef is immutable, so helm upgrade failed with "roleRef: Invalid
value", and the failed upgrade left a stray Role behind.

Outside the release namespace the Role path's Role and RoleBinding are now
named <cluster name>-scoped, so neither direction changes an existing
roleRef. The suffix always survives: the cluster name is at most 149
characters.

Checked with real helm upgrades against a kind API server, only this chart's
RBAC installed: true -> false -> true with one target namespace failed before
the change and passes after, each step leaving exactly that state's objects;
the default (release namespace only) <-> cross-namespace flip passes both
ways. The unit tests pin the new names and the cross-namespace RoleBinding's
name in the same target namespace.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016RAxvjuD1TdxneAHTCFpW8
@braghettos

Copy link
Copy Markdown
Contributor Author

Thanks, good catch: reproduced and fixed in a38e831, with main merged in.

I reproduced it with real helm upgrades against a kind API server, installing only this chart's RBAC: with namespaces: [ns], going includeReleaseNamespace: true -> false failed with RoleBinding ... is invalid: roleRef: Invalid value, and the failed upgrade also left a stray Role in ns.

The fix is your suggestion: outside the release namespace the Role path's Role and RoleBinding are named <cluster name>-scoped. With it, true -> false -> true upgrades cleanly and each step leaves exactly that state's objects, and default (release namespace only) <-> cross-namespace also works both ways. The suffix can't be truncated away (the cluster name is at most 149 characters). The unit tests now pin the new names, plus the cross-namespace RoleBinding's name in the same target namespace, and the changeset says the flag can be flipped on an existing install.

I'll open the watcher memory-limit PR separately.

teeohhem pushed a commit that referenced this pull request Nov 18, 2026
…request) (#327)

At the 64Mi limit the kiwigrid/k8s-sidecar dashboard watcher was
OOMKilled
about once a minute on a kind cluster, with only 4 small dashboard
ConfigMaps
in one namespace: 3 restarts in 4 minutes, the pod never Ready, the
initial
sync never finished. With the limit at 128Mi its working set stayed at
about
72Mi with a 95Mi peak, and it did not restart. So this is the sidecar's
baseline, not load.

The default limit becomes 128Mi (headroom above the measured peak) and
the
request 64Mi (about the measured working set, so the scheduler reserves
what
it uses). CPU is unchanged. The unit test for the default resources pins
the
new values; values.yaml says where the numbers come from.

Follow-up to #315, as discussed in its review.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_016RAxvjuD1TdxneAHTCFpW8

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	charts/clickstack/README.md
@jordan-simonovski
jordan-simonovski enabled auto-merge (squash) November 19, 2026 03:41
@jordan-simonovski
jordan-simonovski merged commit 3dd7921 into ClickHouse:main Nov 19, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants