Repository navigation
feat: let the dashboard watcher skip the release namespace - #315
jordan-simonovski merged 5 commits into
Conversation
Add hyperdx.dashboards.includeReleaseNamespace (default true, unchanged output). With false, the watcher's NAMESPACE and the RBAC cover only hyperdx.dashboards.namespaces: a namespaced Role and RoleBinding in each listed namespace (namespace-qualified names), no ClusterRole, and nothing in the release namespace. Without it, the release namespace is always watched and readable, and granting narrower RBAC out-of-band makes the watcher 403 on the release namespace and crash-loop. The render fails when includeReleaseNamespace is false with no namespaces, or with inline configMaps while the release namespace is not watched (they would never be discovered). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TLFnUd6g7xhDXDvaho4rDW
🦋 Changeset detectedLatest commit: 12561d2 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
jordan-simonovski
left a comment
There was a problem hiding this comment.
Hey, thanks for your contribution! 🙌
Change looks good apart from one thing. Flipping includeReleaseNamespace on an existing install keeps the RoleBinding name in the target namespace but changes its roleRef from the ClusterRole to a Role. Since roleRef can't be changed, I think the upgrade would fail. I've left details and a possible fix inline.
Happy to approve once that's sorted. And yes please to the follow up PR raising the watcher's memory limit, thanks for digging into that.
| {{- /* Default: discovery scoped to the release namespace only — a namespaced Role, no cluster-scoped objects. */}} | ||
| {{- /* Default: discovery scoped to the release namespace only — a namespaced Role, no cluster-scoped objects. With includeReleaseNamespace: false, the same namespaced Role + RoleBinding pair is rendered in each target namespace instead, and nothing in the release namespace. Outside the release namespace the names are namespace-qualified like the cluster-scoped objects above, so two same-named releases can't collide in a shared target namespace. */}} | ||
| {{- range $i, $ns := (include "clickstack.hyperdx.effectiveNamespaces" . | splitList ",") }} | ||
| {{- $name := ternary (printf "%s-dashboard-provisioner" $fullname) $clusterName (eq $ns $.Release.Namespace) }} |
There was a problem hiding this comment.
Flipping includeReleaseNamespace on an existing install changes a RoleBinding's roleRef, which the API server rejects.
Setting includeReleaseNamespace: false renders a RoleBinding with the same name in the same namespace that points at a Role. roleRef is immutable, so helm upgrade would fail with "cannot change roleRef" for the people this flag is for, and again if they flip it back.
A different name for the Role path's objects outside the release namespace should avoid that:
{{- $name := ternary (printf "%s-dashboard-provisioner" $fullname) (printf "%s-scoped" $clusterName) (eq $ns $.Release.Namespace) }}
Deep ReviewA multi-agent review was dispatched (correctness, security, adversarial, testing, maintainability, project-standards, reliability, previous-comments) against the diff at base ✅ No critical issues found. The change is well-contained and the core correctness claims hold up under inspection:
🟡 P2 — recommended
🔵 P3 nitpicks (1)
Reviewers (8): correctness, security, adversarial, testing, maintainability, project-standards, reliability, previous-comments. Testing gaps: None blocking — the 11 new helm-unittest cases cover the new RBAC paths, both guards (including the No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. |
charts/clickstack/README.md: the generated hyperdx.config and hyperdx.deployment rows are main's; the hyperdx.dashboards row is main's plus this branch's includeReleaseNamespace default, its only difference. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016RAxvjuD1TdxneAHTCFpW8
With hyperdx.dashboards.namespaces: [ns], flipping includeReleaseNamespace on an existing install switched the RoleBinding in ns between the cross-namespace path (roleRef ClusterRole) and the Role path (roleRef Role) under the same name. roleRef is immutable, so helm upgrade failed with "roleRef: Invalid value", and the failed upgrade left a stray Role behind. Outside the release namespace the Role path's Role and RoleBinding are now named <cluster name>-scoped, so neither direction changes an existing roleRef. The suffix always survives: the cluster name is at most 149 characters. Checked with real helm upgrades against a kind API server, only this chart's RBAC installed: true -> false -> true with one target namespace failed before the change and passes after, each step leaving exactly that state's objects; the default (release namespace only) <-> cross-namespace flip passes both ways. The unit tests pin the new names and the cross-namespace RoleBinding's name in the same target namespace. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016RAxvjuD1TdxneAHTCFpW8
|
Thanks, good catch: reproduced and fixed in a38e831, with main merged in. I reproduced it with real The fix is your suggestion: outside the release namespace the Role path's Role and RoleBinding are named I'll open the watcher memory-limit PR separately. |
…request) (#327) At the 64Mi limit the kiwigrid/k8s-sidecar dashboard watcher was OOMKilled about once a minute on a kind cluster, with only 4 small dashboard ConfigMaps in one namespace: 3 restarts in 4 minutes, the pod never Ready, the initial sync never finished. With the limit at 128Mi its working set stayed at about 72Mi with a 95Mi peak, and it did not restart. So this is the sidecar's baseline, not load. The default limit becomes 128Mi (headroom above the measured peak) and the request 64Mi (about the measured working set, so the scheduler reserves what it uses). CPU is unchanged. The unit test for the default resources pins the new values; values.yaml says where the numbers come from. Follow-up to #315, as discussed in its review. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_016RAxvjuD1TdxneAHTCFpW8 Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # charts/clickstack/README.md
Problem
The dashboard provisioner always watches the release namespace:
clickstack.hyperdx.effectiveNamespacesisconcat (list .Release.Namespace) $namespaces, and there is no value to turn that off. Withhyperdx.dashboards.namespaces: [dashboards], the chart renders a ClusterRole (get/list/watch on ConfigMaps), a RoleBinding in both the release namespace anddashboards, and the watcher'sNAMESPACEis<release-ns>,dashboards.So you can't confine HyperDX's ConfigMap read access to a dedicated namespace. In our case the install namespace is shared with other components and holds hundreds of unrelated ConfigMaps. The HyperDX pod's ServiceAccount, which the user-facing app container also uses, can read all of them. Working around it with
rbac.create: falseand narrower RBAC doesn't help: the watcher gets a 403 on the release namespace during its startup sync, exits 1 and crash-loops, and the HyperDX pod goes NotReady.Change
hyperdx.dashboards.includeReleaseNamespace, defaulttrue.effectiveNamespaceshonours it, sofalseplusnamespaces: [x]watches onlyx. A missing key, e.g.--reuse-valuesfrom an older chart, counts astrue, likedashboardsRbacCreate.<fullname>-<release-ns>-dashboard-provisioner), so two same-named releases can't collide in a shared target namespace.dashboardsCrossNamespacenow selects the ClusterRole path only when the set has more than one namespace and includes the release namespace. The existing Role branch now loops over the effective set, so the release-namespace-only case renders exactly as before.validateDashboards:includeReleaseNamespace: falsewith nonamespacesfails the render: the watcher would have nothing to watch.configMapsalso fail the render when the release namespace isn't watched. They are created in the release namespace, so they would never be discovered.values.yamlcomments and the regenerated README;minorchangeset. Nodocs/UPGRADE.mdentry, because nothing changes on upgrade unless you opt in.Backward compatibility
With the default
true, output is unchanged. I comparedhelm templateoutput before and after this change for: dashboards disabled; enabled with defaults;[team-a, team-b];[<release-ns>];[<release-ns>, team-a];[ALL];rbac.create: false; and a single extra namespace with a custom ServiceAccount. All of them are byte-identical.[ALL]ignores the new value.Tests
make test: 310/310 pass, including 11 new cases intests/dashboard-provisioner_test.yaml. They cover:true, and a missing key, both keeping today's scopefalse+ one namespace: Role and RoleBinding only in that namespace, the watcher'sNAMESPACEset to it, no ClusterRole, and nothing in the release namespacefalse+ two namespaces: a Role and RoleBinding per namespacefalsewith the release namespace listed: the ClusterRole path as todayfalse+[ALL]false+rbac.create: falsenullRun with helm-unittest v1.0.3 as pinned in
scripts/tool-versions.env.With the old RBAC template, the new RBAC cases fail.
make coverage: 99.85% line, 91.27% branch.make docs-check: clean.Manual kind check, arm64. The base is
integration-tests/api-only(standalone Mongo, no operators). I installed into namespaceclickstackwithincludeReleaseNamespace: false, namespaces: [dashboards], with labelled ConfigMaps in both namespaces, created before install and after startup:dashboards; no ClusterRole or ClusterRoleBinding; nothing inclickstackNAMESPACEisdashboardskubectl auth can-i list configmapsas the HyperDX ServiceAccount returnsnoinclickstack,yesindashboards, andnowith-Adashboardsare copied into/dashboardsby the initial list sync and by the watch; the ones inclickstacknever areSeparate observation (not changed here)
In this kind run, the watcher's default 64Mi memory limit got it OOMKilled about once a minute: 3 restarts in 4 minutes, the pod not Ready, and the initial sync never finished. Raising the limit to 128Mi fixed that: the working set stayed at about 72Mi with a 95Mi peak, and there were 0 restarts. That was with 4 small ConfigMaps in one namespace, so it looks like baseline usage of
kiwigrid/k8s-sidecar:2.10.1rather than load. I kept the default out of this PR so the change stays focused; happy to open a follow-up if you'd like the limit raised.🤖 Generated with Claude Code
https://claude.ai/code/session_01TLFnUd6g7xhDXDvaho4rDW