Skip to content

fix: raise the dashboard watcher's default memory (128Mi limit, 64Mi request) - #327

Merged
teeohhem merged 1 commit into
ClickHouse:mainfrom
braghettos:feat/dashboard-watcher-memory
Nov 18, 2026
Merged

teeohhem merged 1 commit into
ClickHouse:mainfrom
braghettos:feat/dashboard-watcher-memory

Conversation

@braghettos

Copy link
Copy Markdown
Contributor

At the 64Mi limit the kiwigrid/k8s-sidecar dashboard watcher was OOMKilled
about once a minute on a kind cluster, with only 4 small dashboard ConfigMaps
in one namespace: 3 restarts in 4 minutes, the pod never Ready, the initial
sync never finished. With the limit at 128Mi its working set stayed at about
72Mi with a 95Mi peak, and it did not restart. So this is the sidecar's
baseline, not load.

The default limit becomes 128Mi (headroom above the measured peak) and the
request 64Mi (about the measured working set, so the scheduler reserves what
it uses). CPU is unchanged. The unit test for the default resources pins the
new values; values.yaml says where the numbers come from.

Follow-up to #315, as discussed in its review.

🤖 Generated with Claude Code

https://claude.ai/code/session_016RAxvjuD1TdxneAHTCFpW8

…request)

At the 64Mi limit the kiwigrid/k8s-sidecar dashboard watcher was OOMKilled
about once a minute on a kind cluster, with only 4 small dashboard ConfigMaps
in one namespace: 3 restarts in 4 minutes, the pod never Ready, the initial
sync never finished. With the limit at 128Mi its working set stayed at about
72Mi with a 95Mi peak, and it did not restart. So this is the sidecar's
baseline, not load.

The default limit becomes 128Mi (headroom above the measured peak) and the
request 64Mi (about the measured working set, so the scheduler reserves what
it uses). CPU is unchanged. The unit test for the default resources pins the
new values; values.yaml says where the numbers come from.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016RAxvjuD1TdxneAHTCFpW8
@braghettos
braghettos requested a review from a team as a code owner November 18, 2026 12:01
@changeset-bot

changeset-bot Bot commented Nov 18, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 25674bd

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
helm-charts Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions github-actions Bot added the external Opened by an external contributor label Nov 18, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Deep Review

✅ No critical issues found. This is a self-contained default-value bump (dashboard watcher memory: 128Mi limit / 64Mi request) with the unit test, auto-generated README, and a patch changeset all updated in lockstep. Template rendering, values/test/README consistency, and QoS class (unchanged Burstable) all verified clean.

🟡 P2 — recommended

  • docs/UPGRADE.md:5 — No ## Unreleased note for a change that rolls the HyperDX Deployment when hyperdx.dashboards.enabled: true, which AGENTS.md's changeset convention calls for on pod-restarting changes.
    • Fix: Add a bullet under ## Unreleased noting the watcher's default memory rose to a 128Mi limit / 64Mi request, that the Deployment rolls once for installs with dashboards enabled, and that hyperdx.dashboards.resources can be overridden to keep the old values.
🔵 P3 nitpicks (2)
  • charts/clickstack/values.yaml:94 — The comment says "the request reserves about the working set" but the request is 64Mi while the same comment cites the working set as ~72Mi, so it under-reserves the stated figure by ~8Mi.
    • Fix: Reword to reflect that the request reserves close to, but slightly under, the typical working set.
  • charts/clickstack/values.yaml:92 — The new baseline note ("even idle it needs more than 64Mi") sits adjacent to the existing comment that frames memory purely as load-driven ("bump for very large dashboard sets or wide namespaces discovery"), and the two read in tension.
    • Fix: Merge the two notes into one: a ~72Mi/95Mi-peak baseline regardless of load, with a further bump for very large sets or [ALL] discovery.

Reviewers (4): correctness, testing, maintainability, project-standards.

Testing gaps: none — the updated dashboard-provisioner_test.yaml pins all four new default resource fields; no new branch or behavior is left uncovered.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR.

@teeohhem
teeohhem enabled auto-merge (squash) November 18, 2026 19:06
@teeohhem
teeohhem merged commit 9f9aedf into ClickHouse:main Nov 18, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

external Opened by an external contributor

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants