From 25674bd51e41e02fee57123c17cec9d60ac879cd Mon Sep 17 00:00:00 2001 From: Diego Braga Date: Wed, 18 Nov 2026 13:01:00 +0100 Subject: [PATCH] fix: raise the dashboard watcher's default memory (128Mi limit, 64Mi request) At the 64Mi limit the kiwigrid/k8s-sidecar dashboard watcher was OOMKilled about once a minute on a kind cluster, with only 4 small dashboard ConfigMaps in one namespace: 3 restarts in 4 minutes, the pod never Ready, the initial sync never finished. With the limit at 128Mi its working set stayed at about 72Mi with a 95Mi peak, and it did not restart. So this is the sidecar's baseline, not load. The default limit becomes 128Mi (headroom above the measured peak) and the request 64Mi (about the measured working set, so the scheduler reserves what it uses). CPU is unchanged. The unit test for the default resources pins the new values; values.yaml says where the numbers come from. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016RAxvjuD1TdxneAHTCFpW8 --- .changeset/dashboard-watcher-memory.md | 5 +++++ charts/clickstack/README.md | 2 +- charts/clickstack/tests/dashboard-provisioner_test.yaml | 4 ++-- charts/clickstack/values.yaml | 7 +++++-- 4 files changed, 13 insertions(+), 5 deletions(-) create mode 100644 .changeset/dashboard-watcher-memory.md diff --git a/.changeset/dashboard-watcher-memory.md b/.changeset/dashboard-watcher-memory.md new file mode 100644 index 0000000..2ab2afa --- /dev/null +++ b/.changeset/dashboard-watcher-memory.md @@ -0,0 +1,5 @@ +--- +"helm-charts": patch +--- + +fix: raise the dashboard watcher's default memory to a 128Mi limit and a 64Mi request (were 64Mi and 32Mi). At 64Mi the `kiwigrid/k8s-sidecar` watcher was OOMKilled about once a minute even with only a few small dashboards, so its initial sync never finished and the pod never became Ready; at 128Mi its working set stayed near 72Mi with a 95Mi peak and it did not restart. Override `hyperdx.dashboards.resources` to keep the old values. diff --git a/charts/clickstack/README.md b/charts/clickstack/README.md index c9b20d4..28c0285 100644 --- a/charts/clickstack/README.md +++ b/charts/clickstack/README.md @@ -57,7 +57,7 @@ Documented values below. Undocumented keys (operator CRD passthrough fields, sub | global.imageRegistry | string | `""` | Optional prefix for container images (e.g. registry.example.com/). | | hyperdx.autoscaling | object | `{"enabled":false,"spec":{}}` | HorizontalPodAutoscaler for HyperDX. spec is passed through verbatim (autoscaling/v2). Do not include scaleTargetRef — the chart auto-wires it. When enabled, omit deployment.replicas; disabling autoscaling reverts to deployment.replicas (default 1). | | hyperdx.config | object | `{"API_PORT":"8000","APP_PORT":"3000","CLICKHOUSE_ENDPOINT":"tcp://{{ include \"clickstack.clickhouse.svc\" . }}:{{ .Values.clickhouse.nativePort }}?dial_timeout=10s","CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT":"{{ include \"clickstack.clickhouse.svc\" . }}:{{ .Values.clickhouse.prometheus.port }}","CLICKHOUSE_SERVER_ENDPOINT":"{{ include \"clickstack.clickhouse.svc\" . }}:{{ .Values.clickhouse.nativePort }}","CLICKHOUSE_USER":"otelcollector","ENABLE_PROMQL":"{{ if .Values.hyperdx.promql.enabled }}true{{ else }}false{{ end }}","FRONTEND_URL":"http://localhost:3000","HDX_EXPORTER_ENABLED":"{{ if .Values.hyperdx.config.NEXT_PUBLIC_OTEL_EXPORTER_OTLP_ENDPOINT }}true{{ else }}false{{ end }}","HYPERDX_API_PORT":"8000","HYPERDX_APP_PORT":"3000","HYPERDX_LOG_LEVEL":"info","HYPERDX_OTEL_EXPORTER_CLICKHOUSE_DATABASE":"default","HYPERDX_OTEL_EXPORTER_TABLES_TTL":"720h","MONGO_URI":"mongodb://hyperdx:{{ .Values.hyperdx.secrets.MONGODB_PASSWORD }}@{{ include \"clickstack.mongodb.svc\" . }}:27017/hyperdx?authSource=hyperdx","NEXT_PUBLIC_ENABLE_PROMQL":"{{ if .Values.hyperdx.promql.enabled }}true{{ else }}false{{ end }}","OPAMP_PORT":"4320","OPAMP_SERVER_URL":"http://{{ include \"clickstack.hyperdx.fullname\" . }}:{{ .Values.hyperdx.ports.opamp }}","OTEL_EXPORTER_OTLP_ENDPOINT":"http://{{ include \"clickstack.otel.fullname\" . }}:4318","OTEL_SERVICE_NAME":"hdx-oss-api","RUN_SCHEDULED_TASKS_EXTERNALLY":"false","USAGE_STATS_ENABLED":"true"}` | Shared non-sensitive environment variables (clickstack-config ConfigMap). Used by HyperDX and OTEL collector via envFrom. Values support tpl expressions. Override entries with plain strings to point at external services. Changes to the rendered ConfigMap roll the HyperDX Deployment; the otel-collector rolls on every helm upgrade. | -| hyperdx.dashboards | object | `{"configMaps":{},"enabled":false,"namespaces":[],"rbac":{"create":true},"resources":{"limits":{"cpu":"50m","memory":"64Mi"},"requests":{"cpu":"10m","memory":"32Mi"}},"sidecarImage":"kiwigrid/k8s-sidecar:2.10.1","sidecarPullPolicy":"IfNotPresent"}` | Dashboard provisioning: upserts dashboard JSON into MongoDB (matched by name, never deletes). Provide dashboards inline via configMaps, or label ConfigMaps hyperdx.io/dashboard: "true" for a k8s-sidecar watcher to discover. | +| hyperdx.dashboards | object | `{"configMaps":{},"enabled":false,"namespaces":[],"rbac":{"create":true},"resources":{"limits":{"cpu":"50m","memory":"128Mi"},"requests":{"cpu":"10m","memory":"64Mi"}},"sidecarImage":"kiwigrid/k8s-sidecar:2.10.1","sidecarPullPolicy":"IfNotPresent"}` | Dashboard provisioning: upserts dashboard JSON into MongoDB (matched by name, never deletes). Provide dashboards inline via configMaps, or label ConfigMaps hyperdx.io/dashboard: "true" for a k8s-sidecar watcher to discover. | | hyperdx.deployment | object | `{"annotations":{},"containerSecurityContext":{},"defaultConnections":"[\n {\n \"name\": \"Local ClickHouse\",\n \"host\": \"http://{{ include \"clickstack.clickhouse.svc\" . }}:8123\",\n \"port\": 8123,\n \"username\": \"app\",\n \"password\": \"{{ .Values.hyperdx.secrets.CLICKHOUSE_APP_PASSWORD }}\"\n }\n]\n","defaultSources":"[\n {\n \"from\": {\n \"databaseName\": \"default\",\n \"tableName\": \"otel_logs\"\n },\n \"kind\": \"log\",\n \"timestampValueExpression\": \"Timestamp\",\n \"name\": \"Logs\",\n \"displayedTimestampValueExpression\": \"Timestamp\",\n \"implicitColumnExpression\": \"Body\",\n \"serviceNameExpression\": \"ServiceName\",\n \"bodyExpression\": \"Body\",\n \"eventAttributesExpression\": \"LogAttributes\",\n \"resourceAttributesExpression\": \"ResourceAttributes\",\n \"defaultTableSelectExpression\": \"Timestamp,ServiceName,SeverityText,Body\",\n \"severityTextExpression\": \"SeverityText\",\n \"traceIdExpression\": \"TraceId\",\n \"spanIdExpression\": \"SpanId\",\n \"connection\": \"Local ClickHouse\",\n \"traceSourceId\": \"Traces\",\n \"metricSourceId\": \"Metrics\"\n },\n {\n \"from\": {\n \"databaseName\": \"default\",\n \"tableName\": \"otel_traces\"\n },\n \"kind\": \"trace\",\n \"timestampValueExpression\": \"Timestamp\",\n \"name\": \"Traces\",\n \"displayedTimestampValueExpression\": \"Timestamp\",\n \"implicitColumnExpression\": \"SpanName\",\n \"serviceNameExpression\": \"ServiceName\",\n \"eventAttributesExpression\": \"SpanAttributes\",\n \"resourceAttributesExpression\": \"ResourceAttributes\",\n \"defaultTableSelectExpression\": \"Timestamp,ServiceName,StatusCode,round(Duration/1e6),SpanName\",\n \"traceIdExpression\": \"TraceId\",\n \"spanIdExpression\": \"SpanId\",\n \"durationExpression\": \"Duration\",\n \"durationPrecision\": 9,\n \"parentSpanIdExpression\": \"ParentSpanId\",\n \"spanNameExpression\": \"SpanName\",\n \"spanKindExpression\": \"SpanKind\",\n \"statusCodeExpression\": \"StatusCode\",\n \"statusMessageExpression\": \"StatusMessage\",\n \"spanEventsValueExpression\": \"Events\",\n \"spanLinksValueExpression\": \"Links\",\n \"connection\": \"Local ClickHouse\",\n \"logSourceId\": \"Logs\",\n \"sessionSourceId\": \"Sessions\",\n \"metricSourceId\": \"Metrics\"\n },\n {\n \"from\": {\n \"databaseName\": \"default\",\n \"tableName\": \"\"\n },\n \"kind\": \"metric\",\n \"timestampValueExpression\": \"TimeUnix\",\n \"name\": \"Metrics\",\n \"resourceAttributesExpression\": \"ResourceAttributes\",\n \"metricTables\": {\n \"gauge\": \"otel_metrics_gauge\",\n \"histogram\": \"otel_metrics_histogram\",\n \"sum\": \"otel_metrics_sum\",\n \"summary\": \"otel_metrics_summary\",\n \"exponential histogram\": \"otel_metrics_exponential_histogram\"\n },\n \"connection\": \"Local ClickHouse\",\n \"logSourceId\": \"Logs\"\n },\n {\n \"from\": {\n \"databaseName\": \"default\",\n \"tableName\": \"hyperdx_sessions\"\n },\n \"kind\": \"session\",\n \"timestampValueExpression\": \"TimestampTime\",\n \"name\": \"Sessions\",\n \"resourceAttributesExpression\": \"ResourceAttributes\",\n \"connection\": \"Local ClickHouse\",\n \"traceSourceId\": \"Traces\"\n },\n {\n \"from\": {\n \"databaseName\": \"default\",\n \"tableName\": \"metrics_ts\"\n },\n \"kind\": \"promql\",\n \"timestampValueExpression\": \"timestamp\",\n \"name\": \"PromQL\",\n \"connection\": \"Local ClickHouse\"\n }\n]\n","deploymentAnnotations":{},"env":[],"existingConfigConnectionsKey":"connections.json","existingConfigSecret":"","existingConfigSourcesKey":"sources.json","image":{"pullPolicy":"IfNotPresent","repository":"docker.hyperdx.io/hyperdx/hyperdx","tag":null},"initContainers":[],"labels":{},"livenessProbe":{"enabled":true,"failureThreshold":3,"initialDelaySeconds":10,"path":"/health","periodSeconds":30,"timeoutSeconds":5},"nodeSelector":{},"podAnnotations":{},"podLabels":{},"podSecurityContext":{},"priorityClassName":"","readinessProbe":{"enabled":true,"failureThreshold":3,"initialDelaySeconds":1,"path":"/ready","periodSeconds":10,"timeoutSeconds":5},"replicas":1,"resources":{},"tolerations":[],"topologySpreadConstraints":[],"useExistingConfigSecret":false,"volumeMounts":[],"volumes":[],"waitForMongodb":{"image":"busybox@sha256:1fcf5df59121b92d61e066df1788e8df0cc35623f5d62d9679a41e163b6a0cdb","pullPolicy":"IfNotPresent"}}` | HyperDX Deployment spec (image, probes, scheduling, init containers, volumes). defaultConnections/defaultSources JSON: set to "" to disable; use useExistingConfigSecret for external secrets. | | hyperdx.deployment.env | list | `[]` | Extra environment variables for the HyperDX container, rendered verbatim (supports `valueFrom`, e.g. `secretKeyRef`). | | hyperdx.deployment.existingConfigSecret | string | `""` | Secret containing connections and sources JSON when useExistingConfigSecret is true. | diff --git a/charts/clickstack/tests/dashboard-provisioner_test.yaml b/charts/clickstack/tests/dashboard-provisioner_test.yaml index a3da0a3..a8a1596 100644 --- a/charts/clickstack/tests/dashboard-provisioner_test.yaml +++ b/charts/clickstack/tests/dashboard-provisioner_test.yaml @@ -381,7 +381,7 @@ tests: asserts: - equal: path: spec.template.spec.containers[1].resources.limits.memory - value: 64Mi + value: 128Mi template: hyperdx/deployment.yaml - equal: path: spec.template.spec.containers[1].resources.limits.cpu @@ -389,7 +389,7 @@ tests: template: hyperdx/deployment.yaml - equal: path: spec.template.spec.containers[1].resources.requests.memory - value: 32Mi + value: 64Mi template: hyperdx/deployment.yaml - equal: path: spec.template.spec.containers[1].resources.requests.cpu diff --git a/charts/clickstack/values.yaml b/charts/clickstack/values.yaml index c66c25f..5c70c80 100644 --- a/charts/clickstack/values.yaml +++ b/charts/clickstack/values.yaml @@ -89,13 +89,16 @@ hyperdx: # { "name": "Kubernetes Overview", "tiles": [...] } # Sidecar is a lightweight ConfigMap watcher; bump (esp. memory) for very large # dashboard sets or wide `namespaces: [ALL]` discovery to avoid OOMKill. + # Even idle it needs more than 64Mi: with a handful of small dashboards its working set + # is about 72Mi and peaks near 95Mi, so the limit leaves headroom above that peak and + # the request reserves about the working set. resources: limits: cpu: 50m - memory: 64Mi + memory: 128Mi requests: cpu: 10m - memory: 32Mi + memory: 64Mi # -- Shared non-sensitive environment variables (clickstack-config ConfigMap). # Used by HyperDX and OTEL collector via envFrom. Values support tpl expressions.