Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/dashboard-watcher-memory.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"helm-charts": patch
---

fix: raise the dashboard watcher's default memory to a 128Mi limit and a 64Mi request (were 64Mi and 32Mi). At 64Mi the `kiwigrid/k8s-sidecar` watcher was OOMKilled about once a minute even with only a few small dashboards, so its initial sync never finished and the pod never became Ready; at 128Mi its working set stayed near 72Mi with a 95Mi peak and it did not restart. Override `hyperdx.dashboards.resources` to keep the old values.
2 changes: 1 addition & 1 deletion charts/clickstack/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ Documented values below. Undocumented keys (operator CRD passthrough fields, sub
| global.imageRegistry | string | `""` | Optional prefix for container images (e.g. registry.example.com/). |
| hyperdx.autoscaling | object | `{"enabled":false,"spec":{}}` | HorizontalPodAutoscaler for HyperDX. spec is passed through verbatim (autoscaling/v2). Do not include scaleTargetRef — the chart auto-wires it. When enabled, omit deployment.replicas; disabling autoscaling reverts to deployment.replicas (default 1). |
| hyperdx.config | object | `{"API_PORT":"8000","APP_PORT":"3000","CLICKHOUSE_ENDPOINT":"tcp://{{ include \"clickstack.clickhouse.svc\" . }}:{{ .Values.clickhouse.nativePort }}?dial_timeout=10s","CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT":"{{ include \"clickstack.clickhouse.svc\" . }}:{{ .Values.clickhouse.prometheus.port }}","CLICKHOUSE_SERVER_ENDPOINT":"{{ include \"clickstack.clickhouse.svc\" . }}:{{ .Values.clickhouse.nativePort }}","CLICKHOUSE_USER":"otelcollector","ENABLE_PROMQL":"{{ if .Values.hyperdx.promql.enabled }}true{{ else }}false{{ end }}","FRONTEND_URL":"http://localhost:3000","HDX_EXPORTER_ENABLED":"{{ if .Values.hyperdx.config.NEXT_PUBLIC_OTEL_EXPORTER_OTLP_ENDPOINT }}true{{ else }}false{{ end }}","HYPERDX_API_PORT":"8000","HYPERDX_APP_PORT":"3000","HYPERDX_LOG_LEVEL":"info","HYPERDX_OTEL_EXPORTER_CLICKHOUSE_DATABASE":"default","HYPERDX_OTEL_EXPORTER_TABLES_TTL":"720h","MONGO_URI":"mongodb://hyperdx:{{ .Values.hyperdx.secrets.MONGODB_PASSWORD }}@{{ include \"clickstack.mongodb.svc\" . }}:27017/hyperdx?authSource=hyperdx","NEXT_PUBLIC_ENABLE_PROMQL":"{{ if .Values.hyperdx.promql.enabled }}true{{ else }}false{{ end }}","OPAMP_PORT":"4320","OPAMP_SERVER_URL":"http://{{ include \"clickstack.hyperdx.fullname\" . }}:{{ .Values.hyperdx.ports.opamp }}","OTEL_EXPORTER_OTLP_ENDPOINT":"http://{{ include \"clickstack.otel.fullname\" . }}:4318","OTEL_SERVICE_NAME":"hdx-oss-api","RUN_SCHEDULED_TASKS_EXTERNALLY":"false","USAGE_STATS_ENABLED":"true"}` | Shared non-sensitive environment variables (clickstack-config ConfigMap). Used by HyperDX and OTEL collector via envFrom. Values support tpl expressions. Override entries with plain strings to point at external services. Changes to the rendered ConfigMap roll the HyperDX Deployment; the otel-collector rolls on every helm upgrade. |
| hyperdx.dashboards | object | `{"configMaps":{},"enabled":false,"namespaces":[],"rbac":{"create":true},"resources":{"limits":{"cpu":"50m","memory":"64Mi"},"requests":{"cpu":"10m","memory":"32Mi"}},"sidecarImage":"kiwigrid/k8s-sidecar:2.10.1","sidecarPullPolicy":"IfNotPresent"}` | Dashboard provisioning: upserts dashboard JSON into MongoDB (matched by name, never deletes). Provide dashboards inline via configMaps, or label ConfigMaps hyperdx.io/dashboard: "true" for a k8s-sidecar watcher to discover. |
| hyperdx.dashboards | object | `{"configMaps":{},"enabled":false,"namespaces":[],"rbac":{"create":true},"resources":{"limits":{"cpu":"50m","memory":"128Mi"},"requests":{"cpu":"10m","memory":"64Mi"}},"sidecarImage":"kiwigrid/k8s-sidecar:2.10.1","sidecarPullPolicy":"IfNotPresent"}` | Dashboard provisioning: upserts dashboard JSON into MongoDB (matched by name, never deletes). Provide dashboards inline via configMaps, or label ConfigMaps hyperdx.io/dashboard: "true" for a k8s-sidecar watcher to discover. |
| hyperdx.deployment | object | `{"annotations":{},"containerSecurityContext":{},"defaultConnections":"[\n {\n \"name\": \"Local ClickHouse\",\n \"host\": \"http://{{ include \"clickstack.clickhouse.svc\" . }}:8123\",\n \"port\": 8123,\n \"username\": \"app\",\n \"password\": \"{{ .Values.hyperdx.secrets.CLICKHOUSE_APP_PASSWORD }}\"\n }\n]\n","defaultSources":"[\n {\n \"from\": {\n \"databaseName\": \"default\",\n \"tableName\": \"otel_logs\"\n },\n \"kind\": \"log\",\n \"timestampValueExpression\": \"Timestamp\",\n \"name\": \"Logs\",\n \"displayedTimestampValueExpression\": \"Timestamp\",\n \"implicitColumnExpression\": \"Body\",\n \"serviceNameExpression\": \"ServiceName\",\n \"bodyExpression\": \"Body\",\n \"eventAttributesExpression\": \"LogAttributes\",\n \"resourceAttributesExpression\": \"ResourceAttributes\",\n \"defaultTableSelectExpression\": \"Timestamp,ServiceName,SeverityText,Body\",\n \"severityTextExpression\": \"SeverityText\",\n \"traceIdExpression\": \"TraceId\",\n \"spanIdExpression\": \"SpanId\",\n \"connection\": \"Local ClickHouse\",\n \"traceSourceId\": \"Traces\",\n \"metricSourceId\": \"Metrics\"\n },\n {\n \"from\": {\n \"databaseName\": \"default\",\n \"tableName\": \"otel_traces\"\n },\n \"kind\": \"trace\",\n \"timestampValueExpression\": \"Timestamp\",\n \"name\": \"Traces\",\n \"displayedTimestampValueExpression\": \"Timestamp\",\n \"implicitColumnExpression\": \"SpanName\",\n \"serviceNameExpression\": \"ServiceName\",\n \"eventAttributesExpression\": \"SpanAttributes\",\n \"resourceAttributesExpression\": \"ResourceAttributes\",\n \"defaultTableSelectExpression\": \"Timestamp,ServiceName,StatusCode,round(Duration/1e6),SpanName\",\n \"traceIdExpression\": \"TraceId\",\n \"spanIdExpression\": \"SpanId\",\n \"durationExpression\": \"Duration\",\n \"durationPrecision\": 9,\n \"parentSpanIdExpression\": \"ParentSpanId\",\n \"spanNameExpression\": \"SpanName\",\n \"spanKindExpression\": \"SpanKind\",\n \"statusCodeExpression\": \"StatusCode\",\n \"statusMessageExpression\": \"StatusMessage\",\n \"spanEventsValueExpression\": \"Events\",\n \"spanLinksValueExpression\": \"Links\",\n \"connection\": \"Local ClickHouse\",\n \"logSourceId\": \"Logs\",\n \"sessionSourceId\": \"Sessions\",\n \"metricSourceId\": \"Metrics\"\n },\n {\n \"from\": {\n \"databaseName\": \"default\",\n \"tableName\": \"\"\n },\n \"kind\": \"metric\",\n \"timestampValueExpression\": \"TimeUnix\",\n \"name\": \"Metrics\",\n \"resourceAttributesExpression\": \"ResourceAttributes\",\n \"metricTables\": {\n \"gauge\": \"otel_metrics_gauge\",\n \"histogram\": \"otel_metrics_histogram\",\n \"sum\": \"otel_metrics_sum\",\n \"summary\": \"otel_metrics_summary\",\n \"exponential histogram\": \"otel_metrics_exponential_histogram\"\n },\n \"connection\": \"Local ClickHouse\",\n \"logSourceId\": \"Logs\"\n },\n {\n \"from\": {\n \"databaseName\": \"default\",\n \"tableName\": \"hyperdx_sessions\"\n },\n \"kind\": \"session\",\n \"timestampValueExpression\": \"TimestampTime\",\n \"name\": \"Sessions\",\n \"resourceAttributesExpression\": \"ResourceAttributes\",\n \"connection\": \"Local ClickHouse\",\n \"traceSourceId\": \"Traces\"\n },\n {\n \"from\": {\n \"databaseName\": \"default\",\n \"tableName\": \"metrics_ts\"\n },\n \"kind\": \"promql\",\n \"timestampValueExpression\": \"timestamp\",\n \"name\": \"PromQL\",\n \"connection\": \"Local ClickHouse\"\n }\n]\n","deploymentAnnotations":{},"env":[],"existingConfigConnectionsKey":"connections.json","existingConfigSecret":"","existingConfigSourcesKey":"sources.json","image":{"pullPolicy":"IfNotPresent","repository":"docker.hyperdx.io/hyperdx/hyperdx","tag":null},"initContainers":[],"labels":{},"livenessProbe":{"enabled":true,"failureThreshold":3,"initialDelaySeconds":10,"path":"/health","periodSeconds":30,"timeoutSeconds":5},"nodeSelector":{},"podAnnotations":{},"podLabels":{},"podSecurityContext":{},"priorityClassName":"","readinessProbe":{"enabled":true,"failureThreshold":3,"initialDelaySeconds":1,"path":"/ready","periodSeconds":10,"timeoutSeconds":5},"replicas":1,"resources":{},"tolerations":[],"topologySpreadConstraints":[],"useExistingConfigSecret":false,"volumeMounts":[],"volumes":[],"waitForMongodb":{"image":"busybox@sha256:1fcf5df59121b92d61e066df1788e8df0cc35623f5d62d9679a41e163b6a0cdb","pullPolicy":"IfNotPresent"}}` | HyperDX Deployment spec (image, probes, scheduling, init containers, volumes). defaultConnections/defaultSources JSON: set to "" to disable; use useExistingConfigSecret for external secrets. |
| hyperdx.deployment.env | list | `[]` | Extra environment variables for the HyperDX container, rendered verbatim (supports `valueFrom`, e.g. `secretKeyRef`). |
| hyperdx.deployment.existingConfigSecret | string | `""` | Secret containing connections and sources JSON when useExistingConfigSecret is true. |
Expand Down
4 changes: 2 additions & 2 deletions charts/clickstack/tests/dashboard-provisioner_test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -381,15 +381,15 @@ tests:
asserts:
- equal:
path: spec.template.spec.containers[1].resources.limits.memory
value: 64Mi
value: 128Mi
template: hyperdx/deployment.yaml
- equal:
path: spec.template.spec.containers[1].resources.limits.cpu
value: 50m
template: hyperdx/deployment.yaml
- equal:
path: spec.template.spec.containers[1].resources.requests.memory
value: 32Mi
value: 64Mi
template: hyperdx/deployment.yaml
- equal:
path: spec.template.spec.containers[1].resources.requests.cpu
Expand Down
7 changes: 5 additions & 2 deletions charts/clickstack/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -89,13 +89,16 @@ hyperdx:
# { "name": "Kubernetes Overview", "tiles": [...] }
# Sidecar is a lightweight ConfigMap watcher; bump (esp. memory) for very large
# dashboard sets or wide `namespaces: [ALL]` discovery to avoid OOMKill.
# Even idle it needs more than 64Mi: with a handful of small dashboards its working set
# is about 72Mi and peaks near 95Mi, so the limit leaves headroom above that peak and
# the request reserves about the working set.
resources:
limits:
cpu: 50m
memory: 64Mi
memory: 128Mi
requests:
cpu: 10m
memory: 32Mi
memory: 64Mi

# -- Shared non-sensitive environment variables (clickstack-config ConfigMap).
# Used by HyperDX and OTEL collector via envFrom. Values support tpl expressions.
Expand Down
Loading