_base/bin/verify.sh: verified end to end on ClickStack 2.39.1

[litkhai]

Closes #52

_base/bin/verify.sh has now run end to end on clickstack-all-in-one:2.39.1. The first run found that its third layer was wrong, so this PR fixes the script as well as recording the run. It also corrects the collector facts that the pinned image contradicted.

What changed and why

• _base/bin/verify.sh: the "searchable via HyperDX" layer passed when it should not have.
  • On the first run the layer printed PASS searchable via HyperDX (1 rows) while 200 rows were stored.
  • Cause, read from the running 2.39.1 API: POST /api/v2/search with the lucene field verify.run_id answers {"message":"UNKNOWN_IDENTIFIER: Unknown expression or function identifier verify.run_id ..."}. A resource attribute is addressed as ResourceAttributes.verify.run_id.
  • The script counted len(d.get('data', d)). For an error object that is the number of keys, so any error read as "1 rows" and passed.
  • Now the lucene field is ResourceAttributes.verify.run_id, only a data list counts, and a response without one is a FAIL that prints the server's message.
  • What the layer checks is unchanged: rows found through the HyperDX search for this run_id, more than zero.
  • With N=3: the old script passed with 1 rows; a copy with the old field name now prints FAIL ... UNKNOWN_IDENTIFIER ... and exits 1; the fixed script passes with 3 rows.
• _base/README.md: **Verified on:** line in both languages, and the "standalone as well as supervisor" sentence corrected.
• otel-profiles/CONVENTIONS.md, otel-profiles/README.md, otel-profiles/profiles/virt-vsphere/README.md: collector facts corrected, both languages (see below).

How it was run

• Local OSS stack from _base/docker-compose.yml, default profile only, clickstack-all-in-one:2.39.1, on empty volumes.
• ClickHouse was published on 18123/19000 because another local stack held 8123/9000. The change was made through an uncommitted override file that is not in the repository, and nothing else differed from the README's commands.
• A throwaway HyperDX user was registered through the local API (POST /register/password, generated @example.com email and password). Its ingestion key (/team) and personal access key (/me) were appended to an untracked, mode 600 _base/.env and are not in this PR, its description or its commits (a count-only scan of the branch diff and tracked files for all four values found zero occurrences). This is how this run got its keys; it does not change how the README tells a reader to get them (the UI).

bin/check.sh

target: oss

PASS  ClickHouse reachable (version 26.8.7.19)
PASS  database 'default' exists
      otel_* tables in 'default': 11
SKIP  migration target ClickHouse
     CH_TARGET_URL is not set -- only labs/elastic-migration/ uses it
PASS  HyperDX API reachable (4 sources)
PASS  collector health
PASS  collector internal telemetry (1 receiver series)

1 check(s) skipped -- a skip is not a pass.
target 'oss' is ready.

bin/verify.sh (full run, N=200, the fixed script)

run_id=v179082721353927  expecting 200 logs
PASS  emitted 200 logs
PASS  stored in otel_logs (200 rows)
PASS  searchable via HyperDX (200 rows)

verified.
exit=0

telemetrygen emits about one log per second, so the run takes about four minutes.

Negative control

telemetrygen (same image and tag as verify.sh) with a wrong authorization header and its own verify.run_id, 20 logs:

docker run --rm --add-host=host.docker.internal:host-gateway \
  ghcr.io/open-telemetry/opentelemetry-collector-contrib/telemetrygen:v0.155.0 \
  logs --otlp-insecure --logs 20 --otlp-endpoint host.docker.internal:4317 \
  --otlp-header authorization=\"bogus-not-a-key\" \
  --otlp-attributes service.name=\"verify\" --otlp-attributes verify.run_id=\"neg179082722454028\"

telemetrygen exited 1: failed to export batched logs ... rpc error: code = Unauthenticated desc = provided authorization does not match expected scheme or token. Then, polled for 60 s (12 x 5 s, as verify.sh waits), as the api ClickHouse user:

SELECT count() FROM default.otel_logs WHERE ResourceAttributes['verify.run_id'] = 'neg179082722454028'
-> 0   (max seen across all 12 polls: 0)

The same query shape returns 200 for the verify.sh runs' ids, so the 0 is a result and not a broken query. A request with no authorization header at all to OTLP/HTTP :4318 is answered 401 {"code":16,"message":"missing or empty authorization header: Authorization"}.

Layer 1 on the real script: ./bin/verify.sh with a scratch env file carrying a bogus HYPERDX_INGESTION_KEY prints FAIL emitted 200 logs and exits 1.

Versions

• ClickHouse: SELECT version() against the running server returned 26.8.7.19 (also in the check.sh output above).
• ClickStack/HyperDX: GET /health returned "version":"2.39.1"; container image clickhouse/clickstack-all-in-one:2.39.1.
• Collector: otelcol-hyperdx 0.155.0 (/otelcontribcol --version).
• telemetrygen: v0.155.0.

Collector facts read from the image

Read inside clickhouse/clickstack-all-in-one:2.39.1 (otelcol-hyperdx 0.155.0), not from docs:

• Components. /otelcontribcol components:
  • receivers: datadog docker_stats file_log fluent_forward host_metrics k8s_cluster kubelet_stats nop otlp prometheus
  • processors: attributes batch cumulativetodelta filter groupbyattrs k8s_attributes log_dedup memory_limiter metrics_transform probabilistic_sampler redaction resource resource_detection span tail_sampling transform
  • exporters: clickhouse debug nop otlp_grpc otlp_http prometheus_remote_write
  • connectors: forward routing span_metrics
  • extensions: bearertokenauth file_storage health_check memory_limiter oidc opamp pprof zpages
• Receiver spellings. /otelcontribcol validate --config "yaml:...", one receiver per run in a throwaway container: filelog, fluentforward, hostmetrics, kubeletstats are accepted as aliases; docker_stats and k8s_cluster are accepted; dockerstats and k8scluster are rejected as unknown types; statsd is not in this build. The processors resourcedetection and metricstransform are accepted next to resource_detection and metrics_transform.
• How CUSTOM_OTELCOL_CONFIG_FILE is loaded.
  • /etc/local/entry.base.sh exports OPAMP_SERVER_URL, so /otel-entrypoint.sh takes its supervisor branch and starts opampsupervisor.
  • /etc/otel/supervisor.yaml.tmpl renders agent.config_files as /etc/otelcol-contrib/config.yaml followed by $CUSTOM_OTELCOL_CONFIG_FILE (only if set).
  • The OpAMP remote config from the API is merged after those.
  • The standalone branch (--config config.yaml --config standalone-config.yaml --config $CUSTOM_OTELCOL_CONFIG_FILE) exists in /otel-entrypoint.sh but is not taken by the all-in-one image, and /etc/otelcol-contrib/ there holds only config.yaml.
• Base config. /etc/otelcol-contrib/config.yaml defines transform, resourcedetection, batch (send_batch_size 10000, timeout 5s) and memory_limiter (limit_mib 1500). Its pipelines are traces and metrics (memory_limiter, batch), logs/out-default (memory_limiter, transform, batch) and logs/out-rrweb (memory_limiter, batch).
• Runtime-injected components. clickhouse, clickhouse/rrweb and otlp/hyperdx are not in config.yaml. /app/packages/api/build/opamp/controllers/opampController.js builds them, plus the logs/in pipeline and the routing/logs connector. otlp/hyperdx is attached to traces, metrics and logs/in only when a team has an API key (or INGESTION_API_KEY is set). When the team has collectorAuthenticationEnforced (set to true by POST /register/password), a bearertokenauth/hyperdx extension with scheme: '' and the key in bare and Bearer-prefixed forms guards both OTLP protocols.
• Registration and keys (used for this run). /app/packages/api/build/routers/api/root.js: GET /installation (isTeamExisting), POST /register/password (409 if a team exists). routers/api/me.js: GET /me returns accessKey. routers/api/team.js: GET /team returns apiKey. Password rule from packages/common-utils/dist/validation.js: 12 to 72 characters with upper case, lower case, a digit and a special character.
• Search request shape. routers/external-api/v2/search.js: sourceId, where, whereLanguage (lucene by default), maxResults (1 to 2000), and a 15-minute default window; the response is {data, rows}. Errors are {message}.

Corrections made from these (both languages): CONVENTIONS.md rules 1 to 4, the receiver lists in otel-profiles/README.md and virt-vsphere/README.md, and _base/README.md's standalone sentence. The merge-order consequence of a bare pipeline key (rule 2) is marked as read from the code, not tested.

Not run

• STATUS.md is untouched (issue _base/bin/verify.sh: run it end to end once and record what it ran on #52's "Done" list mentions it; that change is not in this PR).
• The standalone branch of /otel-entrypoint.sh, and the collector that runs next to a ClickHouse Cloud service: neither was exercised.
• Whether a bare traces: / metrics: key in a custom config loses the profile's receiver or ClickStack's OTLP ingestion on the supervisor path: not tested.
• The OTLP/HTTP path (:4318) with a valid key, and verify.sh with OTLP_ENDPOINT overridden.
• The elastic, elastic-secure and migration profiles were not started; check.sh reports their ClickHouse target as SKIP.
• Linux hosts: this ran on Docker Desktop for macOS.
• The profile configs (custom.config.yaml / sidecar.config.yaml) were linted and merged (lint.sh, build-config.sh) but not loaded into a running collector.
• Not run locally: the terraform, hygiene and pages jobs. Run locally: lint.sh, the two build-config.sh merges, check_links.py, check_syntax.sh, bash -n _base/bin/verify.sh, and gitleaks on the branch.
• When verify.sh fails layer 1 because the key is wrong, its hint still says "is OTLP on :4317 reachable?" (telemetrygen's output is discarded). I did not change that message.

🤖 Generated with Claude Code