otel-profiles: linux-host and mysql verified in Docker

[litkhai]

Closes #53
Closes #54

Both profiles are now verified in Docker only, on the local _base/ stack: no physical or cloud Linux host, and the MySQL is a container. The Verified lines say so.

What changed

• linux-host/custom.config.yaml: the filelog parser accepts two syslog shapes. Ubuntu 22.04 writes RFC 3164 (Oct 1 04:14:15 host tag[pid]: msg); Ubuntu 24.04 dropped $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat and writes ISO 8601 (2026-10-01T04:13:39.637057+00:00 host tag[pid]: msg), which the old regex did not match at all. The regex takes either ts, and there is one time_parser per shape, selected by if:. The owner chose this on otel-profiles/linux-host: verify in Docker (Docker Desktop VM as the host) #53. metrics.md Logs section updated.
• mysql/custom.config.yaml: a filter operator drops mysqld's three-line slow-log header (/usr/sbin/mysqld, Version: … started with:), which mysqld writes at every start. It is not a query entry, and the restart is already in error.log. The owner chose this on otel-profiles/mysql: verify both halves against a MySQL 8.4 container #54. verify.sql is unchanged. metrics.md has one sentence on it.
• mysql/README.md Prerequisites, both languages: SHOW REPLICA STATUS needs REPLICATION CLIENT (the old text said no grant was needed). Sidecar section: SIDECAR_HEALTH_PORT must be set when the sidecar runs on the same machine as _base.
• mysql/metrics.md, mysql/sidecar.config.yaml (comment): db.system.name at v0.155.0 is an attribute of two disabled log events, not "added in a later release" (see Collector facts).
• sidecar/docker-compose.yml: CLICKSTACK_API_KEY falls back to HYPERDX_INGESTION_KEY; the health port is ${SIDECAR_HEALTH_PORT:-13133} (default unchanged).
• CONVENTIONS.md rule 4, both languages: each receiver alias logs a deprecation warning when the collector starts. Nothing is renamed in the profiles.
• Fixture, new: _base/docker-compose.otel-verify.yml and _base/otel-verify/, with a _base/README.md section in both languages. A plain docker compose up -d does not use it.

Why a fixture

On Docker Desktop, /hostfs is the VM. Its /var/log holds only lastlog, so there is no syslog to read. A named volume can be mounted at /hostfs/var/log but not at /hostfs/var/log/mysql (read-only file system; probed before writing the fixture). So one volume, otel-verify-logs, is mounted at /hostfs/var/log:ro in ClickStack, and rsyslog (two Ubuntu releases, one after the other) and MySQL write into it. The profiles' paths and configs are unchanged. Both rsyslogs run the package's default configuration with only imklog off (a container has no kernel log).

Versions, and how each was read

What	Value	How read
ClickStack	clickstack-all-in-one:2.39.1, collector components 0.155.0	compose pin; components buildinfo
ClickHouse	26.8.7.19	SELECT version() (also in bin/check.sh)
MySQL	8.4.11 (mysql:8.4.11)	SELECT VERSION(); tag from the Docker Hub API 2026-10-01
Sidecar	otel/opentelemetry-collector-contrib:0.155.0	compose pin; --version
Docker Desktop	4.93.0 (240920), engine 29.8.1	docker version --format '{{.Server.Platform.Name}}'
VM kernel	7.0.14-linuxkit	uname -r inside a container
Ubuntu (ISO 8601)	noble-20260911 = 24.04, rsyslog 8.2312.0-3ubuntu9.4	Docker Hub API; apt-cache policy; dpkg -s rsyslog in the built image; rsyslogd -v
Ubuntu (RFC 3164)	jammy-20260901.2 = jammy, rsyslog 8.2112.0-2ubuntu2.5	same

Digests (Docker Hub API, 2026-10-01; the hardware is arm64):

Image	arm64	amd64
ubuntu:noble-20260911	sha256:11dc1ccb427f0464a2369e645454c272bb0baece7357c892ba69d313b3a332cf	sha256:496754492fb28b4d3049432f2ca787449331e23fb14f0dd3fffea86bf5a93eb4
ubuntu:jammy-20260901.2	sha256:1cc7bb38a74c0e126716646e47c0b3c5c139547d386d5ff7a64cf3ea316ca523	sha256:281c5745f657873d78e5531fc5ba8575f46ab7769b94550ac99543f122679986
mysql:8.4.11	sha256:ca3f0494c0f1fc86eb45f5e4786a1bb9f64d2f85b562cc74a9595046f6519a42	sha256:80f4933e3835f9dc4d35a28ec500d7986cb4414e6c6821c5461239cb7beb8995

ClickStack's collector runs as root (ps in the container), so the 0640 syslog:adm and mysql files were readable.

Syslog sample (the fixtures' /var/log/syslog)

2026-10-01T06:10:45.086579+00:00 ubuntu-noble fixture[11]: ubuntu-noble heartbeat 1
2026-10-01T06:10:45.087475+00:00 ubuntu-noble fixture-nopid: ubuntu-noble heartbeat 1 without a pid
2026-10-01T06:10:48.090802+00:00 ubuntu-noble fixture-nopid: ubuntu-noble heartbeat 2 without a pid

Oct  1 06:12:46 ubuntu-jammy fixture[25]: ubuntu-jammy heartbeat 1
Oct  1 06:12:46 ubuntu-jammy fixture-nopid: ubuntu-jammy heartbeat 1 without a pid
Oct  1 06:12:47 ubuntu-jammy rsyslogd: rsyslogd's groupid changed to 101

Pre-validation of the parsers (credential-free)

A throwaway docker run --rm --entrypoint /otelcontribcol of the 2.39.1 image: the profile's filelog block copied by yq with only start_at: beginning and the include path changed, a bind-mounted test file, a debug exporter with verbosity: detailed, no / mount, the running stack untouched. (Validating against the running collector's own configuration needs its environment; I did not do that.)

linux-host, seven test lines:

Line	Timestamp out (UTC)	unit
2026-10-01T04:13:39.637057+00:00 …	04:13:39.637057	fixture
2026-10-01T13:13:41.100000+09:00 …	04:13:41.1	fixture-offset
2026-10-01T04:13:42Z … (no fraction)	04:13:42	fixture-z
Oct 1 04:14:15 … (RFC 3164)	2026-10-01 04:14:15 (year inferred)	fixture
Oct 1 04:14:21 … last message repeated 3 times	1970 (unparsed, on_error: send)	none

The same file through the pre-change profile: no unit and Timestamp 1970 on every ISO line, jammy lines parsed.

mysql filter: a test mysql-slow.log with the header block followed by two # Time: entries, plus an error.log containing a line with Version: mid-line. Result: exactly two slow-log records (both parsed: query_time, rows_examined, Timestamp from the entry), both error.log lines untouched. Without the operator the same file gives three slow-log records, the first being the unparsed header with Timestamp 1970. The expression attributes["log.file.name"] == "mysql-slow.log" and body matches "^\\S+, Version: " compiled and ran without error. The regex is written with two backslashes (\\S) because the expression's string literal unescapes them to \S.

bin/check.sh receiver check (linux-host alone)

PASS  collector health
PASS  collector internal telemetry (2 receiver series)
PASS  receiver 'hostmetrics/linux-host' is live

bin/verify.sh linux-host

(SQL comment echo trimmed; run after the noble fixture and then the jammy fixture had written for several 30 s collection intervals.)

=== statement 1
   ┌─host─────────┬─metric_names─┬─points─┬──────────────newest─┐
1. │ 55abe26c9d1a │            6 │    752 │ 2026-10-01 06:14:21 │
   └──────────────┴──────────────┴────────┴─────────────────────┘

=== statement 2
    ┌─MetricName─────────────────────┬─MetricUnit────┬─points─┐
 1. │ system.cpu.load_average.15m    │ {thread}      │      8 │
 2. │ system.cpu.load_average.1m     │ {thread}      │      8 │
 3. │ system.cpu.load_average.5m     │ {thread}      │      8 │
 4. │ system.cpu.time                │ s             │    768 │
 5. │ system.cpu.utilization         │ 1             │    672 │
 6. │ system.disk.io                 │ By            │     48 │
 7. │ system.disk.io_time            │ s             │     24 │
 8. │ system.disk.merged             │ {operations}  │     48 │
 9. │ system.disk.operation_time     │ s             │     48 │
10. │ system.disk.operations         │ {operations}  │     48 │
11. │ system.disk.pending_operations │ {operations}  │     24 │
12. │ system.disk.weighted_io_time   │ s             │     24 │
13. │ system.filesystem.inodes.usage │ {inodes}      │     64 │
14. │ system.filesystem.usage        │ By            │     96 │
15. │ system.memory.usage            │ By            │     48 │
16. │ system.memory.utilization      │ 1             │     48 │
17. │ system.network.connections     │ {connections} │     96 │
18. │ system.network.dropped         │ {packets}     │    176 │
19. │ system.network.errors          │ {errors}      │    176 │
20. │ system.network.io              │ By            │    176 │
21. │ system.network.packets         │ {packets}     │    176 │
22. │ system.paging.faults           │ {faults}      │     16 │
23. │ system.paging.operations       │ {operations}  │     32 │
24. │ system.paging.usage            │ By            │     16 │
25. │ system.processes.count         │ {processes}   │     32 │
26. │ system.processes.created       │ {processes}   │      8 │
27. │ system.uptime                  │ s             │      8 │
    └────────────────────────────────┴───────────────┴────────┘

=== statement 3
   ┌─host─────────┬─unit──────────┬─lines─┬────────────────────────newest─┐
1. │ 55abe26c9d1a │ fixture-nopid │    70 │ 2026-10-01 06:14:28.000000000 │
2. │ 55abe26c9d1a │ fixture       │    70 │ 2026-10-01 06:14:28.000000000 │
3. │ 55abe26c9d1a │ rsyslogd      │     3 │ 2026-10-01 06:12:47.000000000 │
   └──────────────┴───────────────┴───────┴───────────────────────────────┘

linux-host: 3 statements ran. An empty result means nothing was ingested --
that is a failed verification, not a pass.

Per fixture host (ad hoc, not added to verify.sql): does Timestamp equal the line's own ts, rather than the ingestion time? ISO within 2 s (sample rows are exact to the microsecond); RFC 3164 exact to the second, comparing formatDateTime(Timestamp, '%b %e %H:%i:%S') with ts.

   ┌─fixture_host─┬─log_rows─┬─with_unit─┬─timestamp_matches_line_ts─┬────────────────────────newest─┐
1. │ ubuntu-jammy │      103 │       103 │                       103 │ 2026-10-01 06:15:13.000000000 │
2. │ ubuntu-noble │       70 │        70 │                        70 │ 2026-10-01 06:12:33.312786000 │
   └──────────────┴──────────┴───────────┴───────────────────────────┴───────────────────────────────┘

   ┌─fixture_host─┬─line_ts──────────────────────────┬─────────────────────Timestamp─┬─unit────┬─Body──────────────────────────────────────────────────────────────────────────────────┐
1. │ ubuntu-jammy │ Oct  1 06:15:13                  │ 2026-10-01 06:15:13.000000000 │ fixture │ Oct  1 06:15:13 ubuntu-jammy fixture[192]: ubuntu-jammy heartbeat 50                  │
2. │ ubuntu-jammy │ Oct  1 06:15:10                  │ 2026-10-01 06:15:10.000000000 │ fixture │ Oct  1 06:15:10 ubuntu-jammy fixture[189]: ubuntu-jammy heartbeat 49                  │
3. │ ubuntu-noble │ 2026-10-01T06:12:33.311463+00:00 │ 2026-10-01 06:12:33.311463000 │ fixture │ 2026-10-01T06:12:33.311463+00:00 ubuntu-noble fixture[133]: ubuntu-noble heartbeat 37 │
4. │ ubuntu-noble │ 2026-10-01T06:12:30.305315+00:00 │ 2026-10-01 06:12:30.305315000 │ fixture │ 2026-10-01T06:12:30.305315+00:00 ubuntu-noble fixture[130]: ubuntu-noble heartbeat 36 │
   └──────────────┴──────────────────────────────────┴───────────────────────────────┴─────────┴───────────────────────────────────────────────────────────────────────────────────────┘

bin/verify.sh mysql: failing side, then passing side

Failing side, before the filter operator: the collector was recreated first and MySQL started after it, so the slow log was read from the top, including mysqld's own header written by each of its starts. Statement 4 of that run:

=== statement 4
   ┌─parsed_slow_entries─┬─unparsed_slow_lines─┐
1. │                   4 │                   5 │
   └─────────────────────┴─────────────────────┘

An ad hoc breakdown of those rows: parsed_slow_entries=4, unparsed_slow_lines=5, of_which_mysqld_banner=5, of_which_other=0.

Passing side, with the filter operator. clickstack was recreated with the rebuilt mysql linux-host config; the five earlier unparsed rows are still in otel_logs (nothing was deleted) but their newest Timestamp is 06:17:35, 0 of them inside the 15-minute window. Then mysqld was restarted while the collector was reading, and SELECT SLEEP(2), (3) and (2.5) were run as otel_monitor. grep -c 'Version:' /var/log/mysql/mysql-slow.log inside the mysql container: 3 before the restart, 4 after (a new header was written while the collector was running).

=== statement 1
   ┌─instance──────────────────┬─metric_names─┬─points─┬──────────────newest─┐
1. │ host.docker.internal:3306 │           20 │   1014 │ 2026-10-01 15:08:55 │
   └───────────────────────────┴──────────────┴────────┴─────────────────────┘

=== statement 2
   ┌─db_system─┬─points─┐
1. │ mysql     │   1014 │
   └───────────┴────────┘

=== statement 3
   ┌─file───────────┬─lines─┬────────────────────────newest─┐
1. │ error.log      │    11 │ 2026-10-01 15:08:44.979813000 │
2. │ mysql-slow.log │     3 │ 2026-10-01 15:09:04.188150000 │
   └────────────────┴───────┴───────────────────────────────┘

=== statement 4
   ┌─parsed_slow_entries─┬─unparsed_slow_lines─┐
1. │                   3 │                   0 │
   └─────────────────────┴─────────────────────┘

mysql: 4 statements ran. An empty result means nothing was ingested --
that is a failed verification, not a pass.

Ad hoc: the header never reached otel_logs, and the restart is in error.log:

   ┌─header_rows_in_otel_logs_last_15_min─┬─slow_log_rows─┐
1. │                                    0 │             3 │
   └──────────────────────────────────────┴───────────────┘
    ┌─────────────────────Timestamp─┬─sev─────┬─code──────┬─msg────────────────────────────────────────────────────────────────────────────────────────┐
 1. │ 2026-10-01 15:08:40.842643000 │ System  │ MY-013172 │ Received SHUTDOWN from user <via user signal>. Shutting down mysqld (Version: 8.4.11).     │
 2. │ 2026-10-01 15:08:42.845996000 │ Warning │ MY-010909 │ /usr/sbin/mysqld: Forcing close of thread 13  user: 'otel_monitor'.                        │
 3. │ 2026-10-01 15:08:44.424725000 │ System  │ MY-015015 │ MySQL Server - start.                                                                      │
 4. │ 2026-10-01 15:08:44.660994000 │ System  │ MY-010116 │ /usr/sbin/mysqld (mysqld 8.4.11) starting as process 1                                     │
 5. │ 2026-10-01 15:08:44.667153000 │ System  │ MY-013576 │ InnoDB initialization has started.                                                         │
 6. │ 2026-10-01 15:08:44.825645000 │ System  │ MY-013577 │ InnoDB initialization has ended.                                                           │
 7. │ 2026-10-01 15:08:44.962295000 │ Warning │ MY-010068 │ CA certificate ca.pem is self signed.                                                      │
 8. │ 2026-10-01 15:08:44.962319000 │ System  │ MY-013602 │ Channel mysql_main configured to support TLS. Encrypted connections are now supported for  │
 9. │ 2026-10-01 15:08:44.963737000 │ Warning │ MY-011810 │ Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessibl │
10. │ 2026-10-01 15:08:44.979727000 │ System  │ MY-011323 │ X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/run/mysqld/my │
11. │ 2026-10-01 15:08:44.979813000 │ System  │ MY-010931 │ /usr/sbin/mysqld: ready for connections. Version: '8.4.11'  socket: '/var/run/mysqld/mysql │
    └───────────────────────────────┴─────────┴───────────┴────────────────────────────────────────────────────────────────────────────────────────────┘

Collector facts, at the pinned tag

• mysql is in otel/opentelemetry-collector-contrib:0.155.0 components (metrics Beta, logs Development), and not in ClickStack's components. Read with docker run --rm otel/opentelemetry-collector-contrib:0.155.0 components.
• receiver/mysqlreceiver/metadata.yaml @ v0.155.0: one resource attribute, mysql.instance.endpoint; server.address, server.port and service.instance.id appear nowhere. Query 1's mysql.instance.endpoint is host.docker.internal:3306, and the resource attribute keys on the mysql.* points are db.system.name, deploy.platform, host.name, mysql.instance.endpoint, os.type, service.name; all but mysql.instance.endpoint are added by the sidecar's processors.
• db.system.name appears in that metadata.yaml only as an attribute of the log events db.server.query_sample and db.server.top_query, both enabled: false and not used by this profile. mysql/metrics.md and a sidecar.config.yaml comment said it was "added in a later release"; reworded.
• filelog force_flush_period defaults to 500ms (reader.DefaultFlushPeriod = 500 * time.Millisecond in pkg/stanza/fileconsumer/internal/reader; receiver/filelogreceiver/README.md). include_file_name (default true) sets log.file.name. No change needed.
• filter operator (pkg/stanza/docs/operators/filter.md, operator/transformer/filter/transformer.go @ v0.155.0): entries matching expr are dropped; drop_ratio defaults to 1. If the expression errors, the entry is dropped too, which is why the expression is guarded by the file name. time_parser uses time.ParseInLocation for gotime; if: takes an expr-lang expression (matches is the regex operator, expr v1.17.8).
• Aliases: each of filelog, hostmetrics, fluentforward, kubeletstats logs "<alias>" alias is deprecated; use "<name>" instead when the 2.39.1 collector starts with it; validate prints nothing. The contrib 0.155.0 sidecar logs the same for otlp and resourcedetection.

Sidecar

Run the documented way from otel-profiles/sidecar/ with --env-file ../../_base/.env plus a non-secret env file (CLICKSTACK_OTLP_ENDPOINT=host.docker.internal:4317, CLICKSTACK_OTLP_INSECURE=true, MYSQL_* for the otel_monitor local user, SIDECAR_HEALTH_PORT=13134). The ingestion key was read from _base/.env by Compose and not copied. The interpolation was proven first with dummy env files: with only HYPERDX_INGESTION_KEY set the key is used, an explicit CLICKSTACK_API_KEY wins, with neither it is empty.

• The 13133 collision is real. up with the default failed with Bind for 0.0.0.0:13133 failed: port is already allocated, because _base publishes 13133 for its own collector. Hence SIDECAR_HEALTH_PORT; the default stays 13133.
• Replica status. The otel_monitor user was created exactly as the README says. The sidecar logged, on every 30 s scrape, at info level: Failed to fetch replica status stats … Error 1227 (42000): Access denied; you need (at least one of) the SUPER, REPLICATION CLIENT privilege(s) for this operation. The other 20 mysql.* metric names still arrived. Prerequisites corrected; the grant was not added silently.
• IPv6 dial warnings. For about ten seconds after start the sidecar logged grpc: addrConn.createTransport failed to connect to {Addr: "[fdc4:f303:9324::254]:4317", ServerName: "host.docker.internal:4317"} … network is unreachable (five warnings): host.docker.internal resolves to an IPv6 address that Docker Desktop's VM cannot reach. Metrics reached ClickHouse over IPv4 anyway.
• No host.id. The sidecar logs failed to get host ID from resourcedetection/common, so its metrics have no host.id resource attribute. host.name is present.

Findings worth knowing

• host.name on the linux-host metric rows is the collector container's own hostname, not the VM's: resourcedetection runs inside the container. The values are the VM's.
• The package-default $RepeatedMsgReduction on turns repeated identical messages into last message repeated N times, which has no unit and does not match the regex. The fixture's logger lines carry a counter so they are never repeated.
• The first fixture created the MySQL user from a .sh init fragment. On a Docker Desktop bind mount the entrypoint took its "running" branch for a mode-644 file and failed with Permission denied, with restart: unless-stopped restarting into a half-initialised data directory. The user is now created through MYSQL_USER/MYSQL_PASSWORD plus a .sql grant, and the service is restart: "no".
• Switching from the noble to the jammy fixture rotates /var/log/syslog aside (the two releases' syslog users have different uids), and no lines were lost in the switch.

How it was run

• Local OSS stack from _base/docker-compose.yml plus the new override, clickstack-all-in-one:2.39.1.
• ClickHouse was published on 18123/19000 through an uncommitted port override file (another local stack held 8123/9000), as for _base/bin/verify.sh: run it end to end once and record what it ran on #52; nothing else differed from the README's commands. CH_URL=http://localhost:18123 CH_USER=api CH_PASSWORD=api for verify.sh.
• Keys stay in _base/.env; none is in this PR, its commits, or any file created for it. gitleaks over the branch: no leaks.

Not run

• A replica with REPLICATION CLIENT granted.
• .env.example not updated: not readable by the agent. SIDECAR_HEALTH_PORT and MYSQL_MONITOR_PASSWORD are documented in compose comments and the READMEs only.
• Any physical or cloud Linux host; aws-rds-mysql; a MySQL 5.7 variant; other Ubuntu or MySQL releases; amd64 builds of the fixture (this machine is arm64).
• Whether FLUSH SLOW LOGS also writes the header (a restart does).
• A non-root collector reading the 0640 files.
• STATUS.md (left to the lead).

Verify commands

./otel-profiles/bin/lint.sh                      -> OK: 7 profiles follow the conventions
./otel-profiles/bin/build-config.sh linux-host gpu-nvidia baremetal-node virt-kvm mysql > /dev/null   -> rc 0
./otel-profiles/bin/build-config.sh --tier b virt-vsphere mysql aws-rds-mysql > /dev/null             -> rc 0
python3 .github/scripts/check_links.py           -> OK: every relative link in 42 markdown files resolves
./.github/scripts/check_syntax.sh                -> OK: all files parse
gitleaks detect --no-banner --redact -v --log-opts="verify-sh-e2e..HEAD"   -> 1 commit scanned, no leaks found

🤖 Generated with Claude Code