docs: add symptom-first troubleshooting guides - #1296
docs: add symptom-first troubleshooting guides#1296Jeff Luo (JeffLuoo) wants to merge 2 commits into
Conversation
|
I noticed several drifts between this guide and the metrics definition (like the error code etc) and I think it will become more programatic in the future because it will be very hard to keep the guide the metrics definition synced. I think we should make the guide very thin. The durable structure is: guide owns the decision tree and interpretation, registry owns definitions, and the guide links rather than paraphrases. Every value taxonomy or default it copies inline is future debt. |
The metric registry says what each instrument measures, and the observability guide says how telemetry is collected, but neither takes an operator from a symptom to a cause. A user reporting a slow request or a 503 had no path through the data. These three guides each start at a symptom and end at a component. Every step gives a query, says how to read the result, and names the next step or the guide to switch to. The shared index holds what all three need: the two spellings of the instrument names and which ingest path produces each, the four reasons a query returns nothing, and the pointer to the subsystems that emit no metrics at all. Every query was run against a live cluster, and the readings drove the text: the resume label is only meaningful next to the outcome, a fleet gauge that every replica reports needs max rather than sum, and a phase histogram has to exclude failures before its quantile means anything.
The guides copied label taxonomies and flag defaults inline. A copy goes out of date without a signal, and one already had: the guide said that ate.imagecache.requests reports _OTHER for each status that is not 401, 403 or 429, but the registry and internal/imagecache/metrics.go also pass 404, 500, 502, 503 and 504 through as literal values. Give the guides one job. They own the decision tree: which query to run, how to read the result, and where to go next. The registry owns the definitions. Drop the value tables for ate.snapshot.kind, ate.failure.reason, ate.scheduling.constraint and the parking outcome, and link to the registry group instead. Drop the --parked-request-max default and link to request-parking.md, which owns the flag. Drop the histogram bucket ceilings and the two sample measurements, which the surrounding argument does not need. The phase, reason and outcome tables that remain hold routing and not definitions. Fix two drifts on the way: three guides said "three reasons" for an empty query where the README lists four, and capacity-is-full.md said "two states" above a three-row table.
77863cb to
d10fa0e
Compare
Agreed, and updated to remove some metrics details from the guide. |
2 participants
Fixes #1294
It's converted from this public doc. Please feel free to drop comments in the doc and I can update this PR.