Repository navigation
Show bare Flare servers with their metrics on the dashboard - #395
Merged
Merged
Conversation
A `flare run` server started outside SparkRing, through Orbit, a manual launch, a systemd unit or a shell wrapper, never reached the dashboard: no card named its endpoint, its ranks or its counters, and its Spark appeared only as a model-less group or nothing at all. Behavior: - runtime/host/node_containers.py parses `flare` and `flare-<hex> run` invocations from the bounded argv it already reads for GPU compute processes, and publishes only rank, rendezvous, HTTP bind/port and the served alias. `flare agent` and `flare orbit` are not ranks; package paths, trust stores and API-key paths stay out of the observation row. - The containerized case is read from the entrypoint, so a Flare process inside Docker is discovered the same way. - Ranks sharing a head Spark and rendezvous port form one group without containers; a Flare process without a rendezvous is a one-Spark group, and an absent rank zero is named "head Spark not reporting" rather than counted as complete. - Flare's Prometheus series map onto the shared counter contract, so a discovered server gets the same tiles, history and rates as an installed model. A launch bound to loopback keeps its process facts and says that its counters cannot be read from another Spark. - The page draws discovered Flare servers as the same orange-red cards and ring placements as listed Flare endpoints, with their process and launch facts. Compatibility: host-agent observation rows gain Flare launch fields; the read paths are unchanged. Nothing is managed: no automatic recovery, no install record, and no stop or restart action. Validation: runtime/host/test_flare_dashboard.py covers launch parsing, container entrypoints, grouping, missing heads, loopback binds and the counter adapter; test_dashboard_ring_observation.py and test_dashboard_layout.py cover ring placement and layout. The branch head passes 371 Linux host tests with 86 skipped and 131 Windows browser tests. Not run on Spark hardware, against a live Flare process or with Docker. Limitations: Flare's argv carries no world size, so the expected rank count follows the highest observed rank and higher unused ranks are not detected. Bridged Docker port translation and launches described only by environment variables or an opaque entrypoint are not inferred. Co-Authored-By: Claude Haiku 4.5 (1M context) <noreply@anthropic.com>
The collector asked for /metrics without regard to the response format it accepts, and the Flare parser understood only Prometheus text. A Flare build that answers its /metrics with JSON therefore produced a Serving card with blank figures and no explanation, and a listed endpoint could keep an output rate computed from counters the response no longer carried. Behavior: - runtime/host/dashboard.py sends `Accept: text/plain` for `/metrics` and `Accept: application/json` for `/v1/models` and `/v1/sparkring/status`. The allowed hosts, exact read paths, bodyless GETs, redirect refusal and response bounds are unchanged. - `parse_flare_metrics` accepts Prometheus text or a JSON object. Canonical `flare_*` series and the Helios snapshot fields map through one series-to-short-name dictionary, and snapshot page counts become KV tokens through `page_tokens`. Fields a snapshot does not publish stay unknown rather than being reported as zero. - `parse_flare_as_metrics` reports `metrics_problem` when a response carries no recognized numeric series. Installed and discovered Flare members and listed Flare endpoints publish it; the page shows a **Metrics unavailable** tag and the reason on both card types, and clears them after a usable response. The endpoint stays **Serving**, because it answered. - An external endpoint whose token counters disappear resets its rate baseline, so its card stops showing a rate it can no longer compute. Compatibility: the status document gains `metrics_problem`, null when the last metrics read was usable, and the page and its tests read it. No request leaves the three read paths and no new host is admitted. Validation: runtime/host/test_dashboard.py asserts the actual Accept, User-Agent, URL and bodyless GET for every permitted path; test_dashboard_external.py covers canonical and Helios JSON, unusable bodies, the card figures and recovery; test_dashboard_charts.py checks the tag and the note in both card types. The branch head passes 371 Linux host tests with 86 skipped and 131 Windows browser tests. Not run against a live Flare server. Limitations: only the documented Flare series and the snapshot fields listed in `FLARE_JSON_KEYS` are mapped. Other JSON shapes, and content negotiation beyond the explicit Accept header, are not attempted; a body that is valid JSON in an unexpected schema is reported as unusable rather than interpreted.
The unmanaged-card renderer printed a placeholder for every launch fact it
lacked: "unknown" for an engine, an image id, a served name, a master
address or an API port, and "?" for a rank or node count. A GPU-in-use card
therefore named facts nobody had observed, which the page forbids for the
Spark details and for these cards.
Behavior:
- `unmanagedFacts` emits only facts the observation row or the launch argv
supplied, with the same `known` predicate `sparkDetails` uses: a missing
engine, image id, served name, rank, node count, master or API port is
left out instead of named.
- A GPU compute job renders its container identity (position, name, image,
image id) and nothing else, the rule the terminal view applies too.
- A discovered Flare process lists `engine flare`, its served name and node
rank only when its launch declared them, and its master and API only when
declared; a peer rank keeps `headless`.
- The discovery sentinel `{"engine": "unknown"}` stays in
`runtime/host/node_containers.py`, because grouping, the terminal view and
their tests key off it. Only the page stops presenting it as an observed
fact.
Compatibility: an unmanaged container whose engine is not recognized now
shows its container and image without the word "unknown". The status
document is unchanged.
Validation: runtime/host/test_dashboard_layout.py at 375 and 1400 px
asserts that a GPU-in-use card's launch facts contain no "unknown"; it and
the rest of the Windows browser subset pass (131 tests). The Linux host
suite passes 371 tests with 86 skipped. Ruff, `check_repository_layout.py`
and `check_markdown_links.py` pass. Not run on hardware.
Limitations: the terminal view (`runtime/host/dashboard_tui.py`) still
prints "unknown" placeholders for a non-GPU unmanaged container's missing
facts; this change covers the page.
FujitsuPolycom
force-pushed
the
fix/dashboard-flare-metrics-discovery
branch
from
November 2, 2026 13:59
189d9cf to
01d2fcf
Compare
FujitsuPolycom
marked this pull request as ready for review
November 2, 2026 14:23
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
A
flare runserver started outside SparkRing was invisible on the dashboard, its metrics could be unreadable even when it answered, and its card could name launch facts nobody had observed. Three changes, one per commit:Discovery. The host agent already reads the bounded argv of every GPU compute process, but nothing turned a
flare/flare-<hex> runinvocation into a dashboard member. A server started through Orbit, a manual launch, a systemd unit or a shell wrapper had no card, no ranks and no counters.runtime/host/node_containers.pynow parses those launches and publishes only rank, rendezvous, HTTP bind/port and the served alias;flare agentandflare orbitare not ranks, and containerized servers are read from their entrypoint. Ranks sharing a head Spark and rendezvous port form one group without containers, a Flare process without a rendezvous is a one-Spark group, and an absent rank zero is named "head Spark not reporting" instead of counted as complete. Only rank zero's API port is read. Flare's Prometheus series map onto the shared counter contract, so a discovered server gets the same tiles, history and rates as an installed model; a loopback-bound launch keeps its process facts and says its counters cannot be read from another Spark.Metrics. The collector sent one fixed
Acceptheader for/metricsand the Flare parser read only Prometheus text, so a Flare build that answers/metricswith JSON produced a Serving card with blank figures and no explanation, and a listed endpoint could keep an output rate computed from counters the response no longer carried. The client now sendsAccept: text/plainfor/metricsandAccept: application/jsonfor/v1/modelsand/v1/sparkring/status; allowed hosts, exact read paths, bodyless GETs, redirect refusal and response bounds are unchanged.parse_flare_metricsaccepts Prometheus text or a JSON object, mapping canonicalflare_*series and Helios snapshot fields through one dictionary and expanding snapshot page counts throughpage_tokens; fields the JSON does not publish stay unknown.parse_flare_as_metricsreportsmetrics_problemwhen a response carries no recognized numeric series, the page shows Metrics unavailable with the reason on both card types and clears it after a usable response, and the endpoint stays Serving because it answered.Launch facts. The unmanaged-card renderer printed a placeholder for every fact it lacked ("unknown" for an engine, image id, served name, master address or API port; "?" for a rank or node count), so a GPU-in-use card named facts nobody had observed.
unmanagedFactsnow emits only facts the observation row or argv supplied: a GPU compute job shows its container and image alone, and a Flare process lists engine, served name, rank, master and API only when declared.Validation
Run locally on this branch, not on Spark hardware.
CI run
41368875984(head189d9cfb) completed with 9 of 10 jobs green; thetestsjob failed atruntime/host/test_spread.py:793in a timing-sensitive fabric-spread test this branch does not touch — the file is byte-identical tomain, the same module flaked onmainin run38567342145, and 12 of 12 local repeats of that test passed. The branch was then rebased ontoc78b48dcand pushed, which starts a fresh run. No later CI result is claimed.python -m ruff check runtime/host/dashboard.py runtime/host/node_containers.py runtime/host/test_dashboard.py runtime/host/test_dashboard_charts.py runtime/host/test_dashboard_external.py runtime/host/test_flare_dashboard.py— all checks passed.python -m pytest runtime/host/test_dashboard_charts.py runtime/host/test_dashboard_compact.py runtime/host/test_dashboard_layout.py runtime/host/test_dashboard_ring_labels.py runtime/host/test_dashboard_ring_totals.py runtime/host/test_dashboard_shared_style.py runtime/host/test_dashboard_spark_details.py runtime/host/test_dashboard_startup.py runtime/host/test_flare_dashboard.py -q -rs(Windows, Chromium through Playwright) — 131 passed, 0 failed, 0 skipped.python3 -m pytest runtime/host/test_dashboard*.py runtime/host/test_flare_dashboard.py -q -rs(WSL/Linux) — 371 passed, 86 skipped. Skips: browser cases because that environment has no Playwright, one optional lab-sample comparison, one opt-in local-root/systemd lifecycle test.python scripts/check_markdown_links.py .— 4232 repository-relative links;python scripts/check_repository_layout.py— 50 profiles, 54 exports, 69 catalogued enhancements, 430 preserved inputs, 22 locked profile assets, 1202 Python sources, 33 builders.The branch was rebased onto
c78b48dcbefore these runs;git range-diffshows all three commits unchanged by the rebase.Compatibility or evidence
metrics_problem, null when the last metrics read was usable; the page and its tests read it.{"engine": "unknown"}stays innode_containers.py, because grouping, the terminal view and their tests key off it; only the page stops presenting it as an observed fact.Limitations:
FLARE_JSON_KEYSare mapped; a valid JSON body in an unexpected schema is reported as unusable rather than interpreted.runtime/host/dashboard_tui.py) still prints "unknown" placeholders for a non-GPU unmanaged container's missing facts; this change covers the page.Commit provenance
9b79bbd8Discover bare Flare servers and show them on the dashboard is the operator's pre-existing discovery work, cherry-picked from branchlocal/flare-dashboard-discoveryat325242c3, with authorship and author date preserved. It was amended only to remove an unused import and to restate its message in the repository's problem/behaviour/validation/limitations form; its code otherwise matches that cherry-pick.9fefd3e3Negotiate Flare's metrics formats and report an unusable response is the metrics-negotiation correction.01d2fcfeLeave out launch facts the dashboard could not read is the renderer omission rule.