Skip to content

Show bare Flare servers with their metrics on the dashboard - #395

Merged
FujitsuPolycom merged 3 commits into
mainfrom
fix/dashboard-flare-metrics-discovery
Nov 2, 2026
Merged

FujitsuPolycom merged 3 commits into
mainfrom
fix/dashboard-flare-metrics-discovery

Conversation

@FujitsuPolycom

@FujitsuPolycom FujitsuPolycom commented Nov 2, 2026 •

Copy link
Copy Markdown
Owner

What and why

A flare run server started outside SparkRing was invisible on the dashboard, its metrics could be unreadable even when it answered, and its card could name launch facts nobody had observed. Three changes, one per commit:

Discovery. The host agent already reads the bounded argv of every GPU compute process, but nothing turned a flare/flare-<hex> run invocation into a dashboard member. A server started through Orbit, a manual launch, a systemd unit or a shell wrapper had no card, no ranks and no counters. runtime/host/node_containers.py now parses those launches and publishes only rank, rendezvous, HTTP bind/port and the served alias; flare agent and flare orbit are not ranks, and containerized servers are read from their entrypoint. Ranks sharing a head Spark and rendezvous port form one group without containers, a Flare process without a rendezvous is a one-Spark group, and an absent rank zero is named "head Spark not reporting" instead of counted as complete. Only rank zero's API port is read. Flare's Prometheus series map onto the shared counter contract, so a discovered server gets the same tiles, history and rates as an installed model; a loopback-bound launch keeps its process facts and says its counters cannot be read from another Spark.

Metrics. The collector sent one fixed Accept header for /metrics and the Flare parser read only Prometheus text, so a Flare build that answers /metrics with JSON produced a Serving card with blank figures and no explanation, and a listed endpoint could keep an output rate computed from counters the response no longer carried. The client now sends Accept: text/plain for /metrics and Accept: application/json for /v1/models and /v1/sparkring/status; allowed hosts, exact read paths, bodyless GETs, redirect refusal and response bounds are unchanged. parse_flare_metrics accepts Prometheus text or a JSON object, mapping canonical flare_* series and Helios snapshot fields through one dictionary and expanding snapshot page counts through page_tokens; fields the JSON does not publish stay unknown. parse_flare_as_metrics reports metrics_problem when a response carries no recognized numeric series, the page shows Metrics unavailable with the reason on both card types and clears it after a usable response, and the endpoint stays Serving because it answered.

Launch facts. The unmanaged-card renderer printed a placeholder for every fact it lacked ("unknown" for an engine, image id, served name, master address or API port; "?" for a rank or node count), so a GPU-in-use card named facts nobody had observed. unmanagedFacts now emits only facts the observation row or argv supplied: a GPU compute job shows its container and image alone, and a Flare process lists engine, served name, rank, master and API only when declared.

Validation

Run locally on this branch, not on Spark hardware.

CI run 41368875984 (head 189d9cfb) completed with 9 of 10 jobs green; the tests job failed at runtime/host/test_spread.py:793 in a timing-sensitive fabric-spread test this branch does not touch — the file is byte-identical to main, the same module flaked on main in run 38567342145, and 12 of 12 local repeats of that test passed. The branch was then rebased onto c78b48dc and pushed, which starts a fresh run. No later CI result is claimed.

  • python -m ruff check runtime/host/dashboard.py runtime/host/node_containers.py runtime/host/test_dashboard.py runtime/host/test_dashboard_charts.py runtime/host/test_dashboard_external.py runtime/host/test_flare_dashboard.py — all checks passed.
  • python -m pytest runtime/host/test_dashboard_charts.py runtime/host/test_dashboard_compact.py runtime/host/test_dashboard_layout.py runtime/host/test_dashboard_ring_labels.py runtime/host/test_dashboard_ring_totals.py runtime/host/test_dashboard_shared_style.py runtime/host/test_dashboard_spark_details.py runtime/host/test_dashboard_startup.py runtime/host/test_flare_dashboard.py -q -rs (Windows, Chromium through Playwright) — 131 passed, 0 failed, 0 skipped.
  • python3 -m pytest runtime/host/test_dashboard*.py runtime/host/test_flare_dashboard.py -q -rs (WSL/Linux) — 371 passed, 86 skipped. Skips: browser cases because that environment has no Playwright, one optional lab-sample comparison, one opt-in local-root/systemd lifecycle test.
  • python scripts/check_markdown_links.py . — 4232 repository-relative links; python scripts/check_repository_layout.py — 50 profiles, 54 exports, 69 catalogued enhancements, 430 preserved inputs, 22 locked profile assets, 1202 Python sources, 33 builders.

The branch was rebased onto c78b48dc before these runs; git range-diff shows all three commits unchanged by the rebase.

Compatibility or evidence

  • The status document gains metrics_problem, null when the last metrics read was usable; the page and its tests read it.
  • An unmanaged container whose engine is not recognized shows its container and image without the word "unknown".
  • The discovery sentinel {"engine": "unknown"} stays in node_containers.py, because grouping, the terminal view and their tests key off it; only the page stops presenting it as an observed fact.
  • Nothing here is managed: no automatic recovery, install record, stop or restart action. No request leaves the three read paths and no new host is admitted.

Limitations:

  • Flare's argv carries no world size, so the expected rank count follows the highest observed rank and higher unused ranks are not detected.
  • Bridged Docker port translation and launches described only by environment variables or an opaque entrypoint are not inferred.
  • Only the documented Flare series and the snapshot fields listed in FLARE_JSON_KEYS are mapped; a valid JSON body in an unexpected schema is reported as unusable rather than interpreted.
  • The terminal view (runtime/host/dashboard_tui.py) still prints "unknown" placeholders for a non-GPU unmanaged container's missing facts; this change covers the page.
  • No Spark hardware, live Flare process or Docker daemon was used.

Commit provenance

  • 9b79bbd8 Discover bare Flare servers and show them on the dashboard is the operator's pre-existing discovery work, cherry-picked from branch local/flare-dashboard-discovery at 325242c3, with authorship and author date preserved. It was amended only to remove an unused import and to restate its message in the repository's problem/behaviour/validation/limitations form; its code otherwise matches that cherry-pick.
  • 9fefd3e3 Negotiate Flare's metrics formats and report an unusable response is the metrics-negotiation correction.
  • 01d2fcfe Leave out launch facts the dashboard could not read is the renderer omission rule.

FujitsuPolycom and others added 3 commits November 2, 2026 07:52
A `flare run` server started outside SparkRing, through Orbit, a manual
launch, a systemd unit or a shell wrapper, never reached the dashboard: no
card named its endpoint, its ranks or its counters, and its Spark appeared
only as a model-less group or nothing at all.

Behavior:

- runtime/host/node_containers.py parses `flare` and `flare-<hex> run`
  invocations from the bounded argv it already reads for GPU compute
  processes, and publishes only rank, rendezvous, HTTP bind/port and the
  served alias. `flare agent` and `flare orbit` are not ranks; package
  paths, trust stores and API-key paths stay out of the observation row.
- The containerized case is read from the entrypoint, so a Flare process
  inside Docker is discovered the same way.
- Ranks sharing a head Spark and rendezvous port form one group without
  containers; a Flare process without a rendezvous is a one-Spark group,
  and an absent rank zero is named "head Spark not reporting" rather than
  counted as complete.
- Flare's Prometheus series map onto the shared counter contract, so a
  discovered server gets the same tiles, history and rates as an installed
  model. A launch bound to loopback keeps its process facts and says that
  its counters cannot be read from another Spark.
- The page draws discovered Flare servers as the same orange-red cards and
  ring placements as listed Flare endpoints, with their process and launch
  facts.

Compatibility: host-agent observation rows gain Flare launch fields; the
read paths are unchanged. Nothing is managed: no automatic recovery, no
install record, and no stop or restart action.

Validation: runtime/host/test_flare_dashboard.py covers launch parsing,
container entrypoints, grouping, missing heads, loopback binds and the
counter adapter; test_dashboard_ring_observation.py and
test_dashboard_layout.py cover ring placement and layout. The branch head
passes 371 Linux host tests with 86 skipped and 131 Windows browser tests.
Not run on Spark hardware, against a live Flare process or with Docker.

Limitations: Flare's argv carries no world size, so the expected rank count
follows the highest observed rank and higher unused ranks are not detected.
Bridged Docker port translation and launches described only by environment
variables or an opaque entrypoint are not inferred.

Co-Authored-By: Claude Haiku 4.5 (1M context) <noreply@anthropic.com>
The collector asked for /metrics without regard to the response format it
accepts, and the Flare parser understood only Prometheus text. A Flare
build that answers its /metrics with JSON therefore produced a Serving card
with blank figures and no explanation, and a listed endpoint could keep an
output rate computed from counters the response no longer carried.

Behavior:

- runtime/host/dashboard.py sends `Accept: text/plain` for `/metrics` and
  `Accept: application/json` for `/v1/models` and `/v1/sparkring/status`.
  The allowed hosts, exact read paths, bodyless GETs, redirect refusal and
  response bounds are unchanged.
- `parse_flare_metrics` accepts Prometheus text or a JSON object. Canonical
  `flare_*` series and the Helios snapshot fields map through one
  series-to-short-name dictionary, and snapshot page counts become KV
  tokens through `page_tokens`. Fields a snapshot does not publish stay
  unknown rather than being reported as zero.
- `parse_flare_as_metrics` reports `metrics_problem` when a response carries
  no recognized numeric series. Installed and discovered Flare members and
  listed Flare endpoints publish it; the page shows a **Metrics
  unavailable** tag and the reason on both card types, and clears them after
  a usable response. The endpoint stays **Serving**, because it answered.
- An external endpoint whose token counters disappear resets its rate
  baseline, so its card stops showing a rate it can no longer compute.

Compatibility: the status document gains `metrics_problem`, null when the
last metrics read was usable, and the page and its tests read it. No request
leaves the three read paths and no new host is admitted.

Validation: runtime/host/test_dashboard.py asserts the actual Accept,
User-Agent, URL and bodyless GET for every permitted path;
test_dashboard_external.py covers canonical and Helios JSON, unusable
bodies, the card figures and recovery; test_dashboard_charts.py checks the
tag and the note in both card types. The branch head passes 371 Linux host
tests with 86 skipped and 131 Windows browser tests. Not run against a live
Flare server.

Limitations: only the documented Flare series and the snapshot fields
listed in `FLARE_JSON_KEYS` are mapped. Other JSON shapes, and content
negotiation beyond the explicit Accept header, are not attempted; a body
that is valid JSON in an unexpected schema is reported as unusable rather
than interpreted.
The unmanaged-card renderer printed a placeholder for every launch fact it
lacked: "unknown" for an engine, an image id, a served name, a master
address or an API port, and "?" for a rank or node count. A GPU-in-use card
therefore named facts nobody had observed, which the page forbids for the
Spark details and for these cards.

Behavior:

- `unmanagedFacts` emits only facts the observation row or the launch argv
  supplied, with the same `known` predicate `sparkDetails` uses: a missing
  engine, image id, served name, rank, node count, master or API port is
  left out instead of named.
- A GPU compute job renders its container identity (position, name, image,
  image id) and nothing else, the rule the terminal view applies too.
- A discovered Flare process lists `engine flare`, its served name and node
  rank only when its launch declared them, and its master and API only when
  declared; a peer rank keeps `headless`.
- The discovery sentinel `{"engine": "unknown"}` stays in
  `runtime/host/node_containers.py`, because grouping, the terminal view and
  their tests key off it. Only the page stops presenting it as an observed
  fact.

Compatibility: an unmanaged container whose engine is not recognized now
shows its container and image without the word "unknown". The status
document is unchanged.

Validation: runtime/host/test_dashboard_layout.py at 375 and 1400 px
asserts that a GPU-in-use card's launch facts contain no "unknown"; it and
the rest of the Windows browser subset pass (131 tests). The Linux host
suite passes 371 tests with 86 skipped. Ruff, `check_repository_layout.py`
and `check_markdown_links.py` pass. Not run on hardware.

Limitations: the terminal view (`runtime/host/dashboard_tui.py`) still
prints "unknown" placeholders for a non-GPU unmanaged container's missing
facts; this change covers the page.
@FujitsuPolycom
FujitsuPolycom force-pushed the fix/dashboard-flare-metrics-discovery branch from 189d9cf to 01d2fcf Compare November 2, 2026 13:59
@FujitsuPolycom
FujitsuPolycom marked this pull request as ready for review November 2, 2026 14:23
@FujitsuPolycom
FujitsuPolycom merged commit 7a7edcd into main Nov 2, 2026
10 checks passed
@FujitsuPolycom
FujitsuPolycom deleted the fix/dashboard-flare-metrics-discovery branch November 2, 2026 14:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant