Skip to content

Expose controller fleet-health metrics in Prometheus format #183

Open

Description

@devmittal02

Problem

The controller's fleet state is only visible through the management UI. adminListen must be a loopback address and has no authentication of its own, so a metrics scraper (Prometheus, Grafana Alloy, an OpenTelemetry Collector) cannot reach it.

That makes it hard to alert on the things an operator of a managed fleet cares about:

  • devices that stopped checking in
  • devices whose last configuration apply failed
  • devices that are on an older configuration revision than the active one
  • the spread of agent versions during a rollout
  • whether telemetry events are still arriving

Usage reporting from #160 covers tokens and cost. It does not cover fleet health.

Proposal

An opt-in metricsListen controller setting. When set, the controller serves GET /metrics in the Prometheus text format on that address. Unset keeps today's behavior.

Suggested metrics (names are open to change):

  • agentdesktop_devices, agentdesktop_devices_online, agentdesktop_devices_connected
  • agentdesktop_device_oldest_seen_age_seconds
  • agentdesktop_devices_by_version{os,agent_version}
  • agentdesktop_devices_by_config_state{state} and agentdesktop_devices_config_behind
  • agentdesktop_config_active_revision
  • agentdesktop_inventory_mcp_servers, agentdesktop_inventory_skills
  • agentdesktop_telemetry_events_total{type}, agentdesktop_device_message_errors_total

Labels would come from closed sets or from the OS and agent version of enrolled devices. Hostnames, users, device IDs, MCP server names and tool inputs would never be exported.

Open questions

  • Are the metric names and the metricsListen setting the right shape, or would you rather expose this through an OpenTelemetry exporter?
  • Is a separate unauthenticated listener acceptable, or should the endpoint require a token like the usage ingest does?
  • With more than one controller replica, the connected gauge and the counters are per replica. Is that acceptable, or should connection state live in the database?

Implementation

A draft is up in #180. I'm opening this issue so the design can be discussed before the PR is reviewed in detail.

cc @peterj

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions