Skip to content

feat(go): add gated OTLP metrics, reconcile events, and log bridge (#2148) - #2675

Open
MartinForReal wants to merge 2 commits into
kagent-dev:mainfrom
MartinForReal:issue/2148-go-observability-gaps
Open

feat(go): add gated OTLP metrics, reconcile events, and log bridge (#2148)#2675
MartinForReal wants to merge 2 commits into
kagent-dev:mainfrom
MartinForReal:issue/2148-go-observability-gaps

Conversation

@MartinForReal

@MartinForReal MartinForReal commented Sep 3, 2026

Copy link
Copy Markdown

Implements the Go-side observability gaps from #2148 as three independent, default-OFF additions so the Go runtime/controller expose the same OpenTelemetry signals as the Python runtime.

1. Go ADK — GenAI token-usage metrics (go/adk)

Configures a MeterProvider with a periodic OTLP metric exporter and emits gen_ai.client.token.usage (Int64 histogram, unit {token}) per LLM call.

  • Attribute set matches the Python runtime so one dashboard covers both runtimes: gen_ai.token.type (input/output), gen_ai.request.model, gen_ai.response.model, gen_ai.provider.name, gen_ai.agent.name.
  • The A2A executor records one input + one output observation per LLM call (prompt tokens; output = candidate + reasoning tokens), skipping streamed Partial events so chunked streams are not double-counted.
  • Provider/model labels resolved from agent config (incl. config_usage.ModelName); response model falls back to the request model.
  • Gated behind OTEL_METRICS_ENABLED (default OFF) and the standard OTEL_EXPORTER_OTLP_* endpoint/protocol resolution shared with traces.
  • Tests: adk/pkg/telemetry/metrics_test.go (recording via ManualReader, provider-name mapping) and adk/pkg/a2a/executor_metrics_test.go (token accounting + partial/nil guard).

2. Kubernetes Events on reconcile (go/core)

The discovery reconcilers surface lifecycle events via an optional EventRecorder, mirroring the agentharness-substrate-controller pattern:

  • RemoteMCPServer (internal/controller/remotemcpserver): Normal ToolsDiscovered, Warning ValidationFailed, Warning ReconcileFailed.
  • MCPServer (MCPServerTool role, internal/controller/mcpserver): Normal ToolsDiscovered, Warning ValidationFailed, Warning ReconcileFailed.

Note on Agent/ModelConfig: in current main those objects reconcile through the KRT-based status collections in go/core/v2/controller, not classic controller-runtime Reconcile installers, so they have no EventRecorder path to attach events to. The events gap is therefore implemented on the two remaining classic discovery controllers — the direct successors of the MCPServerTool/RemoteMCPServer roles named in the issue. Emission is a no-op when no recorder is wired (the default).

3. Controller log → OTLP bridge (go/core)

  • telemetry.InitLoggerProvider builds an OTLP LoggerProvider via autoexport (same resource attributes as traces) and sets it global.
  • telemetry.ControllerZapOpts additively tees the controller zap logger with an otelzap bridge core, preserving stdout.
  • Wired into cmd/controller-v2/main.go before the controller logger is built.
  • Gated behind OTEL_LOGGING_ENABLED (already plumbed via the Helm chart otel.logging.enabled); when unset both functions are no-ops and logs are byte-identical.

Validation

  • go build ./..., go vet on all changed packages, gofmt/goimports clean.
  • golangci-lint on changed packages: 0 issues.
  • go test passes for adk/pkg/a2a, adk/pkg/telemetry, adk/pkg/config, core/internal/telemetry, and core/internal/controller/{mcpserver,remotemcpserver} (incl. the new event tests).
    • pkg/session has two pre-existing Windows-only TempDir-cleanup failures (SQLite handle held open during cleanup); unrelated to this change.

Dependencies

Promotes otlplog, otlpmetric(grpc/http), otel/log, otel/metric, otel/sdk/metric to direct requires at the same versions already in the graph, and adds go.opentelemetry.io/contrib/bridges/otelzap v0.19.0. No version bumps.

Refs: #2148

Implements the Go observability gaps in kagent-dev#2148 as three independent,
default-OFF additions so the Go runtime/controller match the OpenTelemetry
capabilities of the Python runtime:

1. go/adk — GenAI token usage metrics. Configure a MeterProvider with a
   periodic OTLP metric exporter and emit gen_ai.client.token.usage (Int64
   histogram, unit {token}) with the same attribute set the Python runtime
   records (gen_ai.token.type input/output, gen_ai.request.model,
   gen_ai.response.model, gen_ai.provider.name, gen_ai.agent.name). The A2A
   executor records one input + one output observation per LLM call (prompt
   tokens; output = candidate + reasoning tokens), skipping streamed Partial
   events. Gated behind OTEL_METRICS_ENABLED.

2. go/core — Kubernetes Events on reconcile. The RemoteMCPServer and
   MCPServer (MCPServerTool) discovery reconcilers emit Normal
   ToolsDiscovered and Warning ValidationFailed / Warning ReconcileFailed
   via an optional EventRecorder. Emission is skipped when no recorder is
   wired (the default), so behavior is byte-identical to today.

3. go/core — controller log -> OTLP bridge. InitLoggerProvider builds an
   OTLP LoggerProvider via autoexport (same resource as traces) and
   ControllerZapOpts tees the controller zap logger with an otelzap bridge
   core while preserving stdout. Gated behind OTEL_LOGGING_ENABLED; when
   unset both functions are no-ops.

Promotes the OTLP metric and log SDK dependencies to direct requires (same
versions already in the graph) and adds the otelzap bridge dependency.

Refs: kagent-dev#2148
Signed-off-by: MartinForReal <fanshangxiang@gmail.com>
@MartinForReal
MartinForReal requested review from a team and supreme-gg-gg as code owners September 3, 2026 06:54
@github-actions github-actions Bot added the enhancement New feature or request label Sep 3, 2026
Signed-off-by: MartinForReal <fanshangxiang@gmail.com>
@github-actions github-actions Bot added enhancement New feature or request and removed enhancement New feature or request labels Sep 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant