Skip to content

feat: v1alpha1 protobuf contract + native gRPC - #251

Draft
krisztianfekete wants to merge 9 commits into
mainfrom
feat/api-v1alpha1
Draft

krisztianfekete wants to merge 9 commits into
mainfrom
feat/api-v1alpha1

Conversation

@krisztianfekete

Copy link
Copy Markdown
Contributor

Typed API: v1alpha1 protobuf contract + native gRPC

Adds a typed contract for agentevals (proto/agentevals/api/v1alpha1) and serves it over native gRPC on port 8002, next to the existing REST API. The server now builds session and turn views itself (tool calls paired with their results, token counts only when recorded, coded gaps), and the UI live view and annotation queue just render them.

What's in it:

  • Contract: generated Go (gen/go, tagged gen/go/vX.Y.Z on release), Python and TypeScript code is committed. CI runs buf lint, breaking checks and a drift check.
  • gRPC services: Session (List, Get, Watch), Evaluation (ListMetrics, streaming EvaluateTraces), Run (Submit, Get, List, Cancel, ListResults, WatchRun), Conversion and System. Health checking is built in; reflection and server traces are opt in. Server traces come from the upstream OTel gRPC instrumentation and are dropped if they're sent back to agentevals.
  • One evaluation core shared by REST and gRPC. A failing evaluation no longer hangs the SSE stream.
  • UI: much less client side logic, eslint is clean and now runs in CI.
  • Fixes: run reads are redacted, webhook logs no longer print URL tokens or response bodies, Kubernetes resolver errors no longer list secret keys, recorded eval runs no longer store server temp paths.
  • Postgres: unscored invocations keep their position, and runs whose lease expired on the last attempt no longer stay running forever.
  • Helm and Docker: grpc-api Service port, grpcApi.{enabled,reflection,grpcReadinessProbe}, EXPOSE 8002. New docs in docs/grpc-api.md.
  • Tests: about 340 new tests (1607 to 1950), plus live Postgres tests behind AGENTEVALS_TEST_DATABASE_URL.

⚠️ Breaking changes

  • /stream/ui-updates now sends protojson WatchSessionsResponse frames: every current session, then synced, then upserts and removals. The old per event messages (span_received, user_input, tool_call, token_update, ...) are gone.
  • GET /api/streaming/sessions no longer returns invocations. Use GetSession or WatchSessions with turns.
  • New port 8002, on by default and bound to --host (0.0.0.0 unless set). It has no authentication. Turn it off with --api-grpc-port 0 / AGENTEVALS_API_GRPC_PORT=0, or grpcApi.enabled=false in the chart.
  • Run reads are redacted: GET /api/runs*, cancel and the 409 body no longer return header values, secret like keys, full sink URLs or credentials in base_url. Stored runs keep the real values for the worker.
  • HTTP run targets: trace_id must be 16 or 32 hex characters, otherwise you get a 422.
  • Python API:
    • Usage token fields are None when not recorded (they used to be 0).
    • Turn.warnings is a list of Gap(code, reason) instead of strings.
    • Custom RoutingPolicy implementations must add is_own_span.
    • The LiveManager broadcast and SSE client API is replaced by subscribe() / unsubscribe() watchers.
  • New required dependencies: grpcio>=1.78, protobuf>=6.33.5,<7, grpcio-health-checking, grpcio-reflection, grpcio-status, opentelemetry-instrumentation-grpc.

Tested

Unit and integration suites (also on Python 3.11), the live Postgres tests, a Helm install on kind with the gRPC readiness probe on and off, and live sessions from openai-agents and langchain + some manual testing.

…ignore rule, and fail proto-check on ignored generated files
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant