Skip to content

Standardise an opt-in SSE progress-streaming response (evaluate, chat, and beyond) #43

Description

@m-messer

The requirement

Lambda Feedback requires SSE streaming, but /evaluate and /chat only document 200 → application/json. ChatCapabilities.supportsStreaming already exists on /chat/health, and LLMConfiguration.stream exists on the request side, so streaming is acknowledged as in scope, but the spec doesn't define what a streaming response actually looks like. Without that, every implementation invents its own frame format and clients have nothing to code against.

(A separate PR already extends callbackUrl/202 Accepted to /chat, so this issue treats async delivery and SSE as parallel, complementary transport options rather than something to resolve here.)

Why this needs to be in the spec

  • /chat and /evaluate need to stream identically, and future endpoints shouldn't have to reinvent it.
  • supportsStreaming and LLMConfiguration.stream are meaningless without a documented contract behind them.
  • A streaming contract is largely generic across operations. The endpoint-specific parts are the terminal payload shape (Feedback[] for evaluate, ChatResponse for chat) and any progress-stage vocabulary; everything else (the opt-in mechanism, keep-alives, error delivery) can be defined once.

Requirements

  1. Opt-in via Accept: text/event-stream. Any operation may support it. When set, the response is a sequence of SseProgressStep frames, optional keep-alive comments, and exactly one terminal frame. X-Request-Id and X-Api-Version are set once, as SSE headers, at stream open, since there's no per-frame HTTP header mechanism.
  2. Shared progress schema. One SseProgressStep type (stage, message, timestamp), reused by both endpoints. Stage values are informative, not a strict enum, so implementations can add stages without a spec change; each service's /health response advertises its own supported stage values, the same way supportedArtefactProfiles advertises formats.
  3. Terminal frame reuses the existing success/error shape. On success, the terminal frame's payload is exactly the operation's normal 200 body (Feedback[] for /evaluate, ChatResponse for /chat), composed via allOf against each operation's existing 200 schema rather than duplicated. On failure, the payload is the existing ErrorResponse schema, so SSE errors carry the same title/code/details shape clients already handle for 400/500/etc.
  4. HTTP status stays 200 for the life of the stream. Failures are always delivered as a terminal ErrorResponse frame, not an HTTP error status.
  5. configuration.llm.stream is distinct from Accept: text/event-stream. The former, if kept, governs token-level streaming from the LLM provider; the latter governs step-level progress streaming at the API layer. The spec should say so explicitly to avoid the two being conflated.
  6. supportsStreaming moves to a shared capability shape. Currently only ChatCapabilities declares it; EvaluateCapabilities should gain the same field so /evaluate/health can advertise streaming support too.
  7. Per operation, add text/event-stream to the existing 200 response with a $ref to the relevant terminal frame. New endpoints get streaming for free.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions