diff --git a/CONTEXT.md b/CONTEXT.md index 474224e7..c402c4f0 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -58,6 +58,21 @@ These three Microsoft offerings are distinct and must not all be called "Azure". ## Protocol terms +- **Model Turn** — one contiguous segment of model output. A model turn may end + while the dialog interaction remains active because the provider is still + reasoning or waiting for asynchronous tool results. + +- **Dialog Interaction** — processing initiated by user input that may span + multiple model turns and tool calls. It completes only when the provider reports + that no reasoning, generation, or tool work remains. + +- **Interaction In Progress** — a model-turn boundary at which the dialog + interaction remains active. More tool calls or model output may follow without + new user input. + +- **Interaction Idle** — the terminal state of a dialog interaction. No reasoning, + generation, or tool work remains, and new model output requires new client input. + - **Initial Start Message** — the first WebSocket message for a conversation. It either contains complete service parameters or declares that they will follow in a deferred params message. diff --git a/Cargo.toml b/Cargo.toml index e0611fea..ac78222c 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -25,7 +25,7 @@ members = [ ] [workspace.package] -version = "3.8.1" +version = "3.9.0" edition = "2024" license = "MIT" repository = "https://github.com/pragmatrix/context-switch" diff --git a/VERSIONING.md b/VERSIONING.md index 33eb9cff..fe63e5d8 100644 --- a/VERSIONING.md +++ b/VERSIONING.md @@ -19,5 +19,6 @@ root `Cargo.toml`); all member crates inherit it via `version.workspace = true`. | Version | Change | |---------|--------| +| 3.9.0 | google-dialog: Gemini 3.8 Live models and interaction semantics | | 3.8.0 | google-transcribe: numerals support via digit-sequence class token (#99) | | 3.8.1 | audio-knife: startup self-test warning for `GOOGLE_APPLICATION_CREDENTIALS` (#100) | diff --git a/docs/adr/0006-gemini-3-8-live-model-semantics.md b/docs/adr/0006-gemini-3-8-live-model-semantics.md new file mode 100644 index 00000000..666137e8 --- /dev/null +++ b/docs/adr/0006-gemini-3-8-live-model-semantics.md @@ -0,0 +1,524 @@ +# ADR 0006: Gemini 3.8 Live model semantics + +Date: 2026-09-18 +Status: Accepted + +## Context + +Google exposes two stable Gemini Developer API models with materially different +runtime contracts: + +- `gemini-3.8-live` is the default low-latency model. It rejects + `thinking_level`, supports blocking and non-blocking function calls, and + supports scheduling non-blocking function responses. +- `gemini-3.8-live-extended-thinking` performs background reasoning. It accepts + no thinking level or `low`, `medium`, or `high`; requires non-blocking + functions; rejects function-response scheduling; and may end a model turn + while the dialog interaction remains active. + +The existing Google dialog API accepts an open model string, maps every provider +`turnComplete` to the public terminal `TurnComplete` event, and depends on a +vendored raw-WebSocket client whose tool and lifecycle types predate these +contracts. Merely accepting the new model strings would therefore advertise +support while exposing incorrect lifecycle and tool behavior. + +The models were verified as stable in the Gemini Developer API. Their +availability through Gemini Enterprise Agent Platform, including EU data +residency, was not established by Google's published model-location +documentation. Paid Gemini API use is governed by Google's Data Processing +Addendum and Google states that paid prompts and responses are not used to +improve its products, but transient abuse-monitoring processing may occur in any +country where Google or its agents maintain facilities. These are dated +deployment facts, not properties guaranteed by this client. + +## Decision + +Support both models as production-ready direct Gemini API models. Keep +`Params.model` as an open `String`, add public `GEMINI_3_8_LIVE` and +`GEMINI_3_8_LIVE_EXTENDED_THINKING` constants, and do not add a model allowlist +or a separate extended-thinking flag. The direct API example defaults to +`GEMINI_3_8_LIVE`; Extended Thinking remains opt-in. Existing model IDs and +Agent Platform behavior remain supported. The 3.8 constants do not assert Agent +Platform availability. + +The public constants have these exact declarations and remain bare model IDs: + +```rust +pub const GEMINI_3_8_LIVE: &str = "gemini-3.8-live"; +pub const GEMINI_3_8_LIVE_EXTENDED_THINKING: &str = + "gemini-3.8-live-extended-thinking"; +``` + +The integration uses the domain distinction in `CONTEXT.md`: a wire +`turnComplete` closes a **Model Turn**, while only `interactionStatus: IDLE` +closes the **Dialog Interaction**. + +- Finalize the current output transcript at every wire `turnComplete`. +- Map `turnComplete` with `interactionStatus: IN_PROGRESS` to the new public + `ServiceOutputEvent::InteractionInProgress`. +- Map `turnComplete` with `interactionStatus: IDLE` to the existing terminal + `ServiceOutputEvent::TurnComplete`. +- Preserve legacy completion behavior when `interactionStatus` is absent. +- Preserve the status through the vendored client's wire and semantic event + types; do not infer interaction state from model names. + +The public output enum gains exactly one unit variant: + +```rust +pub enum ServiceOutputEvent { + // Existing variants remain unchanged. + InteractionInProgress, +} +``` + +Its serialized form is `{"type":"interactionInProgress"}`. `TurnComplete` +retains `{"type":"turnComplete"}` and remains the only terminal interaction +event. No provider status enum is exposed: `IN_PROGRESS` and `IDLE` are wire +states whose domain meanings are represented by the two service events. + +Validate model-specific setup before opening the WebSocket: + +- `gemini-3.8-live` requires `thinking_level` to be absent. +- `gemini-3.8-live-extended-thinking` permits an absent level or `low`, + `medium`, or `high`, and rejects `minimal`. +- When the level is absent for Extended Thinking, `google-dialog` sends + `medium` itself. Google's public model documentation does not specify a + fixed default level, so the service pins `medium` deterministically rather + than depending on an unspecified server default. +- Legacy models retain their existing setup behavior. + +For tools, declaration behavior and result scheduling are separate concepts. +The vendored protocol must model both `BLOCKING` and `NON_BLOCKING` declaration +behavior. Scheduling belongs on `FunctionResponse`, with typed values for +immediate interruption, delivery when idle, and silent context insertion; the +wire spelling must follow the current Google schema. + +The exact types are: + +```rust +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum FunctionBehavior { + Blocking, + NonBlocking, +} + +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum FunctionResponseScheduling { + Silent, + WhenIdle, + Interrupt, +} +``` + +These types are defined by `gemini-live` because they directly represent wire +fields, and are re-exported by `google-dialog` so callers do not need to name +the vendored crate. `FunctionDeclaration::behavior` remains an +`Option`. Its existing declaration-level `scheduling` field +remains only for legacy deserialization and is not re-exported as the response +scheduling type. + +`ServiceInputEvent::FunctionCallResult` changes to this exact shape: + +```rust +FunctionCallResult { + call_id: String, + output: serde_json::Value, + #[serde(default, skip_serializing_if = "Option::is_none")] + scheduling: Option, +}, +``` + +Omitting `scheduling` means no field is sent and preserves Google's original +function-response handling for backward compatibility. The serialized forms +are: + +```json +{"type":"functionCallResult","callId":"call-1","output":{"ok":true}} +``` + +```json +{"type":"functionCallResult","callId":"call-1","output":{"ok":true},"scheduling":"INTERRUPT"} +``` + +- Standard 3.8 permits blocking and non-blocking declarations and optional + scheduling on non-blocking function responses. +- Extended Thinking defaults omitted declaration behavior to `NON_BLOCKING`, + preserves explicit `NON_BLOCKING`, and rejects `BLOCKING`. +- Extended Thinking rejects scheduling both in declarations and function + responses rather than silently dropping it. +- Declaration-level scheduling remains readable for legacy compatibility but is + not used to represent the current 3.8 response contract. +- Function call IDs remain the correlation key, allowing multiple non-blocking + calls to be outstanding and results to arrive out of order. + +For standard 3.8, an explicit response schedule paired with a blocking +declaration is rejected rather than sent as an ignored field. For Extended +Thinking, any explicit response schedule is rejected. These checks happen when +the service event is handled because the call ID identifies the corresponding +declaration. Omitted scheduling is valid for every model. Unknown and legacy +models retain pass-through behavior, and response scheduling is supported on +both direct Gemini API and Agent Platform connections. + +Expose full-session incremental context as a provider input event named +`ClientContent` (the Google wire envelope is `clientContent`). It carries an +explicit `user` or `model` role, text content, and `turn_complete`. Keep the +existing realtime `Prompt` input unchanged. Sending client content with +`turn_complete: true` interrupts active generation; sending it without that flag +appends content and waits for further input. + +Callers pick between these text input paths by intent: + +- Realtime user input (the user speaks or types now): `Prompt`. +- An immediate, ordered response with interruption of active generation: + `ClientContent` with `user` role and `turn_complete: true`. +- Silent context the model uses only when generation is next triggered: + `ClientContent` with `user` role and `turn_complete: false`. +- Restoring or fabricating an earlier assistant turn: `ClientContent` with + `model` role. + +The exact public API is: + +```rust +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "lowercase")] +pub enum ClientContentRole { + User, + Model, +} + +pub enum ServiceInputEvent { + // Existing variants remain, with FunctionCallResult extended as above. + ClientContent { + role: ClientContentRole, + text: String, + #[serde(default)] + turn_complete: bool, + }, +} +``` + +The members have this contract: + +- `role: ClientContentRole` identifies the author of the history entry. `User` + serializes as `user` and `Model` as `model`. No other roles are accepted; + this prevents arbitrary role strings from reaching the Gemini wire protocol. +- `text: String` is the text of exactly one content part. It is required, is + sent as `clientContent.turns[0].parts[0].text`, and is not interpreted as + realtime input. Empty text is allowed because the event represents content, + not a prompt-validation policy. +- `turn_complete: bool` controls generation after the content is appended. The + default is `false`, which appends the content and leaves generation pending. + `true` asks Gemini to start generation immediately and intentionally interrupts + active generation. It serializes as `turnComplete`. + +The roles have distinct conversation semantics: + +- `User` means that `text` is user-authored context supplied by the client. It + represents something the end user said or wrote and is eligible to be used as + user input when Gemini generates the next response. +- `Model` means that `text` is model-authored context supplied by the client. It + represents an earlier assistant/model response that the client is restoring + or appending to conversation history; it does not claim that Gemini generated + this text during the current connection. + +These roles describe conversation authorship, not caller authorization, audio +direction, or the event source. A client must not label newly supplied user +input as `Model`, and the service must not infer a `Model` entry from a +realtime `Prompt`. `ClientContentRole` is provider-owned and restricts callers +to the two roles accepted by Google. One event maps to one +`clientContent.turns` entry with `role`, one text part, and the requested +completion flag. The service sends `turnComplete` explicitly, including when +it is `false`, so the resulting wire intent is unambiguous. Input-event JSON +uses the service tag and field names; the client converts it to the Gemini +envelope rather than forwarding this JSON directly: + +```json +{"type":"clientContent","role":"user","text":"Remember this.","turnComplete":false} +``` + +```json +{"type":"clientContent","role":"model","text":"Earlier answer.","turnComplete":true} +``` + +The corresponding Gemini wire message for the first example is: + +```json +{"clientContent":{"turns":[{"role":"user","parts":[{"text":"Remember this."}]}],"turnComplete":false}} +``` + +`ClientContent` is valid for both `User` and `Model` history entries, but it is +not a replacement for ordinary audio or realtime text input. The event is +provider-specific and is rejected by the service when the selected routing +cannot send direct Gemini Live `clientContent` messages. + +The existing `Prompt { text }` variant and its JSON remain unchanged. It maps to +realtime text input, is always user input, and cannot insert model history. + +Input transcription language hints are exposed on `Params`: + +```rust +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum TranscriptionMode { + Verbatim, + Smart, +} + +#[serde(default)] +pub input_audio_transcription_language_codes: Vec, +#[serde(default = "default_transcription_mode")] +/// Defaults to `VERBATIM`. +pub input_audio_transcription_mode: TranscriptionMode, +#[serde(default = "default_transcription_mode")] +/// Defaults to `VERBATIM`. +pub output_audio_transcription_mode: TranscriptionMode, +``` + +The field serializes as `inputAudioTranscriptionLanguageCodes` and maps to +Google's `inputAudioTranscription.languageCodes`. Values are BCP-47 language +codes used as input ASR hints, not a request to force the model's native audio +response language. The explicit `input_audio_transcription: bool` controls +whether input transcription is enabled; language codes and mode only configure +it when that flag is `true`. The language-code list defaults to empty, and an +empty list means automatic language detection. Both transcription mode fields +default to `VERBATIM`; callers select `SMART` explicitly. The output +transcription boolean likewise controls whether output transcription is +enabled; its mode only configures it when that flag is `true`. +The transcription mode fields map to the corresponding configuration's `mode`: +`inputAudioTranscription.mode` and `outputAudioTranscription.mode`. The public +default is `VERBATIM`; `SMART` removes disfluencies, performs light +grammatical cleanup, applies automatic formatting, and makes minor inline +corrections. `SMART` cannot be combined with word timestamps or diarization; +those controls remain unexposed. Output transcription has no corresponding +language-code parameter because native-audio output language selection is +automatic for these models. `google-dialog` therefore sends +`outputAudioTranscription.languageCodes` as absent and derives the output +transcript language from the generated audio; callers cannot use output +transcription configuration to select or override that language. + +`google-dialog` re-exports the complete new surface from its crate root: + +```rust +pub use types::{ + ClientContentRole, FunctionBehavior, FunctionResponseScheduling, + GEMINI_3_8_LIVE, GEMINI_3_8_LIVE_EXTENDED_THINKING, Params, + ServiceInputEvent, ServiceOutputEvent, VOICES, parse_voice_value, +}; +``` + +## 3.8 Live feature overview + +Beyond the wiring documented above, the two 3.8 Live models provide these +capabilities (verified against Google's model and Live API documentation on +2026-09-18 and 2026-09-21, with the model-specific pages treated as +authoritative): + +- **Async function calling.** Non-blocking tool declarations (`behavior: + NON_BLOCKING`) let longer-running functions execute in the + background while the conversation continues. Omitted `behavior` defaults to + blocking (Google's tool-use guide: a declaration without `behavior` "will + still pause all interactions with the model"); the agent can provide updates + and returns results when ready, controlled by the `FunctionResponseScheduling` + values `SILENT`, `WHEN_IDLE`, and `INTERRUPT` that `FunctionCallResult` + forwards. Extended Thinking runs async-only and rejects scheduling. Google's + capabilities table spells the last value `INTERRUPTED`; the tool-use guide + and this ADR use `INTERRUPT`. +- **Proactive audio.** The agent speaks only when relevant and can remain + quiet unless directly addressed, making conversations less interruptive. This + is enabled permanently on both 3.8 models (see the fixed-choice entry in the + next section). +- **Client content for context injection.** The `clientContent` channel lets + callers add context without forcing a turn (`turn_complete: false`), + backchanneling information silently so the model stays informed without a + spoken response. With `turn_complete: true`, the same channel instead asks + the model to respond immediately, interrupting any active generation. +- **High/background reasoning.** Extended Thinking performs frontier-level + reasoning in the background and supports more complex tasks while remaining + responsive during the ongoing conversation. Standard 3.8 keeps interleaved + reasoning with fixed, non-configurable depth. +- **Native audio experience.** The background reasoning runs inside the native + audio pipeline, so deep processing and real-time voice interaction coexist + instead of trading one for the other. +- **Responsive multitasking.** The model can work on slower complex outputs + while still chatting, balancing deep reasoning with fast conversational + response. +- **Model capability comparison.** Standard `gemini-3.8-live` already carries + strong reasoning; `gemini-3.8-live-extended-thinking` favors deeper + performance on complex or creative tasks over minimal latency. +- **Google Search grounding.** The only supported grounding tool on both + models; exposed through `Tool::GoogleSearch` (its metadata is not forwarded; + see the exclusion list below). +- **Bidirectional audio transcription.** Input and output transcripts in + `VERBATIM` or `SMART` mode are configured through the transcription `Params` + fields documented above; native audio language selection stays automatic. + +## Features not exposed by `google-dialog` + +The public service API is intentionally narrower than the direct Gemini Live +wire API. As verified against Google's model and Live API documentation on +2026-09-18, this integration does not expose the following supported features: + +- Image or video input. Gemini accepts JPEG or PNG video frames at up to one + frame per second, but the core conversation input supports only audio, text, + and service events. `google-dialog` continues to require audio input. +- Arbitrary multimodal or batched `clientContent`. `ClientContent` sends one + role-qualified text part per service event; callers cannot submit multiple + turns, inline media, function parts, or arbitrary Gemini `Part` values. +- Manual and hybrid VAD signals. `Params.realtime_input_config` exposes setup + configuration, but callers cannot directly send `activityStart`, + `activityEnd`, or `audioStreamEnd` service events. The service sends + `audioStreamEnd` itself when its audio input closes. +- Media-resolution selection for image or video input. +- Generation controls other than the existing temperature and model-specific + thinking level. In particular, there are no public controls for `topP`, + `topK`, `maxOutputTokens`, or `seed`. +- Detailed transcription configuration beyond language hints and mode. Input + and output transcription can be enabled as booleans, input ASR language + hints are configurable, and both directions support `VERBATIM` or `SMART` + mode, but custom vocabulary, word timestamps, diarization, speaker labels, + and word timing are not configurable or emitted as service events. +- Thought summaries. `includeThoughts` is never enabled and thought parts are + not emitted. Exposing them requires a distinct core output contract so + reasoning summaries cannot be mistaken for ordinary model output. +- Dedicated Live Translation configuration, including target-language and + echo-target-language controls. A caller may still request language behavior + through instructions, but that is not equivalent to exposing Google's typed + translation configuration. +- Rich function responses. A service event returns one JSON result for one call + ID; it cannot return multiple function responses in one message, media parts, + or streaming/generator results through `willContinue`. +- Grounding metadata and citations produced by Google Search. Search grounding + itself remains available through `Tool::GoogleSearch`, but its structured + metadata is not forwarded to consumers. +- Raw lifecycle and transport events. `generationComplete`, interruption, + `GoAway`, and session-resumption updates are handled or ignored internally; + only the domain-level completion events and tool-call cancellation are public. +- Raw token-usage metadata. Usage is converted into billing records and is not + also emitted as a provider service event. +- Client-managed context compression and session resumption. Callers cannot set + compression thresholds, choose the sliding-window target, supply or retrieve + resume handles, request transparent resumption, or control reconnect timing. + The service enables and manages these mechanisms internally. +- Client-to-server authentication with ephemeral tokens. `google-dialog` is a + server-side service and authenticates direct Gemini API connections with its + configured API key. + +The following are fixed model or integration choices rather than missing +controls: + +- Response modality is audio. Text output is obtained through output-audio + transcription; callers cannot select a text-only Gemini response modality. +- Proactive audio is permanently enabled by both 3.8 models. The service does + not send a `proactivity` override; Google documents that setting + `proactive_audio: false` as an error. +- Native-audio language selection is automatic. Callers can guide language in + the system instructions, but Google does not accept an explicit language code + for these models. + +The following Gemini capabilities are not exposed because both 3.8 Live model +pages mark them unsupported, not because this client withholds a supported +feature: context caching, code execution, file search, Google Maps grounding, +image generation, structured output, and URL context. The models also do not +support the Batch API. Affective dialog is removed from both 3.8 Live model +contracts and is therefore not configurable. The general Live capabilities +guide currently describes affective dialog and proactive audio differently; +the model-specific 3.8 pages are treated as authoritative here. Google Search +grounding and function calling are supported and remain exposed; they are not +part of this exclusion list. + +Keep context-window compression enabled with Google's defaults and retain +automatic session resumption. Compression manages context growth; resumption +preserves the logical session across finite WebSocket lifetimes. Resumption must +be checkpoint-safe: + +- Treat `GoAway` as advance notice and continue consuming the current socket. +- Track the current `resumable` state and newest resumable handle. +- Reconnect before the deadline only from the newest valid checkpoint. +- On an unexpected disconnect, resume only when a currently valid checkpoint + exists. +- If the socket is lost while state is not resumable, fail with an explicit + possible-state-loss error instead of restoring stale state or silently + starting a new session. +- If `GoAway.timeLeft` expires before a valid checkpoint arrives, emit an API + error, close the socket, and mark the session closed. Do not attempt either a + resumed or fresh reconnect; callers must establish a new session explicitly. +- Defer ordinary outbound commands after `GoAway` so they cannot be sent to a + session that is about to terminate. Tool responses and an explicit close + remain allowed while waiting for the checkpoint. + +## Raw WebSocket client requirements + +The vendored `gemini-live-rs` implementation must follow the JSON wire contract, +independently of any Google SDK convenience behavior: + +1. Accept a bare model ID from `Params`, but send it in direct API setup as the + resource name `models/{model}`. +2. Omit unsupported setup fields instead of serializing null or default values. + In particular, omit the entire thinking configuration for standard 3.8. +3. Use audio as the response modality. Enable output-audio transcription when a + text transcript is required, and when input transcription is enabled, map + input language hints to `inputAudioTranscription.languageCodes`. +4. Deserialize `interactionStatus` from `serverContent`, alongside + `turnComplete`, and retain both values when decomposing a server message into + semantic events. +5. Continue reading after `turnComplete(IN_PROGRESS)`; later tool calls, audio, + or another completion may belong to the same dialog interaction. +6. Serialize function behavior on each function declaration. Serialize optional + scheduling on each function response, never as the current 3.8 declaration + contract. +7. Send each function result with the original function call ID. Do not assume + non-blocking results arrive in call order. +8. Serialize incremental context as `clientContent` with explicit part roles. + Treat `turnComplete: true` as an intentional interruption request. +9. Process session-resumption updates as checkpoint state, including + `resumable: false`; possession of an older handle is not proof that current + in-flight work can be restored. +10. Treat `GoAway.timeLeft` as a reconnect deadline while continuing to receive + updates needed to obtain a safe checkpoint. Expiry without a checkpoint is + a terminal client-visible failure, not permission to start a fresh session. + +## Consequences + +The service gains one intermediate output event and provider-specific input and +function-result fields. Exhaustive downstream matches must handle the new event. +Consumers may receive multiple finalized transcript segments before one terminal +`TurnComplete`, but they will no longer be told that an interaction ended while +reasoning or tools remain active. + +Existing serialized `FunctionCallResult` and `Prompt` inputs remain valid. +Existing serialized output events retain their spellings. Rust source +compatibility is intentionally broken in two places: constructors and patterns +for `FunctionCallResult` must add or ignore `scheduling`, and exhaustive matches +on `ServiceOutputEvent` must handle `InteractionInProgress`. A caller preserving +the prior function-result behavior migrates by setting `scheduling: None`. +Introducing a second scheduled-result variant was rejected because it would +preserve a duplicated public concept indefinitely while the output enum already +requires a compatibility release boundary. + +The implementation deliberately performs validation only for the two known 3.8 +contracts. Unknown and legacy model strings continue to pass through, preserving +forward compatibility and existing behavior. + +Prices and data-processing terms must be rechecked before deployment. As of +2026-09-18, Google listed both 3.8 Live models at the same paid rates per one +million tokens: text input $0.75, audio input $3.00, image/video input $1.00, +text output $4.50, and audio output $12.00. This ADR does not make those rates or +any GDPR deployment conclusion part of the software contract. + +## Sources verified on 2026-09-18 + +- +- +- +- +- +- +- +- +- +- +- — 3.8 Live GA announcement, + 2026-09-15 +- diff --git a/examples/dialog.rs b/examples/dialog.rs index bb7cee0c..99370479 100644 --- a/examples/dialog.rs +++ b/examples/dialog.rs @@ -46,6 +46,14 @@ struct Cli { /// Used only with provider `google-agent-platform`. #[arg(long)] location: Option, + /// Override whether server-side input transcription is enabled. Defaults to + /// off for providers that enable it by default. + #[arg(long)] + input_transcription: Option, + /// Override whether server-side output transcription is enabled. Defaults to + /// off for providers that enable it by default. + #[arg(long)] + output_transcription: Option, } #[derive(Debug, Clone, Copy, ValueEnum)] @@ -63,6 +71,26 @@ impl Provider { fn api(self) -> &'static dyn dialog_providers::ProviderApi { dialog_providers::provider_api(self) } + + /// Text input commands supported by the provider (see `send_input_line`). + fn capabilities(self) -> ProviderCapabilities { + let mut capabilities = ProviderCapabilities::default(); + + match self { + Provider::OpenAI | Provider::AzureOpenAI => {} + Provider::Google | Provider::GoogleAgentPlatform => { + capabilities.client_content = true; + } + } + + capabilities + } +} + +#[derive(Debug, Clone, Copy, Default)] +struct ProviderCapabilities { + /// The provider service accepts `clientContent` service input events. + client_content: bool, } #[tokio::main] @@ -91,6 +119,7 @@ async fn main() -> Result<()> { .expect("Failed to get default input config"); println!("Audio device input config: {input_config:?}"); + print_usage_hint(); let channels = input_config.channels(); let sample_rate = input_config.sample_rate(); @@ -200,7 +229,7 @@ async fn main() -> Result<()> { line = stdin_lines.next_line(), if !stdin_closed => { match line? { Some(line) => { - send_prompt_line(&input_sender, &line).await?; + send_input_line(cli.provider, &input_sender, &line).await?; } None => { stdin_closed = true; @@ -261,21 +290,104 @@ fn setup_audio_input_adapter( sender } -async fn send_prompt_line(input: &Sender, line: &str) -> Result<()> { - let prompt = line.trim(); - if prompt.is_empty() { +async fn send_input_line(provider: Provider, input: &Sender, line: &str) -> Result<()> { + let line = line.trim(); + if line.is_empty() { + return Ok(()); + } + + let event = parse_input_line(line); + let Some(event) = event else { + print_usage_hint(); + return Ok(()); + }; + + if matches!(event, InputEvent::ClientContent { .. }) && !provider.capabilities().client_content + { + println!( + "Provider '{}' does not support clientContent commands", + provider + .to_possible_value() + .expect("Provider has a possible value") + .get_name() + ); return Ok(()); } - input - .send(Input::ServiceEvent { - value: json!({ "type": "prompt", "text": prompt }), - }) - .await?; + let value = match event { + InputEvent::Prompt { text } => json!({ "type": "prompt", "text": text }), + InputEvent::ClientContent { + role, + text, + turn_complete, + } => json!({ + "type": "clientContent", + "role": role, + "text": text, + "turnComplete": turn_complete, + }), + }; + + input.send(Input::ServiceEvent { value }).await?; Ok(()) } +enum InputEvent { + Prompt { + text: String, + }, + ClientContent { + role: &'static str, + text: String, + turn_complete: bool, + }, +} + +/// Parses one stdin line into a service input event. +/// +/// Grammar (first word decides): +/// - `prompt ` — realtime prompt. +/// - `user ` / `agent ` — clientContent for the user/model role. +/// A single trailing `!` on the text is stripped and sets +/// `turnComplete: true` ("respond now"); `user !` sends empty text with +/// `turnComplete: true`. Without it, content is added silently. A bare +/// `user`/`agent` sends empty text, silent. +/// +/// Returns `None` for a line without text after the command word; the caller +/// prints the usage hint. +fn parse_input_line(line: &str) -> Option { + let (command, rest) = match line.split_once(' ') { + Some((command, rest)) => (command, rest.trim()), + None => (line, ""), + }; + + match command { + "prompt" if !rest.is_empty() => Some(InputEvent::Prompt { text: rest.into() }), + "user" | "agent" => { + let role = if command == "user" { "user" } else { "model" }; + let turn_complete = rest.ends_with('!'); + Some(InputEvent::ClientContent { + role, + text: rest.strip_suffix('!').unwrap_or(rest).into(), + turn_complete, + }) + } + _ => None, + } +} + +fn print_usage_hint() { + println!( + "Commands, one per line:\n\ + prompt \n\ + user [!]\n\ + agent [!]\n\ + a single trailing ! completes the turn now, otherwise content is added silently;\n\ + lines without a prompt/user/agent first word print this hint" + ); +} + fn list_available_voices(provider: Provider) -> Result<()> { println!("Available voices for {:?}:", provider); for voice in provider.api().voices() { @@ -299,6 +411,8 @@ async fn start_conversation(cli: &Cli, conversation: Conversation) -> Result<()> voice: cli.voice.clone(), project: cli.project.clone(), location: cli.location.clone(), + input_transcription: cli.input_transcription, + output_transcription: cli.output_transcription, }; cli.provider .api() diff --git a/examples/dialog_providers/google.rs b/examples/dialog_providers/google.rs index cae0c100..9c2eff15 100644 --- a/examples/dialog_providers/google.rs +++ b/examples/dialog_providers/google.rs @@ -4,7 +4,7 @@ use anyhow::{Context, Result}; use async_trait::async_trait; use gemini_live::types as gemini_types; use google_dialog::{ - GoogleDialog, ServiceInputEvent as GoogleServiceInputEvent, ServiceOutputEvent, + GEMINI_3_8_LIVE, GoogleDialog, ServiceInputEvent as GoogleServiceInputEvent, ServiceOutputEvent, }; use reqwest::Url; use serde::Deserialize; @@ -29,7 +29,7 @@ impl ProviderApi for GoogleProvider { .model .or_else(|| env::var("GEMINI_LIVE_API_MODEL").ok()) .filter(|model| !model.trim().is_empty()) - .unwrap_or_else(|| "gemini-3.1-flash-live-preview".to_owned()); + .unwrap_or_else(|| GEMINI_3_8_LIVE.to_owned()); let mut params = google_dialog::Params::new(model); params.api_key = Some(key); @@ -42,8 +42,8 @@ impl ProviderApi for GoogleProvider { .as_deref() .map(google_dialog::parse_voice_value) .transpose()?; - params.input_audio_transcription = true; - params.output_audio_transcription = true; + params.input_audio_transcription = request.input_transcription.unwrap_or_default(); + params.output_audio_transcription = request.output_transcription.unwrap_or_default(); params.tools.push(get_time_tool()); GoogleDialog.conversation(params, conversation).await @@ -64,6 +64,10 @@ impl ProviderApi for GoogleProvider { tracing::info!("Turn complete"); Ok(None) } + ServiceOutputEvent::InteractionInProgress => { + tracing::info!("Interaction remains in progress"); + Ok(None) + } ServiceOutputEvent::ToolCallCancellation { call_id } => { tracing::info!("Tool call cancelled: {call_id}"); Ok(None) @@ -73,8 +77,12 @@ impl ProviderApi for GoogleProvider { fn function_result_event(&self, call_id: String, result: String) -> Result { let output = json!({ "time": serde_json::Value::String(result) }); - serde_json::to_value(&GoogleServiceInputEvent::FunctionCallResult { call_id, output }) - .map_err(Into::into) + serde_json::to_value(&GoogleServiceInputEvent::FunctionCallResult { + call_id, + output, + scheduling: None, + }) + .map_err(Into::into) } fn output_format(&self, _input_format: AudioFormat) -> AudioFormat { diff --git a/examples/dialog_providers/google_agent_platform.rs b/examples/dialog_providers/google_agent_platform.rs index 31571597..5e533ddf 100644 --- a/examples/dialog_providers/google_agent_platform.rs +++ b/examples/dialog_providers/google_agent_platform.rs @@ -57,8 +57,8 @@ impl ProviderApi for GoogleAgentPlatformProvider { .as_deref() .map(google_dialog::parse_voice_value) .transpose()?; - params.input_audio_transcription = true; - params.output_audio_transcription = true; + params.input_audio_transcription = request.input_transcription.unwrap_or_default(); + params.output_audio_transcription = request.output_transcription.unwrap_or_default(); params.tools.push(get_time_tool()); GoogleDialog.conversation(params, conversation).await @@ -79,6 +79,10 @@ impl ProviderApi for GoogleAgentPlatformProvider { tracing::info!("Turn complete"); Ok(None) } + ServiceOutputEvent::InteractionInProgress => { + tracing::info!("Interaction remains in progress"); + Ok(None) + } ServiceOutputEvent::ToolCallCancellation { call_id } => { tracing::info!("Tool call cancelled: {call_id}"); Ok(None) @@ -88,8 +92,12 @@ impl ProviderApi for GoogleAgentPlatformProvider { fn function_result_event(&self, call_id: String, result: String) -> Result { let output = json!({ "time": serde_json::Value::String(result) }); - serde_json::to_value(&GoogleServiceInputEvent::FunctionCallResult { call_id, output }) - .map_err(Into::into) + serde_json::to_value(&GoogleServiceInputEvent::FunctionCallResult { + call_id, + output, + scheduling: None, + }) + .map_err(Into::into) } fn output_format(&self, _input_format: AudioFormat) -> AudioFormat { diff --git a/examples/dialog_providers/mod.rs b/examples/dialog_providers/mod.rs index cbe4b0ee..c828015b 100644 --- a/examples/dialog_providers/mod.rs +++ b/examples/dialog_providers/mod.rs @@ -17,6 +17,10 @@ pub struct StartConversationRequest { pub voice: Option, pub project: Option, pub location: Option, + /// Command-line transcription overrides; `Ok`/`Err` explicit on/off, + /// `None` keeps the provider default. + pub input_transcription: Option, + pub output_transcription: Option, } #[async_trait(?Send)] diff --git a/examples/dialog_providers/openai.rs b/examples/dialog_providers/openai.rs index 521798d2..2b9cfea7 100644 --- a/examples/dialog_providers/openai.rs +++ b/examples/dialog_providers/openai.rs @@ -47,8 +47,8 @@ impl ProviderApi for OpenAIProvider { .map(parse_realtime_voice_value) .transpose()?; params.tools.push(get_time_function_definition()); - params.input_audio_transcription = true; - params.output_audio_transcription = true; + params.input_audio_transcription = request.input_transcription.unwrap_or_default(); + params.output_audio_transcription = request.output_transcription.unwrap_or_default(); OpenAIDialog.conversation(params, conversation).await } diff --git a/external/gemini-live-rs b/external/gemini-live-rs index c713ca9a..05053252 160000 --- a/external/gemini-live-rs +++ b/external/gemini-live-rs @@ -1 +1 @@ -Subproject commit c713ca9a87b00469eba5876588ef7e5e75190f6e +Subproject commit 05053252e5b1a7cae3773d819d8a670db5c208cc diff --git a/services/google-dialog/src/client.rs b/services/google-dialog/src/client.rs index 7faf6e8c..0d637348 100644 --- a/services/google-dialog/src/client.rs +++ b/services/google-dialog/src/client.rs @@ -1,6 +1,7 @@ use std::mem; use anyhow::{Context, Result, anyhow, bail}; +use tracing::{debug, info, trace}; use gemini_live::transport::{Auth, Endpoint, TransportConfig}; use gemini_live::types::{ @@ -10,15 +11,16 @@ use gemini_live::types::{ UsageMetadata, VoiceConfig, }; use gemini_live::{ReconnectPolicy, Session, SessionConfig, SessionError}; -use tracing::{debug, info, trace, warn}; -use crate::conversation_state::ConversationState; -use crate::{Params, ServiceInputEvent, ServiceOutputEvent, TextOutputs}; use context_switch_core::{ AI_ASSISTANT_SPEAKER, AudioFormat, AudioFrame, BillingRecord, BillingSchedule, ConversationInput, ConversationOutput, Input, OutputPath, }; +use crate::conversation_state::ConversationState; +use crate::model; +use crate::{ClientContentRole, Params, ServiceInputEvent, ServiceOutputEvent, TextOutputs}; + const LEGACY_TOOL_CALL_ID: &str = "legacy-tool-call"; #[derive(Debug)] @@ -106,11 +108,19 @@ impl Client { .context("Sending text to Gemini Live")?; } Input::ServiceEvent { value } => match serde_json::from_value(value)? { - ServiceInputEvent::FunctionCallResult { call_id, output } => { + ServiceInputEvent::FunctionCallResult { + call_id, + output, + scheduling, + } => { let Some(name) = state.tool_calls.resolve(&call_id)? else { return Ok(()); }; + // Gemini 3.8 adds scheduled responses for non-blocking tools; + // Extended Thinking deliberately rejects this field. + model::validate_response_scheduling(&self.params, &name, scheduling.as_ref())?; + let response = normalize_function_response(output); let response_call_id = if call_id == LEGACY_TOOL_CALL_ID { None @@ -122,6 +132,7 @@ impl Client { id: response_call_id, name, response, + scheduling, }; session .send_tool_response(vec![response]) @@ -132,6 +143,31 @@ impl Client { info!("Received prompt"); session.send_text(&text).await.context("Sending prompt")?; } + ServiceInputEvent::ClientContent { + role, + text, + turn_complete, + } => { + // Gemini 3.8 adds full-session clientContent updates. The + // same envelope is supported on direct and Agent Platform routes. + let role = match role { + ClientContentRole::User => "user", + ClientContentRole::Model => "model", + }; + session + .send_client_content(gemini_live::types::ClientContent { + turns: Some(vec![Content { + role: Some(role.to_owned()), + parts: vec![Part { + text: Some(text), + inline_data: None, + }], + }]), + turn_complete: Some(turn_complete), + }) + .await + .context("Sending client content to Gemini Live")?; + } }, } Ok(()) @@ -169,25 +205,36 @@ impl Client { self.finalize_output_transcription(text_outputs, output, state)?; output.service_event(OutputPath::Media, ServiceOutputEvent::TurnComplete)?; } + ServerEvent::InteractionInProgress => { + self.finalize_output_transcription(text_outputs, output, state)?; + output + .service_event(OutputPath::Media, ServiceOutputEvent::InteractionInProgress)?; + } ServerEvent::Interrupted => { // We expect a TurnComplete afterward, so don't finalize the output transcription // when interrupted. output.clear_audio()?; } + ServerEvent::InterimInputTranscription(text) => { + if self.params.input_audio_transcription && text_outputs.interim { + output.text(false, text, None, None)?; + } + } ServerEvent::InputTranscription(text) => { if self.params.input_audio_transcription { if text_outputs.text { output.text(true, text, None, None)?; } } else { - // Observed with preview Gemini models: transcription events can still arrive - // even when transcription is not enabled in setup. - warn!( + // Observed with preview Gemini models and 3.8: transcription events can still + // arrive even when transcription is not enabled in setup. + trace!( transcript_len = text.len(), - "Received input transcription event while input_audio_transcription is disabled (observed with preview model)" + "Received input transcription event while input_audio_transcription is disabled" ); } } + ServerEvent::InputTranscriptionFinished => {} ServerEvent::OutputTranscription(text) => { if self.params.output_audio_transcription { state.output_transcription_buffer.push_str(&text); @@ -200,11 +247,11 @@ impl Client { )?; } } else { - // Observed with preview Gemini models: transcription events can still arrive - // even when transcription is not enabled in setup. - warn!( + // Observed with preview Gemini models and 3.8: transcription events can still + // arrive even when transcription is not enabled in setup. + trace!( transcript_len = text.len(), - "Received output transcription event while output_audio_transcription is disabled (observed with preview model)" + "Received output transcription event while output_audio_transcription is disabled" ); } } @@ -337,12 +384,33 @@ fn session_config(params: &Params, text_outputs: TextOutputs) -> Result Result { - let input_audio_transcription = params - .input_audio_transcription - .then_some(AudioTranscriptionConfig {}); - let output_audio_transcription = params - .output_audio_transcription - .then_some(AudioTranscriptionConfig {}); + // Gemini 3.8 introduced model-specific thinking policies: standard Live + // omits thinking_config, while Extended Thinking accepts low/medium/high. + let thinking_level = params + .thinking_level + .or_else(|| model::default_thinking_level(¶ms.model)); + model::validate_thinking_level(params)?; + + let input_audio_transcription_language_codes = + (!params.input_audio_transcription_language_codes.is_empty()) + .then(|| params.input_audio_transcription_language_codes.clone()); + + let input_audio_transcription = + params + .input_audio_transcription + .then_some(AudioTranscriptionConfig { + language_codes: input_audio_transcription_language_codes, + custom_vocabulary: None, + mode: Some(params.input_audio_transcription_mode), + }); + let output_audio_transcription = + params + .output_audio_transcription + .then_some(AudioTranscriptionConfig { + language_codes: None, + custom_vocabulary: None, + mode: Some(params.output_audio_transcription_mode), + }); if !(text_outputs.text || text_outputs.interim) && (input_audio_transcription.is_some() || output_audio_transcription.is_some()) @@ -352,6 +420,12 @@ fn setup_config(params: &Params, text_outputs: TextOutputs) -> Result Result Result Option { + match model { + GEMINI_3_8_LIVE => Some(ModelConfig { + // Fixed interleaved reasoning; `thinking_level` is not accepted. + thinking: ThinkingPolicy::Disabled, + // Both legacy blocking and asynchronous tools are supported. + functions: FunctionPolicy::BlockingAndNonBlocking, + // `INTERRUPT`, `WHEN_IDLE`, and `SILENT` are supported for + // non-blocking function responses. + response_scheduling: SchedulingPolicy::SupportedForNonBlocking, + }), + GEMINI_3_8_LIVE_EXTENDED_THINKING => Some(ModelConfig { + // Background reasoning accepts low, medium, and high levels, but + // not `minimal`. + thinking: ThinkingPolicy::Optional { + allows_minimal: false, + }, + // Tools must run in the background while the model reasons. + functions: FunctionPolicy::NonBlockingOnly, + // Function-response scheduling is not accepted. + response_scheduling: SchedulingPolicy::Unsupported, + }), + _ => None, + } +} + +pub fn default_thinking_level(model: &str) -> Option { + match model { + GEMINI_3_8_LIVE_EXTENDED_THINKING => Some(ThinkingLevel::Medium), + _ => None, + } +} + +pub fn validate_thinking_level(params: &Params) -> Result<()> { + let Some(config) = model_config(¶ms.model) else { + return Ok(()); + }; + + match (config.thinking, params.thinking_level) { + (ThinkingPolicy::Disabled, Some(level)) => bail!( + "Model `{}` does not support thinking_level `{level}`; omit thinking_level", + params.model + ), + (ThinkingPolicy::Optional { allows_minimal }, Some(ThinkingLevel::Minimal)) + if !allows_minimal => + { + bail!( + "Model `{}` does not support thinking_level `minimal`", + params.model + ) + } + _ => Ok(()), + } +} + +pub fn tools_for_model(model: &str, input_tools: &[Tool]) -> Result> { + let mut tools = input_tools.to_vec(); + let Some(config) = model_config(model) else { + return Ok(tools); + }; + if !matches!(config.functions, FunctionPolicy::NonBlockingOnly) { + return Ok(tools); + } + + for tool in &mut tools { + let Tool::FunctionDeclarations(declarations) = tool else { + continue; + }; + for declaration in declarations { + if declaration.scheduling.is_some() { + bail!( + "Model `{}` does not support function declaration scheduling", + model + ); + } + match declaration.behavior.clone() { + Some(FunctionBehavior::Blocking) => bail!( + "Model `{}` requires non-blocking function declarations", + model + ), + None => declaration.behavior = Some(FunctionBehavior::NonBlocking), + Some(FunctionBehavior::NonBlocking) => {} + } + } + } + Ok(tools) +} + +pub fn validate_response_scheduling( + params: &Params, + function_name: &str, + scheduling: Option<&FunctionResponseScheduling>, +) -> Result<()> { + let Some(_scheduling) = scheduling else { + return Ok(()); + }; + let Some(config) = model_config(¶ms.model) else { + return Ok(()); + }; + match config.response_scheduling { + SchedulingPolicy::Unsupported => bail!( + "Model `{}` does not support function response scheduling", + params.model + ), + SchedulingPolicy::SupportedForNonBlocking + if function_behavior(¶ms.tools, function_name) + .unwrap_or(FunctionBehavior::Blocking) + == FunctionBehavior::Blocking => + { + bail!( + "Function response scheduling is not valid for blocking function `{function_name}`" + ) + } + SchedulingPolicy::SupportedForNonBlocking => {} + } + Ok(()) +} + +fn function_behavior(tools: &[Tool], function_name: &str) -> Option { + tools.iter().find_map(|tool| { + let Tool::FunctionDeclarations(declarations) = tool else { + return None; + }; + declarations + .iter() + .find(|declaration| declaration.name == function_name) + .and_then(|declaration| declaration.behavior.clone()) + }) +} diff --git a/services/google-dialog/src/types.rs b/services/google-dialog/src/types.rs index cdf693ca..96f5b8b7 100644 --- a/services/google-dialog/src/types.rs +++ b/services/google-dialog/src/types.rs @@ -1,7 +1,8 @@ -use gemini_live::types::{FunctionDeclaration, RealtimeInputConfig, ThinkingLevel, Tool}; +use anyhow::{Result, bail}; use serde::{Deserialize, Deserializer, Serialize}; -use anyhow::{Result, bail}; +pub use gemini_live::types::{FunctionBehavior, FunctionResponseScheduling, TranscriptionMode}; +use gemini_live::types::{FunctionDeclaration, RealtimeInputConfig, ThinkingLevel, Tool}; #[derive(Debug, Serialize, Deserialize)] #[serde(rename_all = "camelCase")] @@ -29,8 +30,11 @@ pub struct Params { /// Sampling temperature. Valid range: `0.0..=2.0`. /// If omitted, Gemini uses the model-specific default temperature. pub temperature: Option, - /// Gemini 3.1 thinking level (`minimal`, `low`, `medium`, or `high`). - /// In Live API, Gemini 3.1 defaults to `minimal` when omitted. + /// Thinking level for Gemini 3.1 and Gemini 3.8 Extended Thinking + /// (`minimal`, `low`, `medium`, or `high`, subject to model support). + /// When omitted, Google applies the model default. `gemini-3.8-live` + /// requires this field to remain omitted; Extended Thinking accepts + /// `low`, `medium`, or `high`. pub thinking_level: Option, /// Enabled by default to avoid context-window exhaustion during long audio sessions. #[serde(default = "default_context_window_compression")] @@ -42,9 +46,27 @@ pub struct Params { /// Enable server-side transcription of user input audio. #[serde(default)] pub input_audio_transcription: bool, + /// BCP-47 language hints for input audio transcription. + /// An explicit `null` is invalid; omit the field to use no language hints. + #[serde(default)] + pub input_audio_transcription_language_codes: Vec, + /// Transcription style for user input audio. Defaults to `VERBATIM`. + #[serde(default = "default_transcription_mode")] + pub input_audio_transcription_mode: TranscriptionMode, /// Enable server-side transcription of model output audio. #[serde(default)] pub output_audio_transcription: bool, + /// Transcription style for model output audio. Defaults to `VERBATIM`. + #[serde(default = "default_transcription_mode")] + pub output_audio_transcription_mode: TranscriptionMode, +} + +fn default_context_window_compression() -> bool { + true +} + +fn default_transcription_mode() -> TranscriptionMode { + TranscriptionMode::Verbatim } impl Params { @@ -63,11 +85,22 @@ impl Params { tools: vec![], realtime_input_config: None, input_audio_transcription: false, + input_audio_transcription_language_codes: vec![], + input_audio_transcription_mode: default_transcription_mode(), output_audio_transcription: false, + output_audio_transcription_mode: default_transcription_mode(), } } } +/// The 30 prebuilt Gemini Live API output voices, kept in Google's documented +/// order (see https://ai.google.dev/gemini-api/docs/speech-generation#voices). +/// One flat list for every native-audio Live model: Google publishes no +/// per-model voice subsetting for Live API (`gemini-3.8-live`, +/// `gemini-3.8-live-extended-thinking`, and Agent Platform-routed models all +/// take the same set). Note the `generateContent` TTS voice set is slightly +/// different and does not apply here; this service only speaks the Live +/// WebSocket protocol. pub const VOICES: &[&str] = &[ "Zephyr", "Puck", @@ -113,10 +146,6 @@ pub fn parse_voice_value(value: &str) -> Result { } } -fn default_context_window_compression() -> bool { - true -} - fn deserialize_tools<'de, D>(deserializer: D) -> Result, D::Error> where D: Deserializer<'de>, @@ -175,6 +204,17 @@ enum OpenAiToolType { Function, } +/// Choice pattern for the text input variants: +/// +/// - [`ServiceInputEvent::Prompt`]: talk to the model now, like a user speaking. +/// - [`ServiceInputEvent::ClientContent`] with [`ClientContentRole::User`] and +/// `turn_complete: true`: ask or instruct the model for an immediate response +/// (interrupts active generation). +/// - [`ServiceInputEvent::ClientContent`] with [`ClientContentRole::User`] and +/// `turn_complete: false`: add context silently; the response comes later +/// from the audio flow. +/// - [`ServiceInputEvent::ClientContent`] with [`ClientContentRole::Model`]: +/// restore or fabricate an earlier assistant turn. #[derive(Debug, Serialize, Deserialize)] #[serde( tag = "type", @@ -185,10 +225,46 @@ pub enum ServiceInputEvent { FunctionCallResult { call_id: String, output: serde_json::Value, + /// Gemini 3.8 scheduling for a non-blocking function response. + #[serde(default, skip_serializing_if = "Option::is_none")] + scheduling: Option, }, - Prompt { + /// Gemini 3.8 incremental conversation content sent during a live session. + /// + /// Appends an ordered history entry via the `clientContent` wire message. + /// With `turn_complete: false` (the default) the content is added and + /// generation stays pending; `true` starts generation immediately and + /// intentionally interrupts active generation. Typical usage: seeding or + /// restoring context without audio, scripted turns, and deterministic + /// prompt delivery. Not a realtime input path. + ClientContent { + /// Author of the appended conversation content. + role: ClientContentRole, + /// Text for one conversation content part; empty text is allowed. text: String, + /// Start generation after appending the content and interrupt active generation. + #[serde(default)] + turn_complete: bool, }, + /// Realtime user text input sent as the `realtimeInput.text` wire message. + /// + /// Behaves like the user just said the text: the model interprets it and + /// responds subject to turn state, without a guaranteed interrupt of + /// active generation or deterministic ordering against the audio stream. + /// Always user-side; cannot insert model history. Typical usage: typed + /// live input in an audio session (including instruction-style text such + /// as "Say: Hello"). For exact synthesis or scripted turns use + /// [`ServiceInputEvent::ClientContent`] instead. + Prompt { text: String }, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "lowercase")] +pub enum ClientContentRole { + /// Content supplied as user-authored conversation context. + User, + /// Content supplied as model-authored conversation context. + Model, } #[derive(Debug, Serialize, Deserialize)] @@ -207,6 +283,8 @@ pub enum ServiceOutputEvent { call_id: String, }, TurnComplete, + /// Gemini 3.8 indicates that the turn ended while the interaction continues. + InteractionInProgress, } #[cfg(test)]