From 33f1ce96e7bbdf98a4d21a654a3b96b727ce2646 Mon Sep 17 00:00:00 2001 From: Armin Sander Date: Fri, 18 Sep 2026 11:53:43 +0200 Subject: [PATCH 01/16] feat(google-dialog): document Gemini 3.8 Live semantics --- CONTEXT.md | 15 + .../0006-gemini-3-8-live-model-semantics.md | 444 ++++++++++++++++++ services/google-dialog/src/client.rs | 7 +- 3 files changed, 463 insertions(+), 3 deletions(-) create mode 100644 docs/adr/0006-gemini-3-8-live-model-semantics.md diff --git a/CONTEXT.md b/CONTEXT.md index 474224e7..c402c4f0 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -58,6 +58,21 @@ These three Microsoft offerings are distinct and must not all be called "Azure". ## Protocol terms +- **Model Turn** — one contiguous segment of model output. A model turn may end + while the dialog interaction remains active because the provider is still + reasoning or waiting for asynchronous tool results. + +- **Dialog Interaction** — processing initiated by user input that may span + multiple model turns and tool calls. It completes only when the provider reports + that no reasoning, generation, or tool work remains. + +- **Interaction In Progress** — a model-turn boundary at which the dialog + interaction remains active. More tool calls or model output may follow without + new user input. + +- **Interaction Idle** — the terminal state of a dialog interaction. No reasoning, + generation, or tool work remains, and new model output requires new client input. + - **Initial Start Message** — the first WebSocket message for a conversation. It either contains complete service parameters or declares that they will follow in a deferred params message. diff --git a/docs/adr/0006-gemini-3-8-live-model-semantics.md b/docs/adr/0006-gemini-3-8-live-model-semantics.md new file mode 100644 index 00000000..7f386cef --- /dev/null +++ b/docs/adr/0006-gemini-3-8-live-model-semantics.md @@ -0,0 +1,444 @@ +# ADR 0006: Gemini 3.8 Live model semantics + +Date: 2026-09-18 +Status: Accepted + +## Context + +Google exposes two stable Gemini Developer API models with materially different +runtime contracts: + +- `gemini-3.8-live` is the default low-latency model. It rejects + `thinking_level`, supports blocking and non-blocking function calls, and + supports scheduling non-blocking function responses. +- `gemini-3.8-live-extended-thinking` performs background reasoning. It accepts + no thinking level or `low`, `medium`, or `high`; requires non-blocking + functions; rejects function-response scheduling; and may end a model turn + while the dialog interaction remains active. + +The existing Google dialog API accepts an open model string, maps every provider +`turnComplete` to the public terminal `TurnComplete` event, and depends on a +vendored raw-WebSocket client whose tool and lifecycle types predate these +contracts. Merely accepting the new model strings would therefore advertise +support while exposing incorrect lifecycle and tool behavior. + +The models were verified as stable in the Gemini Developer API. Their +availability through Gemini Enterprise Agent Platform, including EU data +residency, was not established by Google's published model-location +documentation. Paid Gemini API use is governed by Google's Data Processing +Addendum and Google states that paid prompts and responses are not used to +improve its products, but transient abuse-monitoring processing may occur in any +country where Google or its agents maintain facilities. These are dated +deployment facts, not properties guaranteed by this client. + +## Decision + +Support both models as production-ready direct Gemini API models. Keep +`Params.model` as an open `String`, add public `GEMINI_3_8_LIVE` and +`GEMINI_3_8_LIVE_EXTENDED_THINKING` constants, and do not add a model allowlist +or a separate extended-thinking flag. The direct API example defaults to +`GEMINI_3_8_LIVE`; Extended Thinking remains opt-in. Existing model IDs and +Agent Platform behavior remain supported. The 3.8 constants do not assert Agent +Platform availability. + +The public constants have these exact declarations and remain bare model IDs: + +```rust +pub const GEMINI_3_8_LIVE: &str = "gemini-3.8-live"; +pub const GEMINI_3_8_LIVE_EXTENDED_THINKING: &str = + "gemini-3.8-live-extended-thinking"; +``` + +The integration uses the domain distinction in `CONTEXT.md`: a wire +`turnComplete` closes a **Model Turn**, while only `interactionStatus: IDLE` +closes the **Dialog Interaction**. + +- Finalize the current output transcript at every wire `turnComplete`. +- Map `turnComplete` with `interactionStatus: IN_PROGRESS` to the new public + `ServiceOutputEvent::InteractionInProgress`. +- Map `turnComplete` with `interactionStatus: IDLE` to the existing terminal + `ServiceOutputEvent::TurnComplete`. +- Preserve legacy completion behavior when `interactionStatus` is absent. +- Preserve the status through the vendored client's wire and semantic event + types; do not infer interaction state from model names. + +The public output enum gains exactly one unit variant: + +```rust +pub enum ServiceOutputEvent { + // Existing variants remain unchanged. + InteractionInProgress, +} +``` + +Its serialized form is `{"type":"interactionInProgress"}`. `TurnComplete` +retains `{"type":"turnComplete"}` and remains the only terminal interaction +event. No provider status enum is exposed: `IN_PROGRESS` and `IDLE` are wire +states whose domain meanings are represented by the two service events. + +Validate model-specific setup before opening the WebSocket: + +- `gemini-3.8-live` requires `thinking_level` to be absent. +- `gemini-3.8-live-extended-thinking` permits an absent level or `low`, + `medium`, or `high`, and rejects `minimal`. +- Legacy models retain their existing setup behavior. + +For tools, declaration behavior and result scheduling are separate concepts. +The vendored protocol must model both `BLOCKING` and `NON_BLOCKING` declaration +behavior. Scheduling belongs on `FunctionResponse`, with typed values for +immediate interruption, delivery when idle, and silent context insertion; the +wire spelling must follow the current Google schema. + +The exact types are: + +```rust +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum FunctionBehavior { + Blocking, + NonBlocking, +} + +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum FunctionResponseScheduling { + Silent, + WhenIdle, + Interrupt, +} +``` + +These types are defined by `gemini-live` because they directly represent wire +fields, and are re-exported by `google-dialog` so callers do not need to name +the vendored crate. `FunctionDeclaration::behavior` remains an +`Option`. Its existing declaration-level `scheduling` field +remains only for legacy deserialization and is not re-exported as the response +scheduling type. + +`ServiceInputEvent::FunctionCallResult` changes to this exact shape: + +```rust +FunctionCallResult { + call_id: String, + output: serde_json::Value, + #[serde(default, skip_serializing_if = "Option::is_none")] + scheduling: Option, +}, +``` + +Omitting `scheduling` means no field is sent and lets Google apply its +`WHEN_IDLE` default where scheduling is supported. The serialized forms are: + +```json +{"type":"functionCallResult","callId":"call-1","output":{"ok":true}} +``` + +```json +{"type":"functionCallResult","callId":"call-1","output":{"ok":true},"scheduling":"INTERRUPT"} +``` + +- Standard 3.8 permits blocking and non-blocking declarations and optional + scheduling on non-blocking function responses. +- Extended Thinking defaults omitted declaration behavior to `NON_BLOCKING`, + preserves explicit `NON_BLOCKING`, and rejects `BLOCKING`. +- Extended Thinking rejects scheduling both in declarations and function + responses rather than silently dropping it. +- Declaration-level scheduling remains readable for legacy compatibility but is + not used to represent the current 3.8 response contract. +- Function call IDs remain the correlation key, allowing multiple non-blocking + calls to be outstanding and results to arrive out of order. + +For standard 3.8, an explicit response schedule paired with a blocking +declaration is rejected rather than sent as an ignored field. For Extended +Thinking, any explicit response schedule is rejected. These checks happen when +the service event is handled because the call ID identifies the corresponding +declaration. Omitted scheduling is valid for every model. Unknown and legacy +models retain pass-through behavior except that response scheduling is emitted +only on direct Gemini API connections; Agent Platform support is not inferred. + +Expose full-session incremental context as a provider input event named +`ClientContent` (the Google wire envelope is `clientContent`). It carries an +explicit `user` or `model` role, text content, and `turn_complete`. Keep the +existing realtime `Prompt` input unchanged. Sending client content with +`turn_complete: true` interrupts active generation; sending it without that flag +appends content and waits for further input. + +The exact public API is: + +```rust +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "lowercase")] +pub enum ClientContentRole { + User, + Model, +} + +pub enum ServiceInputEvent { + // Existing variants remain, with FunctionCallResult extended as above. + ClientContent { + role: ClientContentRole, + text: String, + #[serde(default)] + turn_complete: bool, + }, +} +``` + +The members have this contract: + +- `role: ClientContentRole` identifies the author of the history entry. `User` + serializes as `user` and `Model` as `model`. No other roles are accepted; + this prevents arbitrary role strings from reaching the Gemini wire protocol. +- `text: String` is the text of exactly one content part. It is required, is + sent as `clientContent.turns[0].parts[0].text`, and is not interpreted as + realtime input. Empty text is allowed because the event represents content, + not a prompt-validation policy. +- `turn_complete: bool` controls generation after the content is appended. The + default is `false`, which appends the content and leaves generation pending. + `true` asks Gemini to start generation immediately and intentionally interrupts + active generation. It serializes as `turnComplete`. + +The roles have distinct conversation semantics: + +- `User` means that `text` is user-authored context supplied by the client. It + represents something the end user said or wrote and is eligible to be used as + user input when Gemini generates the next response. +- `Model` means that `text` is model-authored context supplied by the client. It + represents an earlier assistant/model response that the client is restoring + or appending to conversation history; it does not claim that Gemini generated + this text during the current connection. + +These roles describe conversation authorship, not caller authorization, audio +direction, or the event source. A client must not label newly supplied user +input as `Model`, and the service must not infer a `Model` entry from a +realtime `Prompt`. `ClientContentRole` is provider-owned and restricts callers +to the two roles accepted by Google. One event maps to one +`clientContent.turns` entry with `role`, one text part, and the requested +completion flag. The service sends `turnComplete` explicitly, including when +it is `false`, so the resulting wire intent is unambiguous. Input-event JSON +uses the service tag and field names; the client converts it to the Gemini +envelope rather than forwarding this JSON directly: + +```json +{"type":"clientContent","role":"user","text":"Remember this.","turnComplete":false} +``` + +```json +{"type":"clientContent","role":"model","text":"Earlier answer.","turnComplete":true} +``` + +The corresponding Gemini wire message for the first example is: + +```json +{"clientContent":{"turns":[{"role":"user","parts":[{"text":"Remember this."}]}],"turnComplete":false}} +``` + +`ClientContent` is valid for both `User` and `Model` history entries, but it is +not a replacement for ordinary audio or realtime text input. The event is +provider-specific and is rejected by the service when the selected routing +cannot send direct Gemini Live `clientContent` messages. + +The existing `Prompt { text }` variant and its JSON remain unchanged. It maps to +realtime text input, is always user input, and cannot insert model history. + +Input transcription language hints are exposed on `Params`: + +```rust +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum AudioTranscriptionMode { + Verbatim, + Smart, +} + +#[serde(default, skip_serializing_if = "Option::is_none")] +pub input_audio_transcription_language_codes: Option>, +#[serde(default, skip_serializing_if = "Option::is_none")] +pub input_audio_transcription_mode: Option, +#[serde(default, skip_serializing_if = "Option::is_none")] +pub output_audio_transcription_mode: Option, +``` + +The field serializes as `inputAudioTranscriptionLanguageCodes` and maps to +Google's `inputAudioTranscription.languageCodes`. Values are BCP-47 language +codes used as input ASR hints, not a request to force the model's native audio +response language. `None` leaves language detection automatic and does not +enable input transcription. A non-empty list enables input transcription and +sends the hints; an empty list is rejected as invalid rather than being sent +as an ambiguous configuration. The existing `input_audio_transcription: +bool` remains valid for enabling input transcription without language hints. +The transcription mode fields map to the corresponding configuration's `mode`: +`inputAudioTranscription.mode` and `outputAudioTranscription.mode`. `None` +uses Google's default, `VERBATIM`; `SMART` removes disfluencies, performs light +grammatical cleanup, applies automatic formatting, and makes minor inline +corrections. `SMART` cannot be combined with word timestamps or diarization; +those controls remain unexposed. Output transcription has no corresponding +language-code parameter because native-audio output language selection is +automatic for these models. + +`google-dialog` re-exports the complete new surface from its crate root: + +```rust +pub use types::{ + ClientContentRole, FunctionBehavior, FunctionResponseScheduling, + GEMINI_3_8_LIVE, GEMINI_3_8_LIVE_EXTENDED_THINKING, Params, + ServiceInputEvent, ServiceOutputEvent, VOICES, parse_voice_value, +}; +``` + +## Features not exposed by `google-dialog` + +The public service API is intentionally narrower than the direct Gemini Live +wire API. As verified against Google's model and Live API documentation on +2026-09-18, this integration does not expose the following supported features: + +- Image or video input. Gemini accepts JPEG or PNG video frames at up to one + frame per second, but the core conversation input supports only audio, text, + and service events. `google-dialog` continues to require audio input. +- Arbitrary multimodal or batched `clientContent`. `ClientContent` sends one + role-qualified text part per service event; callers cannot submit multiple + turns, inline media, function parts, or arbitrary Gemini `Part` values. +- Manual and hybrid VAD signals. `Params.realtime_input_config` exposes setup + configuration, but callers cannot directly send `activityStart`, + `activityEnd`, or `audioStreamEnd` service events. The service sends + `audioStreamEnd` itself when its audio input closes. +- Media-resolution selection for image or video input. +- Generation controls other than the existing temperature and model-specific + thinking level. In particular, there are no public controls for `topP`, + `topK`, `maxOutputTokens`, or `seed`. +- Detailed transcription configuration beyond language hints and mode. Input + and output transcription can be enabled as booleans, input ASR language + hints are configurable, and both directions support `VERBATIM` or `SMART` + mode, but custom vocabulary, word timestamps, diarization, speaker labels, + and word timing are not configurable or emitted as service events. +- Thought summaries. `includeThoughts` is never enabled and thought parts are + not emitted. Exposing them requires a distinct core output contract so + reasoning summaries cannot be mistaken for ordinary model output. +- Dedicated Live Translation configuration, including target-language and + echo-target-language controls. A caller may still request language behavior + through instructions, but that is not equivalent to exposing Google's typed + translation configuration. +- Rich function responses. A service event returns one JSON result for one call + ID; it cannot return multiple function responses in one message, media parts, + or streaming/generator results through `willContinue`. +- Grounding metadata and citations produced by Google Search. Search grounding + itself remains available through `Tool::GoogleSearch`, but its structured + metadata is not forwarded to consumers. +- Raw lifecycle and transport events. `generationComplete`, interruption, + `GoAway`, and session-resumption updates are handled or ignored internally; + only the domain-level completion events and tool-call cancellation are public. +- Raw token-usage metadata. Usage is converted into billing records and is not + also emitted as a provider service event. +- Client-managed context compression and session resumption. Callers cannot set + compression thresholds, choose the sliding-window target, supply or retrieve + resume handles, request transparent resumption, or control reconnect timing. + The service enables and manages these mechanisms internally. +- Client-to-server authentication with ephemeral tokens. `google-dialog` is a + server-side service and authenticates direct Gemini API connections with its + configured API key. + +The following are fixed model or integration choices rather than missing +controls: + +- Response modality is audio. Text output is obtained through output-audio + transcription; callers cannot select a text-only Gemini response modality. +- Proactive audio is always enabled by both 3.8 models. The service exposes no + toggle because Google rejects attempts to disable it. +- Native-audio language selection is automatic. Callers can guide language in + the system instructions, but Google does not accept an explicit language code + for these models. + +The following Gemini capabilities are not exposed because both 3.8 Live model +pages mark them unsupported, not because this client withholds a supported +feature: context caching, code execution, file search, Google Maps grounding, +image generation, structured output, and URL context. The models also do not +support the Batch API. Affective dialog was removed from the 3.8 Live API and is +therefore not configurable. Google Search grounding and function calling are +supported and remain exposed; they are not part of this exclusion list. + +Keep context-window compression enabled with Google's defaults and retain +automatic session resumption. Compression manages context growth; resumption +preserves the logical session across finite WebSocket lifetimes. Resumption must +be checkpoint-safe: + +- Treat `GoAway` as advance notice and continue consuming the current socket. +- Track the current `resumable` state and newest resumable handle. +- Reconnect before the deadline only from the newest valid checkpoint. +- On an unexpected disconnect, resume only when a currently valid checkpoint + exists. +- If the socket is lost while state is not resumable, fail with an explicit + possible-state-loss error instead of restoring stale state or silently + starting a new session. + +## Raw WebSocket client requirements + +The vendored `gemini-live-rs` implementation must follow the JSON wire contract, +independently of any Google SDK convenience behavior: + +1. Accept a bare model ID from `Params`, but send it in direct API setup as the + resource name `models/{model}`. +2. Omit unsupported setup fields instead of serializing null or default values. + In particular, omit the entire thinking configuration for standard 3.8. +3. Use audio as the response modality. Enable output-audio transcription when a + text transcript is required, and map input language hints to + `inputAudioTranscription.languageCodes` while enabling input transcription + when a non-empty hint list is supplied. +4. Deserialize `interactionStatus` from `serverContent`, alongside + `turnComplete`, and retain both values when decomposing a server message into + semantic events. +5. Continue reading after `turnComplete(IN_PROGRESS)`; later tool calls, audio, + or another completion may belong to the same dialog interaction. +6. Serialize function behavior on each function declaration. Serialize optional + scheduling on each function response, never as the current 3.8 declaration + contract. +7. Send each function result with the original function call ID. Do not assume + non-blocking results arrive in call order. +8. Serialize incremental context as `clientContent` with explicit part roles. + Treat `turnComplete: true` as an intentional interruption request. +9. Process session-resumption updates as checkpoint state, including + `resumable: false`; possession of an older handle is not proof that current + in-flight work can be restored. +10. Treat `GoAway.timeLeft` as a reconnect deadline while continuing to receive + updates needed to obtain a safe checkpoint. + +## Consequences + +The service gains one intermediate output event and provider-specific input and +function-result fields. Exhaustive downstream matches must handle the new event. +Consumers may receive multiple finalized transcript segments before one terminal +`TurnComplete`, but they will no longer be told that an interaction ended while +reasoning or tools remain active. + +Existing serialized `FunctionCallResult` and `Prompt` inputs remain valid. +Existing serialized output events retain their spellings. Rust source +compatibility is intentionally broken in two places: constructors and patterns +for `FunctionCallResult` must add or ignore `scheduling`, and exhaustive matches +on `ServiceOutputEvent` must handle `InteractionInProgress`. A caller preserving +the prior function-result behavior migrates by setting `scheduling: None`. +Introducing a second scheduled-result variant was rejected because it would +preserve a duplicated public concept indefinitely while the output enum already +requires a compatibility release boundary. + +The implementation deliberately performs validation only for the two known 3.8 +contracts. Unknown and legacy model strings continue to pass through, preserving +forward compatibility and existing behavior. + +Prices and data-processing terms must be rechecked before deployment. As of +2026-09-18, Google listed both 3.8 Live models at the same paid rates per one +million tokens: text input $0.75, audio input $3.00, image/video input $1.00, +text output $4.50, and audio output $12.00. This ADR does not make those rates or +any GDPR deployment conclusion part of the software contract. + +## Sources verified on 2026-09-18 + +- +- +- +- +- +- +- +- +- +- +- \ No newline at end of file diff --git a/services/google-dialog/src/client.rs b/services/google-dialog/src/client.rs index 7faf6e8c..1e1fe318 100644 --- a/services/google-dialog/src/client.rs +++ b/services/google-dialog/src/client.rs @@ -1,6 +1,7 @@ use std::mem; use anyhow::{Context, Result, anyhow, bail}; +use tracing::{debug, info, trace, warn}; use gemini_live::transport::{Auth, Endpoint, TransportConfig}; use gemini_live::types::{ @@ -10,15 +11,15 @@ use gemini_live::types::{ UsageMetadata, VoiceConfig, }; use gemini_live::{ReconnectPolicy, Session, SessionConfig, SessionError}; -use tracing::{debug, info, trace, warn}; -use crate::conversation_state::ConversationState; -use crate::{Params, ServiceInputEvent, ServiceOutputEvent, TextOutputs}; use context_switch_core::{ AI_ASSISTANT_SPEAKER, AudioFormat, AudioFrame, BillingRecord, BillingSchedule, ConversationInput, ConversationOutput, Input, OutputPath, }; +use crate::conversation_state::ConversationState; +use crate::{Params, ServiceInputEvent, ServiceOutputEvent, TextOutputs}; + const LEGACY_TOOL_CALL_ID: &str = "legacy-tool-call"; #[derive(Debug)] From 190605be762b86156672f58d792ef1f061514e9d Mon Sep 17 00:00:00 2001 From: Armin Sander Date: Fri, 18 Sep 2026 12:00:48 +0200 Subject: [PATCH 02/16] feat(google-dialog): integrate live transcription config --- .../0006-gemini-3-8-live-model-semantics.md | 6 ++-- external/gemini-live-rs | 2 +- services/google-dialog/src/client.rs | 35 +++++++++++++++---- services/google-dialog/src/lib.rs | 5 ++- services/google-dialog/src/types.rs | 11 ++++++ 5 files changed, 48 insertions(+), 11 deletions(-) diff --git a/docs/adr/0006-gemini-3-8-live-model-semantics.md b/docs/adr/0006-gemini-3-8-live-model-semantics.md index 7f386cef..ec3d08e6 100644 --- a/docs/adr/0006-gemini-3-8-live-model-semantics.md +++ b/docs/adr/0006-gemini-3-8-live-model-semantics.md @@ -246,7 +246,7 @@ Input transcription language hints are exposed on `Params`: ```rust #[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] #[serde(rename_all = "SCREAMING_SNAKE_CASE")] -pub enum AudioTranscriptionMode { +pub enum TranscriptionMode { Verbatim, Smart, } @@ -254,9 +254,9 @@ pub enum AudioTranscriptionMode { #[serde(default, skip_serializing_if = "Option::is_none")] pub input_audio_transcription_language_codes: Option>, #[serde(default, skip_serializing_if = "Option::is_none")] -pub input_audio_transcription_mode: Option, +pub input_audio_transcription_mode: Option, #[serde(default, skip_serializing_if = "Option::is_none")] -pub output_audio_transcription_mode: Option, +pub output_audio_transcription_mode: Option, ``` The field serializes as `inputAudioTranscriptionLanguageCodes` and maps to diff --git a/external/gemini-live-rs b/external/gemini-live-rs index c713ca9a..814d6540 160000 --- a/external/gemini-live-rs +++ b/external/gemini-live-rs @@ -1 +1 @@ -Subproject commit c713ca9a87b00469eba5876588ef7e5e75190f6e +Subproject commit 814d654000e88812d6aea3771a471ddfcb5b4898 diff --git a/services/google-dialog/src/client.rs b/services/google-dialog/src/client.rs index 1e1fe318..35786303 100644 --- a/services/google-dialog/src/client.rs +++ b/services/google-dialog/src/client.rs @@ -189,6 +189,12 @@ impl Client { ); } } + ServerEvent::InputTranscriptionFinished => {} + ServerEvent::InterimInputTranscription(text) => { + if self.params.input_audio_transcription && text_outputs.interim { + output.text(false, text, None, None)?; + } + } ServerEvent::OutputTranscription(text) => { if self.params.output_audio_transcription { state.output_transcription_buffer.push_str(&text); @@ -338,12 +344,29 @@ fn session_config(params: &Params, text_outputs: TextOutputs) -> Result Result { - let input_audio_transcription = params - .input_audio_transcription - .then_some(AudioTranscriptionConfig {}); - let output_audio_transcription = params - .output_audio_transcription - .then_some(AudioTranscriptionConfig {}); + if params + .input_audio_transcription_language_codes + .as_ref() + .is_some_and(Vec::is_empty) + { + bail!("input_audio_transcription_language_codes must not be empty"); + } + + let input_audio_transcription = (params.input_audio_transcription + || params.input_audio_transcription_language_codes.is_some() + || params.input_audio_transcription_mode.is_some()) + .then(|| AudioTranscriptionConfig { + language_codes: params.input_audio_transcription_language_codes.clone(), + custom_vocabulary: None, + mode: params.input_audio_transcription_mode, + }); + let output_audio_transcription = (params.output_audio_transcription + || params.output_audio_transcription_mode.is_some()) + .then(|| AudioTranscriptionConfig { + language_codes: None, + custom_vocabulary: None, + mode: params.output_audio_transcription_mode, + }); if !(text_outputs.text || text_outputs.interim) && (input_audio_transcription.is_some() || output_audio_transcription.is_some()) diff --git a/services/google-dialog/src/lib.rs b/services/google-dialog/src/lib.rs index b32a0642..541b3652 100644 --- a/services/google-dialog/src/lib.rs +++ b/services/google-dialog/src/lib.rs @@ -11,7 +11,10 @@ mod conversation_state; mod types; use client::Client; -pub use types::{Params, ServiceInputEvent, ServiceOutputEvent, VOICES, parse_voice_value}; +pub use types::{ + Params, ServiceInputEvent, ServiceOutputEvent, TranscriptionMode, VOICES, + parse_voice_value, +}; #[derive(Debug)] pub struct GoogleDialog; diff --git a/services/google-dialog/src/types.rs b/services/google-dialog/src/types.rs index cdf693ca..eb305765 100644 --- a/services/google-dialog/src/types.rs +++ b/services/google-dialog/src/types.rs @@ -1,6 +1,8 @@ use gemini_live::types::{FunctionDeclaration, RealtimeInputConfig, ThinkingLevel, Tool}; use serde::{Deserialize, Deserializer, Serialize}; +pub use gemini_live::types::TranscriptionMode; + use anyhow::{Result, bail}; #[derive(Debug, Serialize, Deserialize)] @@ -42,9 +44,15 @@ pub struct Params { /// Enable server-side transcription of user input audio. #[serde(default)] pub input_audio_transcription: bool, + /// BCP-47 language hints for input audio transcription. + pub input_audio_transcription_language_codes: Option>, + /// Transcription style for user input audio. + pub input_audio_transcription_mode: Option, /// Enable server-side transcription of model output audio. #[serde(default)] pub output_audio_transcription: bool, + /// Transcription style for model output audio. + pub output_audio_transcription_mode: Option, } impl Params { @@ -63,7 +71,10 @@ impl Params { tools: vec![], realtime_input_config: None, input_audio_transcription: false, + input_audio_transcription_language_codes: None, + input_audio_transcription_mode: None, output_audio_transcription: false, + output_audio_transcription_mode: None, } } } From 33ba3c58ca096c72b8eb604ac57b0acc09316a9f Mon Sep 17 00:00:00 2001 From: Armin Sander Date: Fri, 18 Sep 2026 16:02:48 +0200 Subject: [PATCH 03/16] google-dialog: support Gemini 3.8 Live models --- .../0006-gemini-3-8-live-model-semantics.md | 66 ++++--- examples/dialog_providers/google.rs | 16 +- .../dialog_providers/google_agent_platform.rs | 12 +- external/gemini-live-rs | 2 +- services/google-dialog/src/client.rs | 111 ++++++++---- services/google-dialog/src/lib.rs | 6 +- services/google-dialog/src/model.rs | 161 ++++++++++++++++++ services/google-dialog/src/types.rs | 69 ++++++-- 8 files changed, 359 insertions(+), 84 deletions(-) create mode 100644 services/google-dialog/src/model.rs diff --git a/docs/adr/0006-gemini-3-8-live-model-semantics.md b/docs/adr/0006-gemini-3-8-live-model-semantics.md index ec3d08e6..a261531b 100644 --- a/docs/adr/0006-gemini-3-8-live-model-semantics.md +++ b/docs/adr/0006-gemini-3-8-live-model-semantics.md @@ -81,6 +81,9 @@ Validate model-specific setup before opening the WebSocket: - `gemini-3.8-live` requires `thinking_level` to be absent. - `gemini-3.8-live-extended-thinking` permits an absent level or `low`, `medium`, or `high`, and rejects `minimal`. +- When the level is absent for Extended Thinking, `google-dialog` delegates to + Google's model default; the current public model documentation does not + specify a fixed default level. - Legacy models retain their existing setup behavior. For tools, declaration behavior and result scheduling are separate concepts. @@ -126,8 +129,9 @@ FunctionCallResult { }, ``` -Omitting `scheduling` means no field is sent and lets Google apply its -`WHEN_IDLE` default where scheduling is supported. The serialized forms are: +Omitting `scheduling` means no field is sent and preserves Google's original +function-response handling for backward compatibility. The serialized forms +are: ```json {"type":"functionCallResult","callId":"call-1","output":{"ok":true}} @@ -153,8 +157,8 @@ declaration is rejected rather than sent as an ignored field. For Extended Thinking, any explicit response schedule is rejected. These checks happen when the service event is handled because the call ID identifies the corresponding declaration. Omitted scheduling is valid for every model. Unknown and legacy -models retain pass-through behavior except that response scheduling is emitted -only on direct Gemini API connections; Agent Platform support is not inferred. +models retain pass-through behavior, and response scheduling is supported on +both direct Gemini API and Agent Platform connections. Expose full-session incremental context as a provider input event named `ClientContent` (the Google wire envelope is `clientContent`). It carries an @@ -251,30 +255,37 @@ pub enum TranscriptionMode { Smart, } -#[serde(default, skip_serializing_if = "Option::is_none")] -pub input_audio_transcription_language_codes: Option>, -#[serde(default, skip_serializing_if = "Option::is_none")] -pub input_audio_transcription_mode: Option, -#[serde(default, skip_serializing_if = "Option::is_none")] -pub output_audio_transcription_mode: Option, +#[serde(default)] +pub input_audio_transcription_language_codes: Vec, +#[serde(default = "default_transcription_mode")] +/// Defaults to `VERBATIM`. +pub input_audio_transcription_mode: TranscriptionMode, +#[serde(default = "default_transcription_mode")] +/// Defaults to `VERBATIM`. +pub output_audio_transcription_mode: TranscriptionMode, ``` The field serializes as `inputAudioTranscriptionLanguageCodes` and maps to Google's `inputAudioTranscription.languageCodes`. Values are BCP-47 language codes used as input ASR hints, not a request to force the model's native audio -response language. `None` leaves language detection automatic and does not -enable input transcription. A non-empty list enables input transcription and -sends the hints; an empty list is rejected as invalid rather than being sent -as an ambiguous configuration. The existing `input_audio_transcription: -bool` remains valid for enabling input transcription without language hints. +response language. The explicit `input_audio_transcription: bool` controls +whether input transcription is enabled; language codes and mode only configure +it when that flag is `true`. The language-code list defaults to empty, and an +empty list means automatic language detection. Both transcription mode fields +default to `VERBATIM`; callers select `SMART` explicitly. The output +transcription boolean likewise controls whether output transcription is +enabled; its mode only configures it when that flag is `true`. The transcription mode fields map to the corresponding configuration's `mode`: -`inputAudioTranscription.mode` and `outputAudioTranscription.mode`. `None` -uses Google's default, `VERBATIM`; `SMART` removes disfluencies, performs light +`inputAudioTranscription.mode` and `outputAudioTranscription.mode`. The public +default is `VERBATIM`; `SMART` removes disfluencies, performs light grammatical cleanup, applies automatic formatting, and makes minor inline corrections. `SMART` cannot be combined with word timestamps or diarization; those controls remain unexposed. Output transcription has no corresponding language-code parameter because native-audio output language selection is -automatic for these models. +automatic for these models. `google-dialog` therefore sends +`outputAudioTranscription.languageCodes` as absent and derives the output +transcript language from the generated audio; callers cannot use output +transcription configuration to select or override that language. `google-dialog` re-exports the complete new surface from its crate root: @@ -342,8 +353,9 @@ controls: - Response modality is audio. Text output is obtained through output-audio transcription; callers cannot select a text-only Gemini response modality. -- Proactive audio is always enabled by both 3.8 models. The service exposes no - toggle because Google rejects attempts to disable it. +- Proactive audio is permanently enabled by both 3.8 models. The service does + not send a `proactivity` override; Google documents that setting + `proactive_audio: false` as an error. - Native-audio language selection is automatic. Callers can guide language in the system instructions, but Google does not accept an explicit language code for these models. @@ -352,9 +364,12 @@ The following Gemini capabilities are not exposed because both 3.8 Live model pages mark them unsupported, not because this client withholds a supported feature: context caching, code execution, file search, Google Maps grounding, image generation, structured output, and URL context. The models also do not -support the Batch API. Affective dialog was removed from the 3.8 Live API and is -therefore not configurable. Google Search grounding and function calling are -supported and remain exposed; they are not part of this exclusion list. +support the Batch API. Affective dialog is removed from both 3.8 Live model +contracts and is therefore not configurable. The general Live capabilities +guide currently describes affective dialog and proactive audio differently; +the model-specific 3.8 pages are treated as authoritative here. Google Search +grounding and function calling are supported and remain exposed; they are not +part of this exclusion list. Keep context-window compression enabled with Google's defaults and retain automatic session resumption. Compression manages context growth; resumption @@ -380,9 +395,8 @@ independently of any Google SDK convenience behavior: 2. Omit unsupported setup fields instead of serializing null or default values. In particular, omit the entire thinking configuration for standard 3.8. 3. Use audio as the response modality. Enable output-audio transcription when a - text transcript is required, and map input language hints to - `inputAudioTranscription.languageCodes` while enabling input transcription - when a non-empty hint list is supplied. + text transcript is required, and when input transcription is enabled, map + input language hints to `inputAudioTranscription.languageCodes`. 4. Deserialize `interactionStatus` from `serverContent`, alongside `turnComplete`, and retain both values when decomposing a server message into semantic events. diff --git a/examples/dialog_providers/google.rs b/examples/dialog_providers/google.rs index cae0c100..526268be 100644 --- a/examples/dialog_providers/google.rs +++ b/examples/dialog_providers/google.rs @@ -4,7 +4,7 @@ use anyhow::{Context, Result}; use async_trait::async_trait; use gemini_live::types as gemini_types; use google_dialog::{ - GoogleDialog, ServiceInputEvent as GoogleServiceInputEvent, ServiceOutputEvent, + GEMINI_3_8_LIVE, GoogleDialog, ServiceInputEvent as GoogleServiceInputEvent, ServiceOutputEvent, }; use reqwest::Url; use serde::Deserialize; @@ -29,7 +29,7 @@ impl ProviderApi for GoogleProvider { .model .or_else(|| env::var("GEMINI_LIVE_API_MODEL").ok()) .filter(|model| !model.trim().is_empty()) - .unwrap_or_else(|| "gemini-3.1-flash-live-preview".to_owned()); + .unwrap_or_else(|| GEMINI_3_8_LIVE.to_owned()); let mut params = google_dialog::Params::new(model); params.api_key = Some(key); @@ -64,6 +64,10 @@ impl ProviderApi for GoogleProvider { tracing::info!("Turn complete"); Ok(None) } + ServiceOutputEvent::InteractionInProgress => { + tracing::info!("Interaction remains in progress"); + Ok(None) + } ServiceOutputEvent::ToolCallCancellation { call_id } => { tracing::info!("Tool call cancelled: {call_id}"); Ok(None) @@ -73,8 +77,12 @@ impl ProviderApi for GoogleProvider { fn function_result_event(&self, call_id: String, result: String) -> Result { let output = json!({ "time": serde_json::Value::String(result) }); - serde_json::to_value(&GoogleServiceInputEvent::FunctionCallResult { call_id, output }) - .map_err(Into::into) + serde_json::to_value(&GoogleServiceInputEvent::FunctionCallResult { + call_id, + output, + scheduling: None, + }) + .map_err(Into::into) } fn output_format(&self, _input_format: AudioFormat) -> AudioFormat { diff --git a/examples/dialog_providers/google_agent_platform.rs b/examples/dialog_providers/google_agent_platform.rs index 31571597..7d57ee75 100644 --- a/examples/dialog_providers/google_agent_platform.rs +++ b/examples/dialog_providers/google_agent_platform.rs @@ -79,6 +79,10 @@ impl ProviderApi for GoogleAgentPlatformProvider { tracing::info!("Turn complete"); Ok(None) } + ServiceOutputEvent::InteractionInProgress => { + tracing::info!("Interaction remains in progress"); + Ok(None) + } ServiceOutputEvent::ToolCallCancellation { call_id } => { tracing::info!("Tool call cancelled: {call_id}"); Ok(None) @@ -88,8 +92,12 @@ impl ProviderApi for GoogleAgentPlatformProvider { fn function_result_event(&self, call_id: String, result: String) -> Result { let output = json!({ "time": serde_json::Value::String(result) }); - serde_json::to_value(&GoogleServiceInputEvent::FunctionCallResult { call_id, output }) - .map_err(Into::into) + serde_json::to_value(&GoogleServiceInputEvent::FunctionCallResult { + call_id, + output, + scheduling: None, + }) + .map_err(Into::into) } fn output_format(&self, _input_format: AudioFormat) -> AudioFormat { diff --git a/external/gemini-live-rs b/external/gemini-live-rs index 814d6540..0bce2527 160000 --- a/external/gemini-live-rs +++ b/external/gemini-live-rs @@ -1 +1 @@ -Subproject commit 814d654000e88812d6aea3771a471ddfcb5b4898 +Subproject commit 0bce252778060b2c7e9e582a527a0c658592f5d2 diff --git a/services/google-dialog/src/client.rs b/services/google-dialog/src/client.rs index 35786303..ae0137bc 100644 --- a/services/google-dialog/src/client.rs +++ b/services/google-dialog/src/client.rs @@ -18,7 +18,8 @@ use context_switch_core::{ }; use crate::conversation_state::ConversationState; -use crate::{Params, ServiceInputEvent, ServiceOutputEvent, TextOutputs}; +use crate::model; +use crate::{ClientContentRole, Params, ServiceInputEvent, ServiceOutputEvent, TextOutputs}; const LEGACY_TOOL_CALL_ID: &str = "legacy-tool-call"; @@ -107,11 +108,19 @@ impl Client { .context("Sending text to Gemini Live")?; } Input::ServiceEvent { value } => match serde_json::from_value(value)? { - ServiceInputEvent::FunctionCallResult { call_id, output } => { + ServiceInputEvent::FunctionCallResult { + call_id, + output, + scheduling, + } => { let Some(name) = state.tool_calls.resolve(&call_id)? else { return Ok(()); }; + // Gemini 3.8 adds scheduled responses for non-blocking tools; + // Extended Thinking deliberately rejects this field. + model::validate_response_scheduling(&self.params, &name, scheduling.as_ref())?; + let response = normalize_function_response(output); let response_call_id = if call_id == LEGACY_TOOL_CALL_ID { None @@ -123,6 +132,7 @@ impl Client { id: response_call_id, name, response, + scheduling, }; session .send_tool_response(vec![response]) @@ -133,6 +143,31 @@ impl Client { info!("Received prompt"); session.send_text(&text).await.context("Sending prompt")?; } + ServiceInputEvent::ClientContent { + role, + text, + turn_complete, + } => { + // Gemini 3.8 adds full-session clientContent updates. The + // same envelope is supported on direct and Agent Platform routes. + let role = match role { + ClientContentRole::User => "user", + ClientContentRole::Model => "model", + }; + session + .send_client_content(gemini_live::types::ClientContent { + turns: Some(vec![Content { + role: Some(role.to_owned()), + parts: vec![Part { + text: Some(text), + inline_data: None, + }], + }]), + turn_complete: Some(turn_complete), + }) + .await + .context("Sending client content to Gemini Live")?; + } }, } Ok(()) @@ -170,11 +205,21 @@ impl Client { self.finalize_output_transcription(text_outputs, output, state)?; output.service_event(OutputPath::Media, ServiceOutputEvent::TurnComplete)?; } + ServerEvent::InteractionInProgress => { + self.finalize_output_transcription(text_outputs, output, state)?; + output + .service_event(OutputPath::Media, ServiceOutputEvent::InteractionInProgress)?; + } ServerEvent::Interrupted => { // We expect a TurnComplete afterward, so don't finalize the output transcription // when interrupted. output.clear_audio()?; } + ServerEvent::InterimInputTranscription(text) => { + if self.params.input_audio_transcription && text_outputs.interim { + output.text(false, text, None, None)?; + } + } ServerEvent::InputTranscription(text) => { if self.params.input_audio_transcription { if text_outputs.text { @@ -190,11 +235,6 @@ impl Client { } } ServerEvent::InputTranscriptionFinished => {} - ServerEvent::InterimInputTranscription(text) => { - if self.params.input_audio_transcription && text_outputs.interim { - output.text(false, text, None, None)?; - } - } ServerEvent::OutputTranscription(text) => { if self.params.output_audio_transcription { state.output_transcription_buffer.push_str(&text); @@ -344,29 +384,30 @@ fn session_config(params: &Params, text_outputs: TextOutputs) -> Result Result { - if params - .input_audio_transcription_language_codes - .as_ref() - .is_some_and(Vec::is_empty) - { - bail!("input_audio_transcription_language_codes must not be empty"); - } - - let input_audio_transcription = (params.input_audio_transcription - || params.input_audio_transcription_language_codes.is_some() - || params.input_audio_transcription_mode.is_some()) - .then(|| AudioTranscriptionConfig { - language_codes: params.input_audio_transcription_language_codes.clone(), - custom_vocabulary: None, - mode: params.input_audio_transcription_mode, - }); - let output_audio_transcription = (params.output_audio_transcription - || params.output_audio_transcription_mode.is_some()) - .then(|| AudioTranscriptionConfig { - language_codes: None, - custom_vocabulary: None, - mode: params.output_audio_transcription_mode, - }); + // Gemini 3.8 introduced model-specific thinking policies: standard Live + // omits thinking_config, while Extended Thinking accepts low/medium/high. + model::validate_thinking_level(params)?; + + let input_audio_transcription_language_codes = + (!params.input_audio_transcription_language_codes.is_empty()) + .then(|| params.input_audio_transcription_language_codes.clone()); + + let input_audio_transcription = + params + .input_audio_transcription + .then(|| AudioTranscriptionConfig { + language_codes: input_audio_transcription_language_codes, + custom_vocabulary: None, + mode: Some(params.input_audio_transcription_mode), + }); + let output_audio_transcription = + params + .output_audio_transcription + .then(|| AudioTranscriptionConfig { + language_codes: None, + custom_vocabulary: None, + mode: Some(params.output_audio_transcription_mode), + }); if !(text_outputs.text || text_outputs.interim) && (input_audio_transcription.is_some() || output_audio_transcription.is_some()) @@ -376,6 +417,12 @@ fn setup_config(params: &Params, text_outputs: TextOutputs) -> Result Result Result Option { + match model { + GEMINI_3_8_LIVE => Some(ModelConfig { + // Fixed interleaved reasoning; `thinking_level` is not accepted. + thinking: ThinkingPolicy::Disabled, + // Both legacy blocking and asynchronous tools are supported. + functions: FunctionPolicy::BlockingAndNonBlocking, + // `INTERRUPT`, `WHEN_IDLE`, and `SILENT` are supported for + // non-blocking function responses. + response_scheduling: SchedulingPolicy::SupportedForNonBlocking, + }), + GEMINI_3_8_LIVE_EXTENDED_THINKING => Some(ModelConfig { + // Background reasoning accepts low, medium, and high levels, but + // not `minimal`. + thinking: ThinkingPolicy::Optional { + allows_minimal: false, + }, + // Tools must run in the background while the model reasons. + functions: FunctionPolicy::NonBlockingOnly, + // Function-response scheduling is not accepted. + response_scheduling: SchedulingPolicy::Unsupported, + }), + _ => None, + } +} + +pub fn validate_thinking_level(params: &Params) -> Result<()> { + let Some(config) = model_config(¶ms.model) else { + return Ok(()); + }; + + match (config.thinking, params.thinking_level) { + (ThinkingPolicy::Disabled, Some(level)) => bail!( + "Model `{}` does not support thinking_level `{level}`; omit thinking_level", + params.model + ), + (ThinkingPolicy::Optional { allows_minimal }, Some(ThinkingLevel::Minimal)) + if !allows_minimal => + { + bail!( + "Model `{}` does not support thinking_level `minimal`", + params.model + ) + } + _ => Ok(()), + } +} + +pub fn tools_for_model(model: &str, input_tools: &[Tool]) -> Result> { + let mut tools = input_tools.to_vec(); + let Some(config) = model_config(model) else { + return Ok(tools); + }; + if !matches!(config.functions, FunctionPolicy::NonBlockingOnly) { + return Ok(tools); + } + + for tool in &mut tools { + let Tool::FunctionDeclarations(declarations) = tool else { + continue; + }; + for declaration in declarations { + match declaration.behavior.clone() { + Some(FunctionBehavior::Blocking) => bail!( + "Model `{}` requires non-blocking function declarations", + model + ), + None => declaration.behavior = Some(FunctionBehavior::NonBlocking), + Some(FunctionBehavior::NonBlocking) => {} + } + } + } + Ok(tools) +} + +pub fn validate_response_scheduling( + params: &Params, + function_name: &str, + scheduling: Option<&FunctionResponseScheduling>, +) -> Result<()> { + let Some(_scheduling) = scheduling else { + return Ok(()); + }; + let Some(config) = model_config(¶ms.model) else { + return Ok(()); + }; + match config.response_scheduling { + SchedulingPolicy::Unsupported => bail!( + "Model `{}` does not support function response scheduling", + params.model + ), + SchedulingPolicy::SupportedForNonBlocking + if function_behavior(¶ms.tools, function_name) + == Some(FunctionBehavior::Blocking) => + { + bail!( + "Function response scheduling is not valid for blocking function `{function_name}`" + ) + } + SchedulingPolicy::SupportedForNonBlocking => {} + } + Ok(()) +} + +fn function_behavior(tools: &[Tool], function_name: &str) -> Option { + tools.iter().find_map(|tool| { + let Tool::FunctionDeclarations(declarations) = tool else { + return None; + }; + declarations + .iter() + .find(|declaration| declaration.name == function_name) + .and_then(|declaration| declaration.behavior.clone()) + }) +} diff --git a/services/google-dialog/src/types.rs b/services/google-dialog/src/types.rs index eb305765..33626b40 100644 --- a/services/google-dialog/src/types.rs +++ b/services/google-dialog/src/types.rs @@ -1,9 +1,8 @@ -use gemini_live::types::{FunctionDeclaration, RealtimeInputConfig, ThinkingLevel, Tool}; +use anyhow::{Result, bail}; use serde::{Deserialize, Deserializer, Serialize}; -pub use gemini_live::types::TranscriptionMode; - -use anyhow::{Result, bail}; +pub use gemini_live::types::{FunctionBehavior, FunctionResponseScheduling, TranscriptionMode}; +use gemini_live::types::{FunctionDeclaration, RealtimeInputConfig, ThinkingLevel, Tool}; #[derive(Debug, Serialize, Deserialize)] #[serde(rename_all = "camelCase")] @@ -31,8 +30,11 @@ pub struct Params { /// Sampling temperature. Valid range: `0.0..=2.0`. /// If omitted, Gemini uses the model-specific default temperature. pub temperature: Option, - /// Gemini 3.1 thinking level (`minimal`, `low`, `medium`, or `high`). - /// In Live API, Gemini 3.1 defaults to `minimal` when omitted. + /// Thinking level for Gemini 3.1 and Gemini 3.8 Extended Thinking + /// (`minimal`, `low`, `medium`, or `high`, subject to model support). + /// When omitted, Google applies the model default. `gemini-3.8-live` + /// requires this field to remain omitted; Extended Thinking accepts + /// `low`, `medium`, or `high`. pub thinking_level: Option, /// Enabled by default to avoid context-window exhaustion during long audio sessions. #[serde(default = "default_context_window_compression")] @@ -45,14 +47,25 @@ pub struct Params { #[serde(default)] pub input_audio_transcription: bool, /// BCP-47 language hints for input audio transcription. - pub input_audio_transcription_language_codes: Option>, - /// Transcription style for user input audio. - pub input_audio_transcription_mode: Option, + #[serde(default)] + pub input_audio_transcription_language_codes: Vec, + /// Transcription style for user input audio. Defaults to `VERBATIM`. + #[serde(default = "default_transcription_mode")] + pub input_audio_transcription_mode: TranscriptionMode, /// Enable server-side transcription of model output audio. #[serde(default)] pub output_audio_transcription: bool, - /// Transcription style for model output audio. - pub output_audio_transcription_mode: Option, + /// Transcription style for model output audio. Defaults to `VERBATIM`. + #[serde(default = "default_transcription_mode")] + pub output_audio_transcription_mode: TranscriptionMode, +} + +fn default_context_window_compression() -> bool { + true +} + +fn default_transcription_mode() -> TranscriptionMode { + TranscriptionMode::Verbatim } impl Params { @@ -71,10 +84,10 @@ impl Params { tools: vec![], realtime_input_config: None, input_audio_transcription: false, - input_audio_transcription_language_codes: None, - input_audio_transcription_mode: None, + input_audio_transcription_language_codes: vec![], + input_audio_transcription_mode: default_transcription_mode(), output_audio_transcription: false, - output_audio_transcription_mode: None, + output_audio_transcription_mode: default_transcription_mode(), } } } @@ -124,10 +137,6 @@ pub fn parse_voice_value(value: &str) -> Result { } } -fn default_context_window_compression() -> bool { - true -} - fn deserialize_tools<'de, D>(deserializer: D) -> Result, D::Error> where D: Deserializer<'de>, @@ -196,12 +205,34 @@ pub enum ServiceInputEvent { FunctionCallResult { call_id: String, output: serde_json::Value, + /// Gemini 3.8 scheduling for a non-blocking function response. + #[serde(default, skip_serializing_if = "Option::is_none")] + scheduling: Option, + }, + /// Gemini 3.8 incremental conversation content sent during a live session. + ClientContent { + /// Author of the appended conversation content. + role: ClientContentRole, + /// Text for one conversation content part; empty text is allowed. + text: String, + /// Start generation after appending the content and interrupt active generation. + #[serde(default)] + turn_complete: bool, }, Prompt { text: String, }, } +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "lowercase")] +pub enum ClientContentRole { + /// Content supplied as user-authored conversation context. + User, + /// Content supplied as model-authored conversation context. + Model, +} + #[derive(Debug, Serialize, Deserialize)] #[serde( tag = "type", @@ -218,6 +249,8 @@ pub enum ServiceOutputEvent { call_id: String, }, TurnComplete, + /// Gemini 3.8 indicates that the turn ended while the interaction continues. + InteractionInProgress, } #[cfg(test)] From 7df9f44ecb9529790940c5dbfda05816c41ff883 Mon Sep 17 00:00:00 2001 From: Armin Sander Date: Fri, 18 Sep 2026 16:25:51 +0200 Subject: [PATCH 04/16] gemini-live: update split protocol history --- external/gemini-live-rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/external/gemini-live-rs b/external/gemini-live-rs index 0bce2527..d78ea1bc 160000 --- a/external/gemini-live-rs +++ b/external/gemini-live-rs @@ -1 +1 @@ -Subproject commit 0bce252778060b2c7e9e582a527a0c658592f5d2 +Subproject commit d78ea1bc123feaff22bfeca115b7aacfd63270f0 From 98a454a145c7fc4c1dd81048a5c2dc200d955f51 Mon Sep 17 00:00:00 2001 From: Armin Sander Date: Fri, 18 Sep 2026 16:27:55 +0200 Subject: [PATCH 05/16] clippy --- services/google-dialog/src/client.rs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/services/google-dialog/src/client.rs b/services/google-dialog/src/client.rs index ae0137bc..8f49513a 100644 --- a/services/google-dialog/src/client.rs +++ b/services/google-dialog/src/client.rs @@ -395,7 +395,7 @@ fn setup_config(params: &Params, text_outputs: TextOutputs) -> Result Result Date: Fri, 18 Sep 2026 16:35:38 +0200 Subject: [PATCH 06/16] google-dialog: enforce Gemini tool scheduling policies --- external/gemini-live-rs | 2 +- services/google-dialog/src/model.rs | 9 ++++++++- 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/external/gemini-live-rs b/external/gemini-live-rs index d78ea1bc..e6bd1dde 160000 --- a/external/gemini-live-rs +++ b/external/gemini-live-rs @@ -1 +1 @@ -Subproject commit d78ea1bc123feaff22bfeca115b7aacfd63270f0 +Subproject commit e6bd1dde907e9e744d2fb32fdb0079c865b02697 diff --git a/services/google-dialog/src/model.rs b/services/google-dialog/src/model.rs index 8306f16a..a994866d 100644 --- a/services/google-dialog/src/model.rs +++ b/services/google-dialog/src/model.rs @@ -106,6 +106,12 @@ pub fn tools_for_model(model: &str, input_tools: &[Tool]) -> Result> { continue; }; for declaration in declarations { + if declaration.scheduling.is_some() { + bail!( + "Model `{}` does not support function declaration scheduling", + model + ); + } match declaration.behavior.clone() { Some(FunctionBehavior::Blocking) => bail!( "Model `{}` requires non-blocking function declarations", @@ -137,7 +143,8 @@ pub fn validate_response_scheduling( ), SchedulingPolicy::SupportedForNonBlocking if function_behavior(¶ms.tools, function_name) - == Some(FunctionBehavior::Blocking) => + .unwrap_or(FunctionBehavior::Blocking) + == FunctionBehavior::Blocking => { bail!( "Function response scheduling is not valid for blocking function `{function_name}`" From 3a35fb0a317f9e6bf0335131202cb0fd2f26e3c8 Mon Sep 17 00:00:00 2001 From: Armin Sander Date: Fri, 18 Sep 2026 16:45:03 +0200 Subject: [PATCH 07/16] google-dialog: finalize Gemini 3.8 release metadata --- Cargo.toml | 2 +- VERSIONING.md | 1 + services/google-dialog/src/types.rs | 1 + 3 files changed, 3 insertions(+), 1 deletion(-) diff --git a/Cargo.toml b/Cargo.toml index e0611fea..ac78222c 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -25,7 +25,7 @@ members = [ ] [workspace.package] -version = "3.8.1" +version = "3.9.0" edition = "2024" license = "MIT" repository = "https://github.com/pragmatrix/context-switch" diff --git a/VERSIONING.md b/VERSIONING.md index 33eb9cff..fe63e5d8 100644 --- a/VERSIONING.md +++ b/VERSIONING.md @@ -19,5 +19,6 @@ root `Cargo.toml`); all member crates inherit it via `version.workspace = true`. | Version | Change | |---------|--------| +| 3.9.0 | google-dialog: Gemini 3.8 Live models and interaction semantics | | 3.8.0 | google-transcribe: numerals support via digit-sequence class token (#99) | | 3.8.1 | audio-knife: startup self-test warning for `GOOGLE_APPLICATION_CREDENTIALS` (#100) | diff --git a/services/google-dialog/src/types.rs b/services/google-dialog/src/types.rs index 33626b40..76781348 100644 --- a/services/google-dialog/src/types.rs +++ b/services/google-dialog/src/types.rs @@ -47,6 +47,7 @@ pub struct Params { #[serde(default)] pub input_audio_transcription: bool, /// BCP-47 language hints for input audio transcription. + /// An explicit `null` is invalid; omit the field to use no language hints. #[serde(default)] pub input_audio_transcription_language_codes: Vec, /// Transcription style for user input audio. Defaults to `VERBATIM`. From b4b13bc8219c2c2af63776bdb21dfca56c0c9244 Mon Sep 17 00:00:00 2001 From: Armin Sander Date: Fri, 18 Sep 2026 17:01:02 +0200 Subject: [PATCH 08/16] example(dialog): add input/output-transcription CLI overrides, off by default - Add --input-transcription/--output-transcription flags to the dialog example; providers (google, google-agent-platform, openai) fall back to off when the flags are omitted instead of hardcoding transcription on. - Demote the transcription-events-while-disabled logging in google-dialog from warn to trace: 3.8 sends transcription events regardless of the setup flag. --- examples/dialog.rs | 10 ++++++++++ examples/dialog_providers/google.rs | 4 ++-- .../dialog_providers/google_agent_platform.rs | 4 ++-- examples/dialog_providers/mod.rs | 4 ++++ examples/dialog_providers/openai.rs | 4 ++-- services/google-dialog/src/client.rs | 16 ++++++++-------- 6 files changed, 28 insertions(+), 14 deletions(-) diff --git a/examples/dialog.rs b/examples/dialog.rs index bb7cee0c..bbf9f39c 100644 --- a/examples/dialog.rs +++ b/examples/dialog.rs @@ -46,6 +46,14 @@ struct Cli { /// Used only with provider `google-agent-platform`. #[arg(long)] location: Option, + /// Override whether server-side input transcription is enabled. Defaults to + /// off for providers that enable it by default. + #[arg(long)] + input_transcription: Option, + /// Override whether server-side output transcription is enabled. Defaults to + /// off for providers that enable it by default. + #[arg(long)] + output_transcription: Option, } #[derive(Debug, Clone, Copy, ValueEnum)] @@ -299,6 +307,8 @@ async fn start_conversation(cli: &Cli, conversation: Conversation) -> Result<()> voice: cli.voice.clone(), project: cli.project.clone(), location: cli.location.clone(), + input_transcription: cli.input_transcription, + output_transcription: cli.output_transcription, }; cli.provider .api() diff --git a/examples/dialog_providers/google.rs b/examples/dialog_providers/google.rs index 526268be..9c2eff15 100644 --- a/examples/dialog_providers/google.rs +++ b/examples/dialog_providers/google.rs @@ -42,8 +42,8 @@ impl ProviderApi for GoogleProvider { .as_deref() .map(google_dialog::parse_voice_value) .transpose()?; - params.input_audio_transcription = true; - params.output_audio_transcription = true; + params.input_audio_transcription = request.input_transcription.unwrap_or_default(); + params.output_audio_transcription = request.output_transcription.unwrap_or_default(); params.tools.push(get_time_tool()); GoogleDialog.conversation(params, conversation).await diff --git a/examples/dialog_providers/google_agent_platform.rs b/examples/dialog_providers/google_agent_platform.rs index 7d57ee75..5e533ddf 100644 --- a/examples/dialog_providers/google_agent_platform.rs +++ b/examples/dialog_providers/google_agent_platform.rs @@ -57,8 +57,8 @@ impl ProviderApi for GoogleAgentPlatformProvider { .as_deref() .map(google_dialog::parse_voice_value) .transpose()?; - params.input_audio_transcription = true; - params.output_audio_transcription = true; + params.input_audio_transcription = request.input_transcription.unwrap_or_default(); + params.output_audio_transcription = request.output_transcription.unwrap_or_default(); params.tools.push(get_time_tool()); GoogleDialog.conversation(params, conversation).await diff --git a/examples/dialog_providers/mod.rs b/examples/dialog_providers/mod.rs index cbe4b0ee..c828015b 100644 --- a/examples/dialog_providers/mod.rs +++ b/examples/dialog_providers/mod.rs @@ -17,6 +17,10 @@ pub struct StartConversationRequest { pub voice: Option, pub project: Option, pub location: Option, + /// Command-line transcription overrides; `Ok`/`Err` explicit on/off, + /// `None` keeps the provider default. + pub input_transcription: Option, + pub output_transcription: Option, } #[async_trait(?Send)] diff --git a/examples/dialog_providers/openai.rs b/examples/dialog_providers/openai.rs index 521798d2..2b9cfea7 100644 --- a/examples/dialog_providers/openai.rs +++ b/examples/dialog_providers/openai.rs @@ -47,8 +47,8 @@ impl ProviderApi for OpenAIProvider { .map(parse_realtime_voice_value) .transpose()?; params.tools.push(get_time_function_definition()); - params.input_audio_transcription = true; - params.output_audio_transcription = true; + params.input_audio_transcription = request.input_transcription.unwrap_or_default(); + params.output_audio_transcription = request.output_transcription.unwrap_or_default(); OpenAIDialog.conversation(params, conversation).await } diff --git a/services/google-dialog/src/client.rs b/services/google-dialog/src/client.rs index 8f49513a..6188a2d4 100644 --- a/services/google-dialog/src/client.rs +++ b/services/google-dialog/src/client.rs @@ -226,11 +226,11 @@ impl Client { output.text(true, text, None, None)?; } } else { - // Observed with preview Gemini models: transcription events can still arrive - // even when transcription is not enabled in setup. - warn!( + // Observed with preview Gemini models and 3.8: transcription events can still + // arrive even when transcription is not enabled in setup. + trace!( transcript_len = text.len(), - "Received input transcription event while input_audio_transcription is disabled (observed with preview model)" + "Received input transcription event while input_audio_transcription is disabled" ); } } @@ -247,11 +247,11 @@ impl Client { )?; } } else { - // Observed with preview Gemini models: transcription events can still arrive - // even when transcription is not enabled in setup. - warn!( + // Observed with preview Gemini models and 3.8: transcription events can still + // arrive even when transcription is not enabled in setup. + trace!( transcript_len = text.len(), - "Received output transcription event while output_audio_transcription is disabled (observed with preview model)" + "Received output transcription event while output_audio_transcription is disabled" ); } } From 83d0e6dcc241bc5c3860dbe38b6621438bf58e11 Mon Sep 17 00:00:00 2001 From: Armin Sander Date: Fri, 18 Sep 2026 18:19:03 +0200 Subject: [PATCH 09/16] docs: record GoAway deadline handling --- docs/adr/0006-gemini-3-8-live-model-semantics.md | 9 ++++++++- external/gemini-live-rs | 2 +- 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/docs/adr/0006-gemini-3-8-live-model-semantics.md b/docs/adr/0006-gemini-3-8-live-model-semantics.md index a261531b..c3475efd 100644 --- a/docs/adr/0006-gemini-3-8-live-model-semantics.md +++ b/docs/adr/0006-gemini-3-8-live-model-semantics.md @@ -384,6 +384,12 @@ be checkpoint-safe: - If the socket is lost while state is not resumable, fail with an explicit possible-state-loss error instead of restoring stale state or silently starting a new session. +- If `GoAway.timeLeft` expires before a valid checkpoint arrives, emit an API + error, close the socket, and mark the session closed. Do not attempt either a + resumed or fresh reconnect; callers must establish a new session explicitly. +- Defer ordinary outbound commands after `GoAway` so they cannot be sent to a + session that is about to terminate. Tool responses and an explicit close + remain allowed while waiting for the checkpoint. ## Raw WebSocket client requirements @@ -413,7 +419,8 @@ independently of any Google SDK convenience behavior: `resumable: false`; possession of an older handle is not proof that current in-flight work can be restored. 10. Treat `GoAway.timeLeft` as a reconnect deadline while continuing to receive - updates needed to obtain a safe checkpoint. + updates needed to obtain a safe checkpoint. Expiry without a checkpoint is + a terminal client-visible failure, not permission to start a fresh session. ## Consequences diff --git a/external/gemini-live-rs b/external/gemini-live-rs index e6bd1dde..ebf7d8a0 160000 --- a/external/gemini-live-rs +++ b/external/gemini-live-rs @@ -1 +1 @@ -Subproject commit e6bd1dde907e9e744d2fb32fdb0079c865b02697 +Subproject commit ebf7d8a03c57a25fa8da5991c933424cd28513ae From a8e0842d035bf53487fd5e244b4f4d358cf41cf0 Mon Sep 17 00:00:00 2001 From: Armin Sander Date: Fri, 18 Sep 2026 18:34:09 +0200 Subject: [PATCH 10/16] google-dialog: default extended thinking to medium --- services/google-dialog/src/client.rs | 7 +++++-- services/google-dialog/src/model.rs | 7 +++++++ 2 files changed, 12 insertions(+), 2 deletions(-) diff --git a/services/google-dialog/src/client.rs b/services/google-dialog/src/client.rs index 6188a2d4..0d637348 100644 --- a/services/google-dialog/src/client.rs +++ b/services/google-dialog/src/client.rs @@ -1,7 +1,7 @@ use std::mem; use anyhow::{Context, Result, anyhow, bail}; -use tracing::{debug, info, trace, warn}; +use tracing::{debug, info, trace}; use gemini_live::transport::{Auth, Endpoint, TransportConfig}; use gemini_live::types::{ @@ -386,6 +386,9 @@ fn session_config(params: &Params, text_outputs: TextOutputs) -> Result Result { // Gemini 3.8 introduced model-specific thinking policies: standard Live // omits thinking_config, while Extended Thinking accepts low/medium/high. + let thinking_level = params + .thinking_level + .or_else(|| model::default_thinking_level(¶ms.model)); model::validate_thinking_level(params)?; let input_audio_transcription_language_codes = @@ -436,7 +439,7 @@ fn setup_config(params: &Params, text_outputs: TextOutputs) -> Result Option { } } +pub fn default_thinking_level(model: &str) -> Option { + match model { + GEMINI_3_8_LIVE_EXTENDED_THINKING => Some(ThinkingLevel::Medium), + _ => None, + } +} + pub fn validate_thinking_level(params: &Params) -> Result<()> { let Some(config) = model_config(¶ms.model) else { return Ok(()); From fab79e8c6c7b5237a59840b97354bd6894edd092 Mon Sep 17 00:00:00 2001 From: Armin Sander Date: Mon, 21 Sep 2026 17:26:49 +0200 Subject: [PATCH 11/16] google-dialog: document Prompt and ClientContent input semantics --- services/google-dialog/src/types.rs | 39 ++++++++++++++++++++++++++--- 1 file changed, 36 insertions(+), 3 deletions(-) diff --git a/services/google-dialog/src/types.rs b/services/google-dialog/src/types.rs index 76781348..96f5b8b7 100644 --- a/services/google-dialog/src/types.rs +++ b/services/google-dialog/src/types.rs @@ -93,6 +93,14 @@ impl Params { } } +/// The 30 prebuilt Gemini Live API output voices, kept in Google's documented +/// order (see https://ai.google.dev/gemini-api/docs/speech-generation#voices). +/// One flat list for every native-audio Live model: Google publishes no +/// per-model voice subsetting for Live API (`gemini-3.8-live`, +/// `gemini-3.8-live-extended-thinking`, and Agent Platform-routed models all +/// take the same set). Note the `generateContent` TTS voice set is slightly +/// different and does not apply here; this service only speaks the Live +/// WebSocket protocol. pub const VOICES: &[&str] = &[ "Zephyr", "Puck", @@ -196,6 +204,17 @@ enum OpenAiToolType { Function, } +/// Choice pattern for the text input variants: +/// +/// - [`ServiceInputEvent::Prompt`]: talk to the model now, like a user speaking. +/// - [`ServiceInputEvent::ClientContent`] with [`ClientContentRole::User`] and +/// `turn_complete: true`: ask or instruct the model for an immediate response +/// (interrupts active generation). +/// - [`ServiceInputEvent::ClientContent`] with [`ClientContentRole::User`] and +/// `turn_complete: false`: add context silently; the response comes later +/// from the audio flow. +/// - [`ServiceInputEvent::ClientContent`] with [`ClientContentRole::Model`]: +/// restore or fabricate an earlier assistant turn. #[derive(Debug, Serialize, Deserialize)] #[serde( tag = "type", @@ -211,6 +230,13 @@ pub enum ServiceInputEvent { scheduling: Option, }, /// Gemini 3.8 incremental conversation content sent during a live session. + /// + /// Appends an ordered history entry via the `clientContent` wire message. + /// With `turn_complete: false` (the default) the content is added and + /// generation stays pending; `true` starts generation immediately and + /// intentionally interrupts active generation. Typical usage: seeding or + /// restoring context without audio, scripted turns, and deterministic + /// prompt delivery. Not a realtime input path. ClientContent { /// Author of the appended conversation content. role: ClientContentRole, @@ -220,9 +246,16 @@ pub enum ServiceInputEvent { #[serde(default)] turn_complete: bool, }, - Prompt { - text: String, - }, + /// Realtime user text input sent as the `realtimeInput.text` wire message. + /// + /// Behaves like the user just said the text: the model interprets it and + /// responds subject to turn state, without a guaranteed interrupt of + /// active generation or deterministic ordering against the audio stream. + /// Always user-side; cannot insert model history. Typical usage: typed + /// live input in an audio session (including instruction-style text such + /// as "Say: Hello"). For exact synthesis or scripted turns use + /// [`ServiceInputEvent::ClientContent`] instead. + Prompt { text: String }, } #[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] From d1e29d7b2449344ef64e2d200c093e489ef8fbff Mon Sep 17 00:00:00 2001 From: Armin Sander Date: Mon, 21 Sep 2026 17:34:44 +0200 Subject: [PATCH 12/16] adr: update Gemini 3.8 ADR with feature overview and input-event intent docs --- .../0006-gemini-3-8-live-model-semantics.md | 67 +++++++++++++++++-- 1 file changed, 62 insertions(+), 5 deletions(-) diff --git a/docs/adr/0006-gemini-3-8-live-model-semantics.md b/docs/adr/0006-gemini-3-8-live-model-semantics.md index c3475efd..9844e25e 100644 --- a/docs/adr/0006-gemini-3-8-live-model-semantics.md +++ b/docs/adr/0006-gemini-3-8-live-model-semantics.md @@ -81,9 +81,10 @@ Validate model-specific setup before opening the WebSocket: - `gemini-3.8-live` requires `thinking_level` to be absent. - `gemini-3.8-live-extended-thinking` permits an absent level or `low`, `medium`, or `high`, and rejects `minimal`. -- When the level is absent for Extended Thinking, `google-dialog` delegates to - Google's model default; the current public model documentation does not - specify a fixed default level. +- When the level is absent for Extended Thinking, `google-dialog` sends + `medium` itself. Google's public model documentation does not specify a + fixed default level, so the service pins `medium` deterministically rather + than depending on an unspecified server default. - Legacy models retain their existing setup behavior. For tools, declaration behavior and result scheduling are separate concepts. @@ -167,6 +168,16 @@ existing realtime `Prompt` input unchanged. Sending client content with `turn_complete: true` interrupts active generation; sending it without that flag appends content and waits for further input. +Callers pick between these text input paths by intent: + +- Realtime user input (the user speaks or types now): `Prompt`. +- An immediate, ordered response with interruption of active generation: + `ClientContent` with `user` role and `turn_complete: true`. +- Silent context the model uses only when generation is next triggered: + `ClientContent` with `user` role and `turn_complete: false`. +- Restoring or fabricating an earlier assistant turn: `ClientContent` with + `model` role. + The exact public API is: ```rust @@ -297,6 +308,50 @@ pub use types::{ }; ``` +## 3.8 Live feature overview + +Beyond the wiring documented above, the two 3.8 Live models provide these +capabilities (verified against Google's model and Live API documentation on +2026-09-18 and 2026-09-21, with the model-specific pages treated as +authoritative): + +- **Async function calling.** Non-blocking tool declarations (`behavior: + NON_BLOCKING`, the 3.8 default) let longer-running functions execute in the + background while the conversation continues. The agent can provide updates + and returns results when ready, controlled by the `FunctionResponseScheduling` + values `SILENT`, `WHEN_IDLE`, and `INTERRUPT` that `FunctionCallResult` + forwards. Extended Thinking runs async-only and rejects scheduling. Google's + capabilities table spells the last value `INTERRUPTED`; the tool-use guide + and this ADR use `INTERRUPT`. +- **Proactive audio.** The agent speaks only when relevant and can remain + quiet unless directly addressed, making conversations less interruptive. This + is enabled permanently on both 3.8 models (see the fixed-choice entry in the + next section). +- **Client content for context injection.** The `clientContent` channel lets + callers add context without forcing a turn (`turn_complete: false`), + backchanneling information silently so the model stays informed without a + spoken response. With `turn_complete: true`, the same channel instead asks + the model to respond immediately, interrupting any active generation. +- **High/background reasoning.** Extended Thinking performs frontier-level + reasoning in the background and supports more complex tasks while remaining + responsive during the ongoing conversation. Standard 3.8 keeps interleaved + reasoning with fixed, non-configurable depth. +- **Native audio experience.** The background reasoning runs inside the native + audio pipeline, so deep processing and real-time voice interaction coexist + instead of trading one for the other. +- **Responsive multitasking.** The model can work on slower complex outputs + while still chatting, balancing deep reasoning with fast conversational + response. +- **Model capability comparison.** Standard `gemini-3.8-live` already carries + strong reasoning; `gemini-3.8-live-extended-thinking` favors deeper + performance on complex or creative tasks over minimal latency. +- **Google Search grounding.** The only supported grounding tool on both + models; exposed through `Tool::GoogleSearch` (its metadata is not forwarded; + see the exclusion list below). +- **Bidirectional audio transcription.** Input and output transcripts in + `VERBATIM` or `SMART` mode are configured through the transcription `Params` + fields documented above; native audio language selection stays automatic. + ## Features not exposed by `google-dialog` The public service API is intentionally narrower than the direct Gemini Live @@ -460,6 +515,8 @@ any GDPR deployment conclusion part of the software contract. - - - -- +- - -- \ No newline at end of file +- — 3.8 Live GA announcement, + 2026-09-15 +- From 5b7ae88d76bd74c6861e97a300ad65be655aa115 Mon Sep 17 00:00:00 2001 From: Armin Sander Date: Mon, 21 Sep 2026 18:12:54 +0200 Subject: [PATCH 13/16] dialog-example: Extend prompting possibilities to cover the features of google-dialog gemini v3.8 --- examples/dialog.rs | 123 +++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 114 insertions(+), 9 deletions(-) diff --git a/examples/dialog.rs b/examples/dialog.rs index bbf9f39c..19598822 100644 --- a/examples/dialog.rs +++ b/examples/dialog.rs @@ -71,6 +71,26 @@ impl Provider { fn api(self) -> &'static dyn dialog_providers::ProviderApi { dialog_providers::provider_api(self) } + + /// Text input commands supported by the provider (see `send_input_line`). + fn capabilities(self) -> ProviderCapabilities { + let mut capabilities = ProviderCapabilities::default(); + + match self { + Provider::OpenAI | Provider::AzureOpenAI => {} + Provider::Google | Provider::GoogleAgentPlatform => { + capabilities.client_content = true; + } + } + + capabilities + } +} + +#[derive(Debug, Clone, Copy, Default)] +struct ProviderCapabilities { + /// The provider service accepts `clientContent` service input events. + client_content: bool, } #[tokio::main] @@ -99,6 +119,7 @@ async fn main() -> Result<()> { .expect("Failed to get default input config"); println!("Audio device input config: {input_config:?}"); + print_usage_hint(); let channels = input_config.channels(); let sample_rate = input_config.sample_rate(); @@ -208,7 +229,7 @@ async fn main() -> Result<()> { line = stdin_lines.next_line(), if !stdin_closed => { match line? { Some(line) => { - send_prompt_line(&input_sender, &line).await?; + send_input_line(cli.provider, &input_sender, &line).await?; } None => { stdin_closed = true; @@ -269,21 +290,105 @@ fn setup_audio_input_adapter( sender } -async fn send_prompt_line(input: &Sender, line: &str) -> Result<()> { - let prompt = line.trim(); - if prompt.is_empty() { +async fn send_input_line(provider: Provider, input: &Sender, line: &str) -> Result<()> { + let line = line.trim(); + if line.is_empty() { + return Ok(()); + } + + let event = parse_input_line(line); + let Some(event) = event else { + print_usage_hint(); + return Ok(()); + }; + + if matches!(event, ServiceEvent::ClientContent { .. }) + && !provider.capabilities().client_content + { + println!( + "Provider '{}' does not support clientContent commands", + provider + .to_possible_value() + .expect("Provider has a possible value") + .get_name() + ); return Ok(()); } - input - .send(Input::ServiceEvent { - value: json!({ "type": "prompt", "text": prompt }), - }) - .await?; + let value = match event { + ServiceEvent::Prompt { text } => json!({ "type": "prompt", "text": text }), + ServiceEvent::ClientContent { + role, + text, + turn_complete, + } => json!({ + "type": "clientContent", + "role": role, + "text": text, + "turnComplete": turn_complete, + }), + }; + + input.send(Input::ServiceEvent { value }).await?; Ok(()) } +enum ServiceEvent { + Prompt { + text: String, + }, + ClientContent { + role: &'static str, + text: String, + turn_complete: bool, + }, +} + +/// Parses one stdin line into a service input event. +/// +/// Grammar (first word decides): +/// - `prompt ` — realtime prompt. +/// - `user ` / `agent ` — clientContent for the user/model role. +/// A single trailing `!` on the text is stripped and sets +/// `turnComplete: true` ("respond now"); `user !` sends empty text with +/// `turnComplete: true`. Without it, content is added silently. A bare +/// `user`/`agent` sends empty text, silent. +/// +/// Returns `None` for a line without text after the command word; the caller +/// prints the usage hint. +fn parse_input_line(line: &str) -> Option { + let (command, rest) = match line.split_once(' ') { + Some((command, rest)) => (command, rest.trim()), + None => (line, ""), + }; + + match command { + "prompt" if !rest.is_empty() => Some(ServiceEvent::Prompt { text: rest.into() }), + "user" | "agent" => { + let role = if command == "user" { "user" } else { "model" }; + let turn_complete = rest.ends_with('!'); + Some(ServiceEvent::ClientContent { + role, + text: rest.strip_suffix('!').unwrap_or(rest).into(), + turn_complete, + }) + } + _ => None, + } +} + +fn print_usage_hint() { + println!( + "Commands, one per line:\n\ + prompt \n\ + user [!]\n\ + agent [!]\n\ + a single trailing ! completes the turn now, otherwise content is added silently;\n\ + lines without a prompt/user/agent first word print this hint" + ); +} + fn list_available_voices(provider: Provider) -> Result<()> { println!("Available voices for {:?}:", provider); for voice in provider.api().voices() { From 2d9ce0241ce9f742175cd40d6e2069ddb966e0e6 Mon Sep 17 00:00:00 2001 From: Armin Sander Date: Mon, 21 Sep 2026 18:22:53 +0200 Subject: [PATCH 14/16] dialog: rename stdin input enum ServiceEvent to InputEvent --- examples/dialog.rs | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/examples/dialog.rs b/examples/dialog.rs index 19598822..99370479 100644 --- a/examples/dialog.rs +++ b/examples/dialog.rs @@ -302,8 +302,7 @@ async fn send_input_line(provider: Provider, input: &Sender, line: &str) return Ok(()); }; - if matches!(event, ServiceEvent::ClientContent { .. }) - && !provider.capabilities().client_content + if matches!(event, InputEvent::ClientContent { .. }) && !provider.capabilities().client_content { println!( "Provider '{}' does not support clientContent commands", @@ -316,8 +315,8 @@ async fn send_input_line(provider: Provider, input: &Sender, line: &str) } let value = match event { - ServiceEvent::Prompt { text } => json!({ "type": "prompt", "text": text }), - ServiceEvent::ClientContent { + InputEvent::Prompt { text } => json!({ "type": "prompt", "text": text }), + InputEvent::ClientContent { role, text, turn_complete, @@ -334,7 +333,7 @@ async fn send_input_line(provider: Provider, input: &Sender, line: &str) Ok(()) } -enum ServiceEvent { +enum InputEvent { Prompt { text: String, }, @@ -357,18 +356,18 @@ enum ServiceEvent { /// /// Returns `None` for a line without text after the command word; the caller /// prints the usage hint. -fn parse_input_line(line: &str) -> Option { +fn parse_input_line(line: &str) -> Option { let (command, rest) = match line.split_once(' ') { Some((command, rest)) => (command, rest.trim()), None => (line, ""), }; match command { - "prompt" if !rest.is_empty() => Some(ServiceEvent::Prompt { text: rest.into() }), + "prompt" if !rest.is_empty() => Some(InputEvent::Prompt { text: rest.into() }), "user" | "agent" => { let role = if command == "user" { "user" } else { "model" }; let turn_complete = rest.ends_with('!'); - Some(ServiceEvent::ClientContent { + Some(InputEvent::ClientContent { role, text: rest.strip_suffix('!').unwrap_or(rest).into(), turn_complete, From b4903c89bb14fb53b826ad18a4b0d6f5ab0ace48 Mon Sep 17 00:00:00 2001 From: Armin Sander Date: Mon, 21 Sep 2026 19:04:05 +0200 Subject: [PATCH 15/16] adr: omitted function behavior defaults to blocking, not non-blocking --- docs/adr/0006-gemini-3-8-live-model-semantics.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/adr/0006-gemini-3-8-live-model-semantics.md b/docs/adr/0006-gemini-3-8-live-model-semantics.md index 9844e25e..666137e8 100644 --- a/docs/adr/0006-gemini-3-8-live-model-semantics.md +++ b/docs/adr/0006-gemini-3-8-live-model-semantics.md @@ -316,8 +316,10 @@ capabilities (verified against Google's model and Live API documentation on authoritative): - **Async function calling.** Non-blocking tool declarations (`behavior: - NON_BLOCKING`, the 3.8 default) let longer-running functions execute in the - background while the conversation continues. The agent can provide updates + NON_BLOCKING`) let longer-running functions execute in the + background while the conversation continues. Omitted `behavior` defaults to + blocking (Google's tool-use guide: a declaration without `behavior` "will + still pause all interactions with the model"); the agent can provide updates and returns results when ready, controlled by the `FunctionResponseScheduling` values `SILENT`, `WHEN_IDLE`, and `INTERRUPT` that `FunctionCallResult` forwards. Extended Thinking runs async-only and rejects scheduling. Google's From cb02b3f35703548c72151b73519e3951c7d4c00e Mon Sep 17 00:00:00 2001 From: Armin Sander Date: Mon, 21 Sep 2026 19:16:27 +0200 Subject: [PATCH 16/16] gemini-live: update submodule for optional tool call ids --- external/gemini-live-rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/external/gemini-live-rs b/external/gemini-live-rs index ebf7d8a0..05053252 160000 --- a/external/gemini-live-rs +++ b/external/gemini-live-rs @@ -1 +1 @@ -Subproject commit ebf7d8a03c57a25fa8da5991c933424cd28513ae +Subproject commit 05053252e5b1a7cae3773d819d8a670db5c208cc