aws: LLM opens a new bedrock-runtime client every turn, so each reply waits on a fresh TCP+TLS handshake

[kimnamu]

Bug Description

Thanks for the LLM.prewarm() hook from #6484. The anthropic, openai, google and mistralai plugins implement it; the AWS plugin can't yet, because it has no client that lives past one turn.

LLMStream._run enters self._session.create_client("bedrock-runtime", ...) on every chat() call (llm.py L324). So each turn builds a new client and a new TCP + TLS connection, then closes both. Timing OpenAI GPT-6 on Bedrock:

"Say pong.", us-east-1	new client per turn (today)	one reused client
aws.LLM time to first token, us.openai.gpt-6-sol, n=10 median	1.10 s	0.69 s
aws.LLM time to first token, us.openai.gpt-6-luna, n=10 median	1.13 s	0.67 s
aiobotocore converse_stream end to end, Sol, 3 runs (n=5, 5, 10)	1.10 / 0.95 / 1.02 s	0.67 / 0.70 / 0.71 s

The "reused" aws.LLM rows monkeypatch create_client to hand back one already-open client. Split (n=10 medians): TCP connect 194 ms, TLS 1.3 handshake 199 ms, building the client 6 ms. So every turn pays two extra round trips: ~0.4 s from my machine (~190 ms from us-east-1), far less for an agent in the same region.

Expected Behavior

One bedrock-runtime client per aws.LLM instance, reused across turns and closed in aclose(), the way the openai and google plugins keep theirs. _prewarm_impl could then open the connection before the first reply.

Reproduction Steps

# aiobotocore directly, same Config the plugin uses; AWS credentials in the environment
uv run python - <<'PY'
import asyncio, statistics, time
from aiobotocore.session import get_session
from botocore.config import Config

REQ = dict(modelId="us.openai.gpt-6-sol",
           messages=[{"role": "user", "content": [{"text": "Say pong."}]}],
           inferenceConfig={"maxTokens": 64})
CFG = Config(user_agent_extra="x-client-framework:livekit-plugins-aws")

async def one(client):
    async for _ in (await client.converse_stream(**REQ))["stream"]:
        pass

async def main():
    s = get_session(); s.set_config_variable("region", "us-east-1")
    new, reuse = [], []
    async with s.create_client("bedrock-runtime", config=CFG) as shared:
        await one(shared)
        for _ in range(5):
            t0 = time.perf_counter()
            async with s.create_client("bedrock-runtime", config=CFG) as c:  # what _run does
                await one(c)
            new.append(time.perf_counter() - t0)
            t0 = time.perf_counter(); await one(shared); reuse.append(time.perf_counter() - t0)
    print(f"new client {statistics.median(new):.2f}s vs reused {statistics.median(reuse):.2f}s")

asyncio.run(main())
PY

new client 1.15s vs reused 0.72s

Operating System

macOS 26 (network-bound, not OS-specific)

Models Used

AWS Bedrock us.openai.gpt-6-sol via aws.LLM (any Bedrock model)

Package Versions

livekit-agents==1.8.2
livekit-plugins-aws==1.8.2   # main at b4df92e
aiobotocore==3.8.0
botocore==1.43.46
python==3.13

Session/Room/Call IDs

N/A

Proposed Solution

Create the client lazily on the first chat() (an AsyncExitStack on the LLM), reuse it in LLMStream._run, close it in aclose(), add _prewarm_impl. Two choices I'd leave to you: which event loop owns the client, and what a cheap request for _prewarm_impl would be (bedrock-runtime has no list-models call). Happy to open a PR once that's settled.

Additional Context

Related

• Polly TTS has the same per-request pattern (tts.py L162). I did not measure it.
• 250-word answers, n=3 medians: aws.LLM TTFT was 1.38 s (Sol) and 1.80 s (Luna). A reused boto3 client got 0.76 s and 1.32 s.