feat(harness): run hosted LiveKit agents by agent name

[cdileep23]

Summary

Lets the harness run against a LiveKit agent that is already deployed (hosted, connect_only), not only one it starts from source. A hosted LiveKit job carries the agent's registered name and a pasted system prompt; the call runner dispatches that name into a room on the customer's LiveKit project, and authoring builds the agent profile from the prompt.

Pairs with the platform PR: future-agi/future-agi#3402

Why the changes are done (what is the problem)

A hosted LiveKit job could never complete:

• _dispatch_agent_name read the agent name only from runtime.metadata["livekit_agent_name"], which is set only when ALK starts the agent from source and reads its LIVEKIT_AGENT_NAME. A hosted agent has no such process, so every call aborted with voice_dispatch_identity_unavailable.
• Bundle preflight let connect-only jobs claim customer target_provider secrets only for Vapi and Retell, so the LiveKit key and secret failed with secret_unclaimed.
• Authoring had no profile for LiveKit: there is no LiveKit API that returns an agent's prompt or tools.
• AgentConnection accepted livekit + connect_only with no required fields, so a bad job only failed at dispatch.

What changed

src/fi/alk/harness/job.py

A livekit + connect_only AgentConnection requires config.agent_name and a target_system_prompt (max 65,536 chars), mirroring the phone branch. Fails with livekit_connect_only_requires_agent_name / livekit_connect_only_requires_target_system_prompt.

src/fi/alk/harness/authoring_entrypoint.py

_load_provider_import_profile returns {"provider": "livekit", "modality": "voice", "agent_name", "system_prompt", "tools": []} for hosted LiveKit, and deletes the one-shot target-secrets file if one was passed (it exists only for read-only provider inspection).

src/fi/alk/harness/call_runner.py

_dispatch_agent_name(runtime, job) reads job.agent.config["agent_name"] for connect_only jobs and keeps reading runtime.metadata["livekit_agent_name"] otherwise. The call site passes the job.

src/fi/alk/harness/process_preflight.py

livekit joins vapi, retell, retell_chat in the connect-only set that may claim customer target-provider secrets. Phone stays out: it dials with platform SIP credentials.

src/fi/alk/harness/sandbox_server.py

The local sandbox server's provider_only requests accept livekit.

src/fi/alk/harness/sources.py

The provider briefing tells the authoring model that a phone or LiveKit connection may contain only the user-supplied prompt.

Known limitations (follow-ups)

Tools are empty for hosted LiveKit. Vapi and Retell tools are fetched from their APIs (inspect_provider_target: GET /assistant/{id} + /tool/{id}, get-agent + get-retell-llm / get-conversation-flow). A LiveKit agent's tools are functions in the customer's code and LiveKit exposes no API for them, so the profile has "tools": [], the same as phone targets. Scenarios are generated from the prompt alone. Follow-up options: read tools from the agent repository (the hosted form already has an optional GitHub repo field) or accept pasted tool schemas.

Agent check happens in platform preflight. The platform PR adds a preflight probe that dispatches the agent into a throwaway room and waits up to 8s for it to join, because LiveKit accepts a dispatch for any name. It starts a short real session of the customer's agent (billed to their LiveKit and AI provider accounts) and a wrong name costs the full 8s. Planned follow-up: pass once the dispatch has an accepted job instead of waiting for the join, after testing against a real project.

Tests included (what each scenario covers)

#	Test	Setup	Action	What it pins
1	test_livekit_connect_only_requires_agent_name_and_prompt (3 cases)	AgentConnection livekit + connect_only	missing name / missing prompt / prompt over 65,536 chars	each raises its specific error
2	test_livekit_connect_only_authoring_uses_the_pasted_prompt_and_drops_secrets	hosted LiveKit job + a target-secrets file	_load_provider_import_profile	profile is the pasted prompt with no tools; secrets file deleted
3	test_livekit_connect_only_dispatches_the_configured_agent_name	runtime metadata names agent-from-source	run one call	spec dispatches returns-agent from config, on the customer's URL
4	test_connect_only_provider_claims_the_customer_target_secret (vapi, retell, retell_chat, livekit)	bundle with no process claiming secrets	preflight_bundle	passes
5	test_connect_only_phone_does_not_claim_a_customer_target_secret	same, connector phone	preflight_bundle	secret_unclaimed
6	test_provider_only_accepts_a_connect_only_livekit_agent	LocalSandboxRequest	build request	accepted

Each test was checked to fail with its source change reverted.

Commands to run tests

# from the repo root
uv sync --frozen --all-extras
.venv/bin/python -m pytest -q tests/harness/test_call_runner.py tests/harness/test_process_preflight.py tests/test_harness_sandbox_server.py

Local verification results

tests/harness/test_call_runner.py + tests/harness/test_process_preflight.py: 145 passed
tests/test_harness_sandbox_server.py: 27 passed, 2 failed

The 2 failures (test_public_github_branch_url_populates_job_ref, test_private_github_submission_becomes_hosted_job) fail the same way on untouched origin/dev.

End to end on a local stack with an E2B template built from this branch (alk-hosted-local:f8aeae48-d44b-4283-8276-645e9c3a0b2b): a hosted LiveKit environment built (contract, world, 10 scenarios) and a run completed all 10 calls, dispatching the deployed agent by name.

Local end-to-end run

Distributed dev stack, runner wheel and E2B template built from agent-learning-kit#146 (alk-hosted-local:f8aeae48-d44b-4283-8276-645e9c3a0b2b), hosted LiveKit agent on a LiveKit Cloud project.

Environment build	41f39d82-f7d4-412a-8050-b36d6dbf9dac: completed on attempt 3 (contract, world, 10 scenarios). Attempts 1 and 2 failed on local setup (AI gateway proxy route, missing storage buckets), not on the LiveKit code
Run	run test e95a2d0f-0d1b-410a-ab77-b82be697aafd, execution 02f694fd-aece-42fb-ad85-eb9278d204d7
Calls	10 / 10 completed, all dispatched to the deployed agent by name
Durations	66 to 147 s per call (about 111 s average)
How calls ended	6 simulator hung up (simulator_end_call), 4 loop detector ended a repetitive closing (closing_loop)
Recordings	10 / 10 stored (mono and stereo WAV)

Video

None.

Screenshot of test suit run

None.

Backwards compatibility

No schema change. Behaviour changes only for connector: livekit with mode: connect_only, which previously could not complete.

Caller	Impact
LiveKit source builds (mode unset)	None: dispatch name still comes from runtime metadata
Vapi / Retell / phone connect-only	None
Hosted LiveKit jobs	Now validated, authored from the prompt, and dispatched by config.agent_name

Manual verification (UI)

• Build the runner wheel and an E2B template from this branch; point the platform at them
• Create a hosted LiveKit environment with a registered agent name and a system prompt
• Confirm the build completes and scenarios are generated from the prompt
• Run a simulation and confirm calls connect to the deployed agent and produce transcripts and recordings

Type of change

• 🐛 Bug fix
• ✨ New feature
• 💥 Breaking change
• 📖 Docs
• 🧹 Chore / refactor
• 🚀 Performance
• 🧪 Test-only

Checklist

• Follows the coding standards
• Tests added for every changed behaviour, including failure paths
• No secrets, URLs or PII committed

Backout / rollback

Revert this PR. Hosted LiveKit jobs go back to failing at dispatch; nothing else is affected. Release order: ship this ALK change (new version, runner wheel and E2B template) before the platform PR enables the LiveKit tile.

Linear

Closes TH-8481

🤖 Generated with Claude Code