gen_agent_server

A runnable OTP application for using Claude and Codex through one GenAgent entry point. It uses the published GenAgent, Ensemble, and CLI backend packages. The default Echo backend needs no login and makes the application easy to smoke-test. This is an early dogfooding host, not a durable job service.

The base server drives agents: instances, invocations, results, cancellation, the GenAgentServer.Ops catalogue, the CLI, and release rpc. MCP, scheduled jobs, JSON pattern runs, public source reads, peers, and the control and selection libraries are optional extensions.

Run locally

mix deps.get
mix test
mix gen_agent_server agents
mix gen_agent_server ask echo "hello"

To use both locally authenticated CLI backends against a project:

GEN_AGENT_SERVER_PROVIDERS=claude,codex \
GEN_AGENT_SERVER_CWD=/path/to/project \
iex -S mix

iex> GenAgentServer.agents()
{:ok, ["claude", "codex"]}
iex> GenAgentServer.ask("claude", "Summarize this project")
iex> GenAgentServer.ask("codex", "Find the test entry points")
iex> {:ok, id} = GenAgentServer.invoke("codex", "Find the test entry points")
iex> GenAgentServer.result(id)

Access defaults to inherit for profiles, startup agents, dynamic routes, and pattern agents. Inheritance omits server permission-mode, tools, sandbox, and ignore-user-config overrides and uses the server process user's CLI settings. It does not transfer transient client permissions automatically. Headless sessions cannot answer prompts; Codex retains approvals disabled. Effective access depends on CLI configuration and host permissions.

Claude accepts inherit, read_only, plan, accept_edits, auto, bypass_permissions, and dont_ask. Explicit read_only uses dontAsk with only Read, Grep, and Glob; other explicit modes do not restrict tools. Codex sandbox accepts inherit, read_only, and workspace_write. Codex user config accepts inherit (default) or ignore; reproducing the previous complete Codex configuration requires both read_only and ignore.

Both keep their native project instructions and separate provider sessions. mix gen_agent_server ask PROVIDER PROMPT starts a fresh application for a single request; use IEx for a persistent multi-turn session.

Retained invocation summaries (issue #75, first increment)

{:ok, summaries} = GenAgentServer.invocations("review", limit: 50)

The same read is available as Ops invocations (and as an MCP tool with the MCP extension):

MIX_QUIET=1 mix gen_agent_server.ops invocations --instance review --limit 50 --remote

Instance is required; limit defaults to 50 and accepts integers from 1 to 200. Entries are ordered by newest admission, using a stable sequence even when wallclock timestamps tie. Fields are id, route, source, status, admitted_at_unix_ms, completed_at_unix_ms, duration_ms, and error_category. Statuses are pending, completed, failed, or cancelled. Pending completion time and duration are nil; terminal duration uses monotonic elapsed time and timestamps use Unix milliseconds. Error categories are limited to timeout, task_crashed, backend_error, and cancelled (nil for pending/success). No prompt, response, raw error reason, recipient, or backend configuration is included. Terminal timestamps and durations reflect the owner's first observation of completion, rather than the provider's finish time.

Reads collect current completions from the invocation owner and are repeatable while entries remain retained. Limit bounds the returned list; existing max_in_flight and max_results bound storage. Eviction, instance shutdown, and restart discard corresponding summaries. Unknown/stopped instances return instance_not_found; a recreated instance begins empty. The Elixir API returns invalid_limit for invalid limits; Ops also validates argument types. An opt-in, read-only web observer for one running release, its allowlisted instances and these summaries is in examples/operator_view. It is a separate Mix project: the server application and release gain no Phoenix dependency and open no listener for it. Broader #75 work remains open.

Multiple projects in one server

Set GEN_AGENT_SERVER_CONFIG to a JSON file with named project profiles. Each profile has its own working directory, backend sessions, and invocation result store. Paths in cwd are relative to the JSON file; absolute paths also work.

{
  "profiles": [
    {"name": "home", "cwd": "/path/to/home/project", "providers": ["claude", "codex"]},
    {"name": "work", "cwd": "/path/to/work/project", "providers": ["codex"],
     "codex_user_config": "inherit"}
  ]
}

MIX_ENV=prod mix release
GEN_AGENT_SERVER_CONFIG=/path/to/projects.json \
RELEASE_NODE=gen_agent_server_dogfood \
_build/prod/rel/gen_agent_server/bin/gen_agent_server start

Provider child environment

A release leaks its boot environment (RELEASE_*, BINDIR, ROOTDIR, and release bin/erts-*/bin PATH entries) into every child process. Project tools spawned by Claude or Codex, such as a project's own mix, then fail looking for the release's start.boot. The server never changes its own environment; instead every Claude and Codex agent gets a child :env from GenAgentServer.ChildEnv that unsets those variables and removes only the exact release runtime directories from PATH. Shared MCP bearer configuration (GEN_AGENT_SERVER_SHARED_MCP_*) is also forced unset in provider children. Provider credentials and other inherited variables pass through untouched; their inherited values are not copied into options or logged. This applies to startup agents, profiles, dynamic routes, patterns, and direct start_instance/3 tuples.

Operators can extend the child environment with the trusted :provider_overrides application setting, for example config :gen_agent_server, provider_overrides: %{"codex" => [env: %{"PATH" => "/opt/tools/bin:/usr/bin", "NO_COLOR" => "1", "TMPDIR" => false}]}. A string sets a variable, false unsets it, and a PATH override replaces the inherited one. Cleanup runs after overrides, so release boot names stay unset and release directories are stripped even from an override PATH. No client operation or MCP schema accepts env.

The backend streaming launch paths use false for Port unsets. This does not change standalone wrapper one-shot commands: ClaudeWrapper's System.cmd runner requires nil for unsets. Optional Forcola runners are not enabled or validated by this server. Non-CLI backends and custom raw strategy_opts are passed through unchanged; typed patterns use the provider configuration above.

In another terminal, set the same node name before issuing remote commands:

export RELEASE_NODE=gen_agent_server_dogfood
mix gen_agent_server.remote instances
mix gen_agent_server.remote --instance home agents
mix gen_agent_server.remote --instance home invoke codex "Find a small issue worth fixing"
mix gen_agent_server.remote --instance home result INVOCATION_ID

Remote control and read commands have a 30-second RPC wait limit; synchronous ask has a one-hour limit. Set GEN_AGENT_SERVER_RPC_TIMEOUT_MS and GEN_AGENT_SERVER_ASK_RPC_TIMEOUT_MS to positive millisecond values to change them. A timed-out RPC process is terminated. For work that may outlast a synchronous ask, use invoke and check result later; the invocation keeps running in the server after the short invoke RPC returns its ID.

The default server/default instance remains available and uses GEN_AGENT_SERVER_PROVIDERS and GEN_AGENT_SERVER_CWD. Profile names must be unique and cannot contain / or control characters. A missing directory, unknown provider, or duplicate name fails startup. Profiles use the same inherited provider access defaults as startup agents and support the same explicit access settings. These are provider controls, not a server-enforced file allowlist. Use a separate worktree for coding tasks and review changes before merging.

"codex_user_config": "ignore" passes the Codex CLI's --ignore-user-config option on fresh and resumed turns, avoiding host-specific MCP servers, model defaults, and other settings in the operator's user configuration. The default, "inherit", leaves those settings available to the CLI. The installed Codex CLI still handles its own authentication; this setting does not provide credentials. Pattern specs accept the same choices.

"codex_response_text" accepts "all_messages" (the default) or "final_message". The default joins every completed Codex agent message with a blank line, preserving existing behavior. "final_message" makes the completed result text only the last agent message, which is useful when Codex emits commentary before a final structured answer. Text events are still preserved. Static profiles and pattern specs accept the same choices.

The server uses published Hex packages: core ~> 0.8.0, Ensemble ~> 0.7.0, Claude adapter ~> 0.3.0, and Codex adapter ~> 0.6.0. The Codex adapter includes response_text; no Git dependency pins or core override are needed. Each owned Ensemble retains at least max_in_flight unread completions until the invocation owner collects them. The server's max_results separately bounds retained results after collection.

Invocations, results, and cancellation

Wait on an existing invocation with GenAgentServer.result(instance, id, wait_ms: 60_000). wait_ms defaults to 0 and accepts integers from 0 to 600_000; invalid values return {:error, :invalid_wait}. Completion, failure, or cancellation wakes all waiters. Expiry returns {:ok, :pending} and leaves the invocation running. Other requests on the instance continue while callers wait.

mix gen_agent_server.ops result --id INVOCATION_ID --wait_ms 60000 --remote
mix gen_agent_server.remote result INVOCATION_ID --wait 60000

Both remote commands allow the requested wait plus a 30 second RPC margin.

invoke/2 returns an instance-scoped ID after Ensemble admits the turn. result/1 returns {:ok, :pending}, {:ok, :completed, response}, or {:ok, :failed, reason}. Completed results can be read repeatedly, including after the caller exits. The default instance admits at most 16 in-flight turns and retains up to 100 completions; additional admissions return {:error, :busy} while all in-flight slots remain occupied. A new admission checks for completed turns before applying this limit. An ask/3 timeout only ends the wait: the invocation keeps running and its result remains available through its ID if the caller used invoke/2.

The Elixir API also supports cancellation and optional terminal delivery:

{:ok, id} = GenAgentServer.invoke("review", "codex", "Review this module",
  recipient: self(), source: :api)
GenAgentServer.cancel("review", id)
# {:ok, :cancelled} or {:ok, :cancelled_unconfirmed} when cancellation wins
# receive {:gen_agent_server, :completion, metadata, terminal_result}

Cancellation collects existing completions first. A retained terminal ID returns {:error, :already_finished}; unknown, evicted, wrong-instance or previous-generation IDs return {:error, :not_found}, and a missing instance returns {:error, :instance_not_found}. Custom strategies without cancellation support return {:error, :unsupported} and leave work pending. Results retain the existing shape: cancellation is {:ok, :failed, :cancelled} and reads remain repeatable until eviction.

The canonical cancel operation requires both a nonempty exact instance and id; it never selects the default instance or retries. All adapters return instance, id, and status (cancelled or cancelled_unconfirmed). Errors retain not_found, already_finished, instance_not_found, and unsupported. Retained results still report failed with error code cancelled. A lost acknowledgement is uncertain: read result with the same pair instead of automatically retrying. Acknowledgement does not prove provider cleanup.

mix gen_agent_server.ops cancel --instance review --id "$ID" --remote
mix gen_agent_server.remote --instance review cancel "$ID"

With the MCP extension, stdio and shared MCP accept cancel with {"instance":"review","id":"..."}; shared MCP also requires its bearer token and an allowed instance.

Recipient must be a PID or nil; invalid values are rejected before admission. Recipient options are consumed by the server and never forwarded to Ensemble or backends. For every observed terminal success, error or cancellation, the owner attempts one send before result eviction. Metadata includes instance, agent, invocation_id, ensemble_token and source; cancellation includes cancellation_ack. Cancellation observed without an owner acknowledgement uses :cancelled_unconfirmed. The message carries the full retained result, even if that result is subsequently evicted in the same batch. There is no durable delivery, dead-recipient receipt or instance-shutdown notification guarantee.

Cancellation is synchronous for the invocation owner; a slow child acknowledgement can delay its invoke/result/cancel calls. Ensemble 0.7.4 keeps its control loop responsive during child admission and cancellation (dependency #221). The server requires Ensemble >=0.7.4 and its compatible core >=0.10.0 and adapters Claude >=0.3.2 / Codex >=0.6.2. Its acknowledgement covers request cancellation, not external provider process settlement. Finalization releases raw admission capacity while unrelated work continues. The owner remains the sole Ensemble inbox consumer; do not drain its managed session directly.

An embedding application can run independent instances with distinct agent specs and result stores:

{:ok, _pid} = GenAgentServer.start_instance("review", agents,
  max_in_flight: 4, max_results: 50)
{:ok, id} = GenAgentServer.invoke("review", "codex", "Review this module")
GenAgentServer.result("review", id)
GenAgentServer.ask_instance("review", "codex", "Summarize the finding")
GenAgentServer.stop_instance("review")

Each agents entry has the same {name, callback_module, backend_options} shape as config/runtime.exs. Result IDs are meaningful only within their instance. All results and provider sessions remain process-local and volatile.

Configured route descriptions also include review_read_only_explicit: true only when the Claude/Codex route config explicitly supplied its read_only mode, false for omitted modes and Echo. It describes configuration provenance for the optional controller's review policy; it is not a backend option. Existing provider defaults still apply to ordinary invocations.

Pattern instances use one external route while Ensemble owns their internal stages or workers. For example, a Pipeline can use the same invocation IDs and result store as a Switchboard instance:

{:ok, _pid} = GenAgentServer.start_pattern_instance(
  "review-pipeline", "review", GenAgentEnsemble.Strategies.Pipeline,
  stages: [
    {"draft", GenAgentEnsemble.Agents.Simple, backend: GenAgentEnsemble.Backends.Echo},
    {"revise", GenAgentEnsemble.Agents.Simple, backend: GenAgentEnsemble.Backends.Echo}
  ]
)
{:ok, id} = GenAgentServer.invoke("review-pipeline", "review", "a short note")
GenAgentServer.result("review-pipeline", id)

agents("review-pipeline") lists the external review route, while status("review-pipeline") shows both that route and Ensemble's internal agents. The instance is created at runtime and disappears on restart; an embedding application can start it again during its own startup. The JSON profile format currently creates Switchboard instances only.

To start an OTP release:

MIX_ENV=prod mix release
RELEASE_NODE=gen_agent_server_dogfood \
_build/prod/rel/gen_agent_server/bin/gen_agent_server start

From another terminal, use the same RELEASE_NODE with the local remote command. It passes the prompt as data, so quotes and newlines do not need to be embedded in an Elixir expression:

RELEASE_NODE=gen_agent_server_dogfood \
mix gen_agent_server.remote ask echo "hello"

Use mix gen_agent_server.remote invoke echo "hello" and then mix gen_agent_server.remote result ID to submit and inspect a turn across separate shell commands. The local mix gen_agent_server task starts a fresh VM, so its invoke and result commands are intentionally unavailable. Both CLI tasks preserve UTF-8 output when piped. Expected request errors exit non-zero with a single-line message that names the searched instance. For a turn with multiple completed assistant messages, the CLI displays the final message. GenAgent.Response.text still contains the assembled text of the full turn, including earlier progress messages. Elixir API callers can read the original response and its retained events with GenAgentServer.ask/3 or GenAgentServer.result/2; check Response.event_coverage before treating the retained event list as complete.

mix gen_agent_server.ops lists the typed operations available to scripts and to adapters such as the MCP extension, including operations added by configured extensions. --remote runs an operation in the release above, where invocation results persist across commands. Set MIX_QUIET=1 when parsing stdout so a first-time Mix compile does not prefix the JSON with build notices:

MIX_QUIET=1 mix gen_agent_server.ops instances
MIX_QUIET=1 mix gen_agent_server.ops invoke --agent echo --prompt "hello" --remote
MIX_QUIET=1 mix gen_agent_server.ops result --id INVOCATION_ID --remote

Set GEN_AGENT_SERVER_RELEASE_BIN if the release binary lives elsewhere. The release's rpc command remains available for direct Elixir calls.

Extensions

Everything above driving agents is an optional extension under extensions/. An extension is a separate Mix project; a host lists the ones it wants:

config :gen_agent_server, extensions: [GenAgentServer.MCP]

With none listed, the server starts only its instances and the base Ops catalogue. See GenAgentServer.Extension for the callbacks.

Each extension directory is also a runnable project that lists itself, so cd extensions/mcp && mix gen_agent_server.mcp serves MCP from a checkout. Hosts call GenAgentServer.RuntimeConfig.from_env!/1 from their own config/runtime.exs to read the same GEN_AGENT_SERVER_* variables as this repository.

Extension	What it adds
mcp	Stdio MCP server and opt-in shared HTTP listener over Ops
public_source	public_* operations: anonymous reads of public GitHub source and issues
selection	Pure route suggestions (library)
control	Caller-started controller and evidence ledger (library)
patterns	run_pattern and patterns operations, mix gen_agent_server.run
schedule	Quantum calendar jobs from the configuration file
peers	Experimental messaging to existing Claude Desktop Code sessions

Current contract

The default instance and any additional instances each own a named Ensemble Switchboard and bounded result store. If one instance stops, its agents, provider sessions, IDs, and results disappear. The default instance stops the application if it fails rather than silently restarting with a fresh session. An ask timeout or caller exit does not establish that an underlying provider process stopped; GenAgent's turn watchdog bounds active work. There is no durable admission, automatic retry, or cross-node result store yet.

The app has repeatable result reads for multiple clients. Ops, the CLI, and the MCP extension expose explicit invocation cancellation with the exact instance and ID. Cancellation acknowledgement does not establish provider process cleanup.

Telemetry

The server emits [:gen_agent_server, :invocation, :start | :stop | :error] for admitted turns, [:gen_agent_server, :admission, :rejected] for requests that never received an ID, and [:gen_agent_server, :wait, :timeout] when a synchronous caller stops waiting for a turn that may still be running. Terminal invocation events have duration_ms; start and rejection events have system_time in the VM native unit. The module GenAgentServer.Telemetry documents the metadata contract.

Invocation events carry instance, agent, invocation_id, ensemble_token, and source (:api, :local_cli, :remote_cli, or an extension's source such as :scheduler or :mcp). They never contain prompts, responses, provider state, or raw errors. Use IDs and names to correlate a trace, not as metric labels. Repeated result reads do not emit another terminal event.