From 7b285df253d40a424995ce2a75e8b888c31fb2a7 Mon Sep 17 00:00:00 2001 From: Josh Rotenberg Date: Sun, 1 Nov 2026 05:37:17 -0800 Subject: [PATCH 1/2] chore: start work on #150 From cd602af95f5096a19504ec8d1d60227a4f7eaf73 Mon Sep 17 00:00:00 2001 From: Josh Rotenberg Date: Sun, 1 Nov 2026 05:47:12 -0800 Subject: [PATCH 2/2] feat: add typed turn results and session usage totals --- README.md | 52 +++++ lib/codex_wrapper.ex | 12 +- lib/codex_wrapper/evidenced_execution.ex | 39 ++-- lib/codex_wrapper/execution_observation.ex | 6 +- lib/codex_wrapper/session.ex | 78 +++++-- lib/codex_wrapper/session_server.ex | 31 ++- lib/codex_wrapper/structured.ex | 75 ++----- lib/codex_wrapper/turn.ex | 151 ++++++++++++++ mix.exs | 1 + .../evidenced_execution_test.exs | 115 +++++++++- test/codex_wrapper/session_server_test.exs | 2 + test/codex_wrapper/session_test.exs | 55 ++++- test/codex_wrapper/session_turn_test.exs | 82 +++++++- test/codex_wrapper/structured_test.exs | 56 ++++- test/codex_wrapper/turn_test.exs | 196 ++++++++++++++++++ test/fixtures/turns/0.165.0-completed.ndjson | 4 + .../turns/0.165.0-invalid-usage.ndjson | 4 + test/fixtures/turns/README.md | 13 ++ 18 files changed, 874 insertions(+), 98 deletions(-) create mode 100644 lib/codex_wrapper/turn.ex create mode 100644 test/codex_wrapper/turn_test.exs create mode 100644 test/fixtures/turns/0.165.0-completed.ndjson create mode 100644 test/fixtures/turns/0.165.0-invalid-usage.ndjson create mode 100644 test/fixtures/turns/README.md diff --git a/README.md b/README.md index 7ac91ea..c133753 100644 --- a/README.md +++ b/README.md @@ -142,8 +142,28 @@ session = CodexWrapper.Session.new(config, model: "o3") CodexWrapper.Session.turn_count(session) #=> 2 + +turn = CodexWrapper.Session.last_turn(session) +# Also available as session.last_turn +turn.status +#=> :completed +turn.final_message +#=> "Added tests for lib/foo.ex" + +CodexWrapper.Session.total_usage(session) +#=> %{input_tokens: 120, cached_input_tokens: 100, cache_write_input_tokens: nil, +# output_tokens: 7, reasoning_output_tokens: nil} ``` +`send/3` keeps its raw `%Result{}` return value and result history. The updated +session exposes the typed JSON turn as `session.last_turn` or +`Session.last_turn/1`. `SessionServer.last_turn/1` and +`SessionServer.total_usage/1` provide the same access through a session server. +Usage totals sum reported nonnegative integers from recorded turns. A field +is `nil` if it was never reported; the whole total is `nil` if no recorded +turn included usage. Nonzero exits leave the session unchanged and do not +contribute to usage totals. + Session options are the exec options in the table under [Exec options](#exec-options) and apply to every turn. Resumed turns run `codex exec resume`, which rejects `--profile`, `--cd`, `--add-dir`, @@ -153,6 +173,38 @@ Session options are the exec options in the table under are emitted on resumed turns as `-c sandbox_mode="..."` and `-c approval_policy="..."`. +## Typed turn results + +`CodexWrapper.Turn.from_result/1` projects an existing `--json` result without +starting a process. `%Result{}` continues to hold the raw stdout, stderr and +exit code. + +```elixir +config = CodexWrapper.Config.new() +exec = CodexWrapper.Exec.new("Summarize the changes") |> CodexWrapper.Exec.json() +{:ok, result} = CodexWrapper.Exec.execute(exec, config) +turn = CodexWrapper.Turn.from_result(result) + +turn.status # :completed, :failed or :no_terminal +turn.final_message # last completed agent message from the last turn, or nil +turn.thread_id # thread.started identity, or nil +turn.usage # five token counters, or nil +turn.error # native error data, or nil +turn.events # parsed events, including unrecognized fields +``` + +The last `turn.completed` or `turn.failed` sets the native turn outcome. A +top-level `error` fails the turn if no terminal event follows it. A nonzero +process exit makes the projection `:failed`, including after a completion +event. `CodexWrapper.exec/2` and `CodexWrapper.review/1` still return +`{:ok, %Result{success: false}}` for a nonzero exit; runner errors return +`{:error, reason}`. + +Usage has the keys `:input_tokens`, `:cached_input_tokens`, +`:cache_write_input_tokens`, `:output_tokens` and `:reasoning_output_tokens`. +Only nonnegative integers are retained; missing or invalid counters are +`nil`. Original values and any extra fields remain in `turn.events`. + ## IEx REPL Use Codex conversationally from IEx: diff --git a/lib/codex_wrapper.ex b/lib/codex_wrapper.ex index b078f70..fdde635 100644 --- a/lib/codex_wrapper.ex +++ b/lib/codex_wrapper.ex @@ -92,8 +92,10 @@ defmodule CodexWrapper do Execute a prompt non-interactively and return the result. Convenience wrapper that builds a `Config` and `Exec` from keyword options. - Returns `{:ok, %Result{}}` when codex ran, or `{:error, reason}` when it - could not be run or timed out. + Returns `{:ok, %Result{}}` when codex ran, including a nonzero exit: + `{:ok, %Result{success: false, exit_code: code}}`. Returns `{:error, reason}` + when it could not be run or timed out. For `--json` output, use + `CodexWrapper.Turn.from_result/1` to inspect the turn outcome. ## Options @@ -202,8 +204,10 @@ defmodule CodexWrapper do Run a code review via `codex exec review`. Convenience wrapper that builds a `Config` and `Review` from keyword options. - Returns `{:ok, %Result{}}` when codex ran, or `{:error, reason}` when it - could not be run or timed out. + Returns `{:ok, %Result{}}` when codex ran, including a nonzero exit: + `{:ok, %Result{success: false, exit_code: code}}`. Returns `{:error, reason}` + when it could not be run or timed out. For `--json` output, use + `CodexWrapper.Turn.from_result/1` to inspect the turn outcome. ## Options diff --git a/lib/codex_wrapper/evidenced_execution.ex b/lib/codex_wrapper/evidenced_execution.ex index 7a01a8c..f55e06f 100644 --- a/lib/codex_wrapper/evidenced_execution.ex +++ b/lib/codex_wrapper/evidenced_execution.ex @@ -1,6 +1,15 @@ defmodule CodexWrapper.EvidencedExecution do @moduledoc false - alias CodexWrapper.{Command, Config, ExecutionObservation, Result, Runner, SessionObservation} + alias CodexWrapper.{ + Command, + Config, + ExecutionObservation, + JsonLineEvent, + Result, + Runner, + SessionObservation, + Turn + } @default_output 1_048_576 @default_timeout 300_000 @@ -138,7 +147,6 @@ defmodule CodexWrapper.EvidencedExecution do session: session, buffer: "", dropping?: false, - parsed: nil, stdout: "", stderr: "", bytes: 0, @@ -298,13 +306,7 @@ defmodule CodexWrapper.EvidencedExecution do %{state | session: nil} end - parsed = - case Jason.decode(state.buffer) do - {:ok, %{"type" => type} = value} when type in ["turn.completed", "turn.failed"] -> value - _ -> state.parsed - end - - %{state | buffer: "", parsed: parsed} + %{state | buffer: ""} end defp finish(state, {:returned, {outcome, evidence}}, execution) when is_map(evidence) do @@ -348,7 +350,7 @@ defmodule CodexWrapper.EvidencedExecution do outcome: outcome, stdout: streams.stdout, stderr: streams.stderr, - parsed_result: streams.parsed + parsed_result: parsed_result(streams.stdout) }) ) @@ -366,7 +368,7 @@ defmodule CodexWrapper.EvidencedExecution do outcome: {:error, {kind, reason}}, stdout: state.stdout, stderr: state.stderr, - parsed_result: state.parsed, + parsed_result: parsed_result(state.stdout), termination: {:runner_exception, kind} }) @@ -386,12 +388,19 @@ defmodule CodexWrapper.EvidencedExecution do defp parsed_result(stdout) do stdout |> :binary.split("\n", [:global]) - |> Enum.reduce(nil, fn line, result -> - case Jason.decode(line) do - {:ok, %{"type" => type} = value} when type in ["turn.completed", "turn.failed"] -> value - _ -> result + |> Enum.filter(&(byte_size(&1) <= SessionObservation.max_line_bytes())) + |> Enum.flat_map(fn line -> + case JsonLineEvent.parse(line) do + {:ok, event} -> [event] + {:error, _} -> [] end end) + |> Turn.from_events() + |> Map.fetch!(:terminal_event) + |> case do + %JsonLineEvent{data: data} -> data + nil -> nil + end end defp bounded_result({:ok, {_stdout, code, _stderr}}, streams), diff --git a/lib/codex_wrapper/execution_observation.ex b/lib/codex_wrapper/execution_observation.ex index b4f607f..fec4d01 100644 --- a/lib/codex_wrapper/execution_observation.ex +++ b/lib/codex_wrapper/execution_observation.ex @@ -4,8 +4,10 @@ defmodule CodexWrapper.ExecutionObservation do Sent as `{reference, observation}` by the execution caller before returning. A preceding `SessionObservation` establishes identity only. `:outcome` is the - legacy execution result before JSON/fork convenience projections; `:parsed_result` is the last bounded native - `turn.completed` or `turn.failed` JSON object, independent of process outcome. + legacy execution result before JSON/fork convenience projections. + `:parsed_result` is the last bounded native `turn.completed` or `turn.failed` + JSON object, selected through `CodexWrapper.Turn` independently of process + outcome. It retains the native field names and values, including usage. `:child_status` and `:child_signal` are raw native reports, never inferred from JSON, transport closure, or a pre-spawn failure. `:termination` records diff --git a/lib/codex_wrapper/session.ex b/lib/codex_wrapper/session.ex index 3313dcd..d2f9021 100644 --- a/lib/codex_wrapper/session.ex +++ b/lib/codex_wrapper/session.ex @@ -20,23 +20,30 @@ defmodule CodexWrapper.Session do CodexWrapper.Session.turns(session) #=> [%Result{...}, %Result{...}] + # Inspect the typed turn and cumulative token usage + CodexWrapper.Session.last_turn(session) + #=> %CodexWrapper.Turn{status: :completed, ...} + CodexWrapper.Session.total_usage(session) + # Resume a previous session session = CodexWrapper.Session.resume(config, "session-id-abc") """ alias CodexWrapper.Commands.Archive - alias CodexWrapper.{Config, Exec, ExecResume, JsonLineEvent, Options, Result, Telemetry} + alias CodexWrapper.{Config, Exec, ExecResume, JsonLineEvent, Options, Result, Telemetry, Turn} @type t :: %__MODULE__{ config: Config.t(), session_id: String.t() | nil, history: [Result.t()], + last_turn: Turn.t() | nil, exec_opts: keyword() } defstruct [ :config, :session_id, + :last_turn, history: [], exec_opts: [] ] @@ -90,6 +97,11 @@ defmodule CodexWrapper.Session do Internally uses `--json` and takes the session ID from the turn's `thread.started` event. + The updated session exposes the typed projection as `session.last_turn` + and through `last_turn/1`. The returned result and history remain raw + `%Result{}` values. Inspect the projection's `status` for the JSON turn + outcome, which may be `:failed` even when codex exited 0. + Returns: * `{:ok, session, result}` when codex exited 0. `result` carries the @@ -108,9 +120,15 @@ defmodule CodexWrapper.Session do | {:thread_mismatch, String.t(), String.t(), Result.t()} | term()} def send(%__MODULE__{} = session, prompt, opts \\ []) do - with {:ok, result, events} <- execute_turn(session, prompt, opts), - {:ok, session_id} <- next_session_id(session.session_id, events, result) do - updated = %{session | session_id: session_id, history: session.history ++ [result]} + with {:ok, result, turn} <- execute_turn(session, prompt, opts), + {:ok, session_id} <- next_session_id(session.session_id, turn.thread_id, result) do + updated = %{ + session + | session_id: session_id, + history: session.history ++ [result], + last_turn: turn + } + {:ok, updated, result} end end @@ -170,6 +188,34 @@ defmodule CodexWrapper.Session do @spec total_cost(t()) :: float() | nil def total_cost(%__MODULE__{}), do: nil + @doc """ + Sum the reported token usage across recorded turns. + + Returns `nil` when no turn reported a usage map. Otherwise returns the + five usage fields from `CodexWrapper.Turn`, summing only reported + nonnegative integers. A field is `nil` when no turn reported a valid + value for it; an explicitly reported zero remains zero. Missing values + are not estimates, so a total may cover fewer turns than the history. + + Turns rejected by `send/3`, including nonzero exits, are not recorded + and do not contribute to these totals. + """ + @spec total_usage(t()) :: Turn.usage() | nil + def total_usage(%__MODULE__{history: history}) do + Enum.reduce(history, nil, fn result, total -> + result |> Turn.from_result() |> Map.fetch!(:usage) |> sum_usage(total) + end) + end + + @doc """ + Get the typed projection of the last recorded turn, if any. + + Failed calls to `send/3` leave this projection unchanged, along with the + rest of the session. The raw result remains available via `last_result/1`. + """ + @spec last_turn(t()) :: Turn.t() | nil + def last_turn(%__MODULE__{last_turn: turn}), do: turn + @doc """ Get the last result, if any. """ @@ -267,7 +313,7 @@ defmodule CodexWrapper.Session do end defp turn_outcome({:ok, %Result{exit_code: 0} = result}), - do: {:ok, result, JsonLineEvent.parse_lines(result.stdout)} + do: {:ok, result, Turn.from_result(result)} defp turn_outcome({:ok, %Result{exit_code: code} = result}), do: {:error, {:exit, code, result}} @@ -277,8 +323,8 @@ defmodule CodexWrapper.Session do # A first turn adopts the reported thread. A resumed turn keeps its id when # the turn reports none or the same one. A session resumed by name adopts # the UUID codex reports; one resumed by UUID refuses a different id. - defp next_session_id(current, events, result) do - case {current, extract_session_id(events)} do + defp next_session_id(current, reported, result) do + case {current, reported} do {current, nil} -> {:ok, current} {nil, reported} -> {:ok, reported} {same, same} -> {:ok, same} @@ -302,16 +348,16 @@ defmodule CodexWrapper.Session do # applies through `SessionObservation.parse/1`. @doc false @spec extract_session_id([JsonLineEvent.t()]) :: String.t() | nil - def extract_session_id(events) do - Enum.find_value(events, fn - %JsonLineEvent{event_type: "thread.started"} = event -> - thread_id(JsonLineEvent.get(event, "thread_id")) + def extract_session_id(events), do: Turn.from_events(events).thread_id - _event -> - nil + defp sum_usage(nil, total), do: total + defp sum_usage(usage, nil), do: usage + + defp sum_usage(usage, total) do + Map.merge(total, usage, fn + _key, nil, value -> value + _key, value, nil -> value + _key, previous, value -> previous + value end) end - - defp thread_id(id) when is_binary(id), do: if(String.trim(id) != "", do: id) - defp thread_id(_id), do: nil end diff --git a/lib/codex_wrapper/session_server.ex b/lib/codex_wrapper/session_server.ex index 48dd517..b0dd9a1 100644 --- a/lib/codex_wrapper/session_server.ex +++ b/lib/codex_wrapper/session_server.ex @@ -37,7 +37,7 @@ defmodule CodexWrapper.SessionServer do use GenServer - alias CodexWrapper.{Config, Result, Session} + alias CodexWrapper.{Config, Result, Session, Turn} @type server :: GenServer.server() @@ -81,6 +81,9 @@ defmodule CodexWrapper.SessionServer do returns: `{:error, {:exit, code, result}}` for a nonzero exit and `{:error, {:thread_mismatch, expected, reported, result}}` for a different thread ID. A failed turn leaves the session unchanged. + + Use `last_turn/1` to inspect the typed JSON turn outcome and `total_usage/1` + to get reported usage totals after a successful call. """ @spec send_message(server(), String.t(), keyword()) :: {:ok, Result.t()} | {:error, term()} @@ -115,6 +118,24 @@ defmodule CodexWrapper.SessionServer do GenServer.call(server, :total_cost) end + @doc """ + Sum reported token usage across recorded turns. + + See `CodexWrapper.Session.total_usage/1` for missing-value semantics. + """ + @spec total_usage(server()) :: Turn.usage() | nil + def total_usage(server) do + GenServer.call(server, :total_usage) + end + + @doc """ + Get the typed projection of the last recorded turn, if any. + """ + @spec last_turn(server()) :: Turn.t() | nil + def last_turn(server) do + GenServer.call(server, :last_turn) + end + @doc """ Get the last result, if any. """ @@ -173,6 +194,14 @@ defmodule CodexWrapper.SessionServer do {:reply, Session.total_cost(session), session} end + def handle_call(:total_usage, _from, session) do + {:reply, Session.total_usage(session), session} + end + + def handle_call(:last_turn, _from, session) do + {:reply, Session.last_turn(session), session} + end + def handle_call(:last_result, _from, session) do {:reply, Session.last_result(session), session} end diff --git a/lib/codex_wrapper/structured.ex b/lib/codex_wrapper/structured.ex index 1281043..0a5fae1 100644 --- a/lib/codex_wrapper/structured.ex +++ b/lib/codex_wrapper/structured.ex @@ -38,7 +38,9 @@ defmodule CodexWrapper.Structured do ## Outcomes Success is `{:ok, value, detail}` where `detail` is - `%{result: Result.t(), events: [JsonLineEvent.t()], text: String.t()}`. + `%{result: Result.t(), events: [JsonLineEvent.t()], turn: Turn.t(), text: String.t()}`. + The `turn` is the shared `CodexWrapper.Turn` projection, including native + outcome, thread identity and token usage. Failures after the process ran carry the same `detail` map (plus the fields noted) so the native output is never lost: @@ -64,13 +66,14 @@ defmodule CodexWrapper.Structured do `{:unsupported, capability}` are returned unchanged. """ - alias CodexWrapper.{Config, Exec, ExecFork, ExecResume, JsonLineEvent, Result, Review} + alias CodexWrapper.{Config, Exec, ExecFork, ExecResume, JsonLineEvent, Result, Review, Turn} @type command :: Exec.t() | ExecResume.t() | ExecFork.t() | Review.t() @type detail :: %{ required(:result) => Result.t(), required(:events) => [JsonLineEvent.t()], + required(:turn) => Turn.t(), optional(atom()) => term() } @@ -130,11 +133,11 @@ defmodule CodexWrapper.Structured do """ @spec decode(Result.t(), keyword()) :: {:ok, term(), detail()} | {:error, error()} def decode(%Result{} = result, opts \\ []) do - events = JsonLineEvent.parse_lines(result.stdout) - detail = %{result: result, events: events} + turn = Turn.from_result(result) + detail = %{result: result, events: turn.events, turn: turn} - with :ok <- check_turn(events, result, detail), - {:ok, text} <- final_message(events, detail), + with :ok <- check_turn(turn, detail), + {:ok, text} <- final_message(turn, detail), detail = Map.put(detail, :text, text), {:ok, value} <- decode_json(text, detail), {:ok, parsed} <- parse(Keyword.get(opts, :parse), value, detail) do @@ -175,61 +178,21 @@ defmodule CodexWrapper.Structured do end end - defp check_turn(events, result, detail) do - case terminal(events) do - %JsonLineEvent{event_type: "turn.failed", data: data} -> - {:error, {:turn_failed, Map.put(detail, :error, data["error"])}} + defp check_turn(%Turn{status: :completed}, _detail), do: :ok - %JsonLineEvent{event_type: "turn.completed"} when result.exit_code != 0 -> - {:error, {:process_failed, detail}} + defp check_turn(%Turn{status: :no_terminal}, detail), + do: {:error, {:no_terminal, detail}} - %JsonLineEvent{event_type: "turn.completed"} -> - :ok - - nil -> - no_terminal(events, result, detail) - end - end - - defp no_terminal(events, result, detail) do - case Enum.filter(events, &(&1.event_type == "error")) do - [] when result.exit_code != 0 -> {:error, {:process_failed, detail}} - [] -> {:error, {:no_terminal, detail}} - errors -> {:error, {:turn_failed, Map.put(detail, :error, List.last(errors).data)}} - end - end - - defp terminal(events) do - events - |> Enum.filter(&(&1.event_type in ["turn.completed", "turn.failed"])) - |> List.last() - end - - defp final_message(events, detail) do - events - |> final_turn() - |> Enum.filter(&agent_message?/1) - |> List.last() - |> case do - %JsonLineEvent{data: %{"item" => %{"text" => text}}} when is_binary(text) and text != "" -> - {:ok, text} - - _ -> - {:error, {:no_structured_output, detail}} - end - end - - defp final_turn(events) do - case Enum.find_index(Enum.reverse(events), &(&1.event_type == "turn.started")) do - nil -> events - index -> Enum.take(events, -index) - end + defp check_turn(%Turn{} = turn, detail) do + if turn.error != nil or match?(%JsonLineEvent{event_type: "turn.failed"}, turn.terminal_event), + do: {:error, {:turn_failed, Map.put(detail, :error, turn.error)}}, + else: {:error, {:process_failed, detail}} end - defp agent_message?(%JsonLineEvent{event_type: "item.completed", data: data}), - do: match?(%{"item" => %{"type" => "agent_message"}}, data) + defp final_message(%Turn{final_message: text}, _detail) when is_binary(text) and text != "", + do: {:ok, text} - defp agent_message?(_event), do: false + defp final_message(_turn, detail), do: {:error, {:no_structured_output, detail}} defp decode_json(text, detail) do case Jason.decode(text) do diff --git a/lib/codex_wrapper/turn.ex b/lib/codex_wrapper/turn.ex new file mode 100644 index 0000000..4eed61e --- /dev/null +++ b/lib/codex_wrapper/turn.ex @@ -0,0 +1,151 @@ +defmodule CodexWrapper.Turn do + @moduledoc """ + A pure projection of a command's `--json` output. + + `from_result/1` starts no process and leaves the raw `CodexWrapper.Result` + unchanged. Ordinary text messages do not need to contain JSON: + + {:ok, result} = CodexWrapper.exec("Summarize the project", json: true) + turn = CodexWrapper.Turn.from_result(result) + turn.final_message + + The last `turn.completed` or `turn.failed` determines the native outcome. + A top-level `error` fails the turn until a later terminal event supersedes + it. A nonzero process exit always makes `status` `:failed`, including after + `turn.completed`. Without a terminal event or error, status is `:no_terminal` + when the process exited zero or its exit code is unknown. + + `final_message` is the last completed `agent_message` text after the last + `turn.started`. `thread_id` is the first nonblank string ID announced by + `thread.started`. Usage comes from the last terminal event of the last + turn. Only the five documented token fields are projected, each as a + nonnegative integer or `nil`; missing usage is `nil`, not zero. + + `error` retains native error data, or is `nil` when no native error was + reported. A process failure alone does not fabricate an error object. + `events` keeps all parsed events, including unknown fields and invalid + usage values. `terminal_event` retains the last native terminal event, + independently of process exit and subsequent top-level errors. + """ + + alias CodexWrapper.{JsonLineEvent, Result} + + @usage_fields [ + :input_tokens, + :cached_input_tokens, + :cache_write_input_tokens, + :output_tokens, + :reasoning_output_tokens + ] + + @type usage :: %{ + input_tokens: non_neg_integer() | nil, + cached_input_tokens: non_neg_integer() | nil, + cache_write_input_tokens: non_neg_integer() | nil, + output_tokens: non_neg_integer() | nil, + reasoning_output_tokens: non_neg_integer() | nil + } + + @type t :: %__MODULE__{ + status: :completed | :failed | :no_terminal, + exit_code: non_neg_integer() | nil, + thread_id: String.t() | nil, + final_message: String.t() | nil, + usage: usage() | nil, + error: term(), + events: [JsonLineEvent.t()], + terminal_event: JsonLineEvent.t() | nil + } + + defstruct [ + :exit_code, + :thread_id, + :final_message, + :usage, + :error, + :terminal_event, + status: :no_terminal, + events: [] + ] + + @doc """ + Project a completed command result without starting a process. + + The command must have requested `--json`. Non-JSON lines are ignored by + `CodexWrapper.JsonLineEvent.parse_lines/1` and remain in the original result. + """ + @spec from_result(Result.t()) :: t() + def from_result(%Result{} = result) do + result.stdout + |> JsonLineEvent.parse_lines() + |> from_events(result.exit_code) + end + + @doc """ + Project already parsed events, optionally with a known process exit code. + + With the default `nil` exit code, status describes only the native events; + it does not establish that a process exited successfully. + """ + @spec from_events([JsonLineEvent.t()], non_neg_integer() | nil) :: t() + def from_events(events, exit_code \\ nil) do + turn = Enum.reduce(events, %__MODULE__{events: events, exit_code: exit_code}, &project/2) + + if is_integer(exit_code) and exit_code != 0, + do: %{turn | status: :failed}, + else: turn + end + + defp project(%JsonLineEvent{event_type: "thread.started", data: data}, turn) do + %{turn | thread_id: turn.thread_id || thread_id(data["thread_id"])} + end + + defp project(%JsonLineEvent{event_type: "turn.started"}, turn), + do: %{turn | final_message: nil, usage: nil} + + defp project( + %JsonLineEvent{ + event_type: "item.completed", + data: %{"item" => %{"type" => "agent_message"} = item} + }, + turn + ) do + text = if is_binary(item["text"]), do: item["text"] + %{turn | final_message: text} + end + + defp project(%JsonLineEvent{event_type: "turn.completed", data: data} = event, turn), + do: %{ + turn + | status: :completed, + error: nil, + usage: usage(data["usage"]), + terminal_event: event + } + + defp project(%JsonLineEvent{event_type: "turn.failed", data: data} = event, turn), + do: %{ + turn + | status: :failed, + error: data["error"], + usage: usage(data["usage"]), + terminal_event: event + } + + defp project(%JsonLineEvent{event_type: "error", data: data}, turn), + do: %{turn | status: :failed, error: data} + + defp project(_event, turn), do: turn + + defp thread_id(id) when is_binary(id), do: if(String.trim(id) != "", do: id) + defp thread_id(_id), do: nil + + defp usage(data) when is_map(data) do + Map.new(@usage_fields, fn field -> + value = data[Atom.to_string(field)] + {field, if(is_integer(value) and value >= 0, do: value)} + end) + end + + defp usage(_data), do: nil +end diff --git a/mix.exs b/mix.exs index b28e7b3..cc0f208 100644 --- a/mix.exs +++ b/mix.exs @@ -61,6 +61,7 @@ defmodule CodexWrapperEx.MixProject do "Config, results, support": [ CodexWrapper.Config, CodexWrapper.Result, + CodexWrapper.Turn, CodexWrapper.LastMessage, CodexWrapper.SessionObservation, CodexWrapper.ObservedExecution, diff --git a/test/codex_wrapper/evidenced_execution_test.exs b/test/codex_wrapper/evidenced_execution_test.exs index a66fc6f..10880e5 100644 --- a/test/codex_wrapper/evidenced_execution_test.exs +++ b/test/codex_wrapper/evidenced_execution_test.exs @@ -8,7 +8,8 @@ defmodule CodexWrapper.EvidencedExecutionTest do ExecResume, ExecutionObservation, Result, - SessionObservation + SessionObservation, + Turn } defmodule Fake do @@ -77,6 +78,52 @@ defmodule CodexWrapper.EvidencedExecutionTest do response(stdout, 7) end + # Synthetic native event sequences for projection and bounded-output tests. + defp scenario("completed-then-error", stdout, _timeout, _collector, _ref) do + completed = %{ + "type" => "turn.completed", + "usage" => %{ + "input_tokens" => 12, + "cached_input_tokens" => 3, + "cache_write_input_tokens" => 2, + "output_tokens" => 7, + "reasoning_output_tokens" => 1, + "future_tokens" => 9 + } + } + + response( + stdout <> + ~s({"type":"turn.failed","error":{"message":"earlier failure"}}\n) <> + ~s({"type":"error","message":"reconnecting"}\n) <> + Jason.encode!(completed) <> + "\n" <> + ~s({"type":"error","message":"late failure"}\n), + 0 + ) + end + + defp scenario("last-failed", stdout, _timeout, _collector, _ref) do + response(stdout <> ~s({"type":"turn.failed","error":{"message":"last failure"}}\n), 0) + end + + defp scenario("replacement-no-terminal", _stdout, _timeout, _collector, _ref) do + response(~s({"type":"turn.started"}\n), 0) + end + + defp scenario("replacement-empty", _stdout, _timeout, _collector, _ref) do + response("", 0) + end + + defp scenario("oversized-terminal", stdout, _timeout, _collector, _ref) do + failed = %{ + "type" => "turn.failed", + "error" => %{"message" => String.duplicate("x", SessionObservation.max_line_bytes())} + } + + response(stdout <> Jason.encode!(failed) <> "\n", 0) + end + defp scenario("worker-death", _stdout, _timeout, _collector, _ref) do Process.exit(self(), :kill) end @@ -245,6 +292,72 @@ defmodule CodexWrapper.EvidencedExecutionTest do end end + test "raw parsed results share the projection's terminal selection" do + for mode <- ["completed-then-error", "last-failed"] do + ref = make_ref() + assert {:ok, %Result{} = result} = execute(mode, ref) + assert_receive {:invoked, _, _, _, _, _, _, _} + assert_receive {^ref, %SessionObservation{}} + assert_receive {^ref, %ExecutionObservation{} = fact} + + turn = Turn.from_result(result) + assert turn.status == :failed + assert fact.parsed_result == turn.terminal_event.data + assert fact.stdout == result.stdout + assert fact.child_status == 0 + + if mode == "completed-then-error" do + assert fact.parsed_result["type"] == "turn.completed" + assert fact.parsed_result["usage"]["future_tokens"] == 9 + assert turn.usage.reasoning_output_tokens == 1 + else + assert fact.parsed_result["type"] == "turn.failed" + end + end + end + + test "final evidence replaces a previously streamed terminal event" do + for mode <- ["replacement-no-terminal", "replacement-empty"] do + ref = make_ref() + assert {:ok, %Result{} = result} = execute(mode, ref) + assert_receive {:invoked, _, _, _, _, _, _, _} + assert_receive {^ref, %SessionObservation{}} + assert_receive {^ref, %ExecutionObservation{parsed_result: nil} = fact} + assert fact.stdout == result.stdout + assert Turn.from_result(result).terminal_event == nil + end + end + + test "truncated final events cannot replace the last complete bounded event" do + first = ~s({"type":"thread.started","thread_id":"native-thread"}\n) + completed = ~s({"type":"turn.completed","usage":{}}\n) + limit = byte_size(first <> first <> completed) + 12 + ref = make_ref() + + assert {:ok, %Result{} = result} = execute("last-failed", ref, max_output_bytes: limit) + assert_receive {:invoked, _, _, _, _, _, _, _} + assert_receive {^ref, %SessionObservation{}} + assert_receive {^ref, %ExecutionObservation{output_truncated: true} = fact} + assert byte_size(fact.stdout) + byte_size(fact.stderr) == limit + assert fact.parsed_result == Turn.from_result(result).terminal_event.data + assert fact.parsed_result["type"] == "turn.completed" + end + + test "terminal projection preserves the evidenced line-size limit" do + ref = make_ref() + limit = SessionObservation.max_line_bytes() * 2 + + assert {:ok, %Result{}} = execute("oversized-terminal", ref, max_output_bytes: limit) + assert_receive {:invoked, _, _, _, _, _, _, _} + assert_receive {^ref, %SessionObservation{}} + + assert_receive {^ref, + %ExecutionObservation{ + parsed_result: %{"type" => "turn.completed"}, + output_truncated: :unknown + }} + end + test "partial combined output stays bounded on limit and worker death" do for mode <- ["oversize", "worker-death"] do ref = make_ref() diff --git a/test/codex_wrapper/session_server_test.exs b/test/codex_wrapper/session_server_test.exs index 9bcb4b0..9e2c843 100644 --- a/test/codex_wrapper/session_server_test.exs +++ b/test/codex_wrapper/session_server_test.exs @@ -11,6 +11,8 @@ defmodule CodexWrapper.SessionServerTest do assert SessionServer.session_id(pid) == nil assert SessionServer.turn_count(pid) == 0 assert SessionServer.total_cost(pid) == nil + assert SessionServer.total_usage(pid) == nil + assert SessionServer.last_turn(pid) == nil assert SessionServer.last_result(pid) == nil assert SessionServer.history(pid) == [] end diff --git a/test/codex_wrapper/session_test.exs b/test/codex_wrapper/session_test.exs index 223f4ba..39398ee 100644 --- a/test/codex_wrapper/session_test.exs +++ b/test/codex_wrapper/session_test.exs @@ -1,7 +1,7 @@ defmodule CodexWrapper.SessionTest do use ExUnit.Case, async: true - alias CodexWrapper.{Config, JsonLineEvent, Session} + alias CodexWrapper.{Config, JsonLineEvent, Result, Session} defp event(type, data), do: %JsonLineEvent{event_type: type, data: data, raw: ""} @@ -12,6 +12,7 @@ defmodule CodexWrapper.SessionTest do assert session.config == config assert session.session_id == nil assert session.history == [] + assert session.last_turn == nil assert session.exec_opts == [] end @@ -81,6 +82,58 @@ defmodule CodexWrapper.SessionTest do end end + describe "last_turn/1" do + test "returns nil before a turn is recorded" do + assert Session.last_turn(Session.new(Config.new())) == nil + end + end + + describe "total_usage/1" do + test "returns nil when there is no reported usage" do + session = Session.new(Config.new()) + assert Session.total_usage(session) == nil + + result = Result.from_cmd({~s({"type":"turn.completed"}), 0}) + assert Session.total_usage(%{session | history: [result]}) == nil + end + + test "sums known counters while preserving unknown values and zero" do + usages = [ + %{"input_tokens" => 12, "cached_input_tokens" => 0, "output_tokens" => 3}, + %{"input_tokens" => 5, "cache_write_input_tokens" => 2, "output_tokens" => -1}, + %{"input_tokens" => "9", "cached_input_tokens" => nil, "output_tokens" => 1.5} + ] + + results = + Enum.map(usages, fn usage -> + Result.from_cmd({Jason.encode!(%{"type" => "turn.completed", "usage" => usage}), 0}) + end) + + session = %{Session.new(Config.new()) | history: results} + + assert Session.total_usage(session) == %{ + input_tokens: 17, + cached_input_tokens: 0, + cache_write_input_tokens: 2, + output_tokens: 3, + reasoning_output_tokens: nil + } + end + + test "a reported empty usage map retains unknown counters" do + result = Result.from_cmd({~s({"type":"turn.completed","usage":{}}), 0}) + session = %{Session.new(Config.new()) | history: [result]} + + assert Session.total_usage(session) == %{ + input_tokens: nil, + cached_input_tokens: nil, + cache_write_input_tokens: nil, + output_tokens: nil, + reasoning_output_tokens: nil + } + end + end + describe "turns/1" do test "returns empty list initially" do config = Config.new() diff --git a/test/codex_wrapper/session_turn_test.exs b/test/codex_wrapper/session_turn_test.exs index 1c6d791..b178b99 100644 --- a/test/codex_wrapper/session_turn_test.exs +++ b/test/codex_wrapper/session_turn_test.exs @@ -2,7 +2,7 @@ defmodule CodexWrapper.SessionTurnTest do # Not async: attaches a global telemetry handler. use ExUnit.Case, async: false - alias CodexWrapper.{Config, Result, Session, SessionServer} + alias CodexWrapper.{Config, Result, Session, SessionServer, Turn} @moduletag :tmp_dir @@ -98,6 +98,86 @@ defmodule CodexWrapper.SessionTurnTest do assert Session.session_id(session) == @uuid assert Session.turn_count(session) == 1 end + + test "exposes typed turns and sums usage over repeated sends", %{tmp_dir: dir} do + first_lines = [ + started(@uuid), + ~s({"type":"turn.started"}), + ~s({"type":"item.completed","item":{"type":"agent_message","text":"first"}}), + ~s({"type":"turn.completed","usage":{"input_tokens":12,"cached_input_tokens":0,"output_tokens":3}}) + ] + + config = fake(dir, first_lines, nil, 0) + assert {:ok, first, %Result{} = first_result} = Session.send(Session.new(config), "first") + + assert %Turn{status: :completed, final_message: "first", thread_id: @uuid} = + Session.last_turn(first) + + assert first.last_turn == Turn.from_result(first_result) + assert Session.history(first) == [first_result] + + second_lines = [ + started(@uuid), + ~s({"type":"turn.started"}), + ~s({"type":"item.completed","item":{"type":"agent_message","text":"second"}}), + ~s({"type":"turn.completed","usage":{"input_tokens":5,"cache_write_input_tokens":2,"output_tokens":4,"reasoning_output_tokens":1}}) + ] + + fake(dir, second_lines, nil, 0) + assert {:ok, second, %Result{} = second_result} = Session.send(first, "second") + + assert Session.last_turn(second).final_message == "second" + assert Session.last_result(second) == second_result + assert Session.history(second) == [first_result, second_result] + + assert Session.total_usage(second) == %{ + input_tokens: 17, + cached_input_tokens: 0, + cache_write_input_tokens: 2, + output_tokens: 7, + reasoning_output_tokens: 1 + } + end + + test "exposes a failed native outcome despite a zero process exit", %{tmp_dir: dir} do + lines = [ + started(@uuid), + ~s({"type":"turn.failed","error":{"message":"synthetic failure"}}) + ] + + config = fake(dir, lines, nil, 0) + + assert {:ok, session, %Result{success: true}} = Session.send(Session.new(config), "first") + + assert %Turn{status: :failed, error: %{"message" => "synthetic failure"}} = + session.last_turn + + assert Session.total_usage(session) == nil + end + end + + describe "typed session server access" do + test "reports usage and preserves the previous turn when a later call fails", %{tmp_dir: dir} do + lines = [ + started(@uuid), + ~s({"type":"turn.completed","usage":{"input_tokens":8,"output_tokens":0}}) + ] + + config = fake(dir, lines, nil, 0) + {:ok, pid} = SessionServer.start_link(config: config) + assert {:ok, %Result{} = result} = SessionServer.send_message(pid, "first") + turn = Turn.from_result(result) + assert SessionServer.last_turn(pid) == turn + assert SessionServer.total_usage(pid) == turn.usage + assert SessionServer.last_result(pid) == result + + fake(dir, lines, "synthetic failure", 7) + assert {:error, {:exit, 7, %Result{}}} = SessionServer.send_message(pid, "second") + assert SessionServer.last_turn(pid) == turn + assert SessionServer.total_usage(pid) == turn.usage + assert SessionServer.history(pid) == [result] + assert SessionServer.turn_count(pid) == 1 + end end describe "thread id" do diff --git a/test/codex_wrapper/structured_test.exs b/test/codex_wrapper/structured_test.exs index 2b4c2aa..92cc40a 100644 --- a/test/codex_wrapper/structured_test.exs +++ b/test/codex_wrapper/structured_test.exs @@ -1,7 +1,7 @@ defmodule CodexWrapper.StructuredTest do use ExUnit.Case, async: true - alias CodexWrapper.{Config, Exec, ExecResume, JsonLineEvent, Result, Review, Structured} + alias CodexWrapper.{Config, Exec, ExecResume, JsonLineEvent, Result, Review, Structured, Turn} @fake Path.expand("../fixtures/fake_codex_structured", __DIR__) @@ -167,6 +167,60 @@ defmodule CodexWrapper.StructuredTest do end describe "decode/2" do + test "shares the typed projection on success and every native failure" do + message = %{ + "type" => "item.completed", + "item" => %{"type" => "agent_message", "text" => ~s({"name":"Ada"})} + } + + completed = %{"type" => "turn.completed", "usage" => %{"input_tokens" => 12}} + failed = %{"type" => "turn.failed"} + error = %{"type" => "error", "message" => "late failure"} + + for {events, code, expected} <- [ + {[message, completed], 0, :ok}, + {[message, completed], 1, :process_failed}, + {[message, error, completed], 0, :ok}, + {[message, completed, error], 0, :turn_failed}, + {[message, completed, error], 1, :turn_failed}, + {[message, failed, completed], 0, :ok}, + {[message, completed, failed], 0, :turn_failed}, + {[message, failed], 0, :turn_failed}, + {[message], 0, :no_terminal}, + {[message], 1, :process_failed} + ] do + result = Result.from_cmd({Enum.map_join(events, "\n", &Jason.encode!/1), code}) + + detail = + case Structured.decode(result) do + {:ok, %{"name" => "Ada"}, detail} -> + assert expected == :ok + detail + + {:error, {kind, detail}} -> + assert kind == expected + detail + end + + assert detail.turn == Turn.from_result(result) + assert detail.events == detail.turn.events + assert detail.result == result + end + end + + test "does not decode a previous turn's message when the final turn has none" do + stdout = """ + {"type":"turn.started"} + {"type":"item.completed","item":{"type":"agent_message","text":"{}"}} + {"type":"turn.completed"} + {"type":"turn.started"} + {"type":"turn.completed"} + """ + + assert {:error, {:no_structured_output, %{turn: %Turn{final_message: nil}}}} = + Structured.decode(Result.from_cmd({stdout, 0})) + end + test "decodes a result the caller already holds" do stdout = Enum.map_join( diff --git a/test/codex_wrapper/turn_test.exs b/test/codex_wrapper/turn_test.exs new file mode 100644 index 0000000..0e80cdf --- /dev/null +++ b/test/codex_wrapper/turn_test.exs @@ -0,0 +1,196 @@ +defmodule CodexWrapper.TurnTest do + use ExUnit.Case, async: true + + alias CodexWrapper.{JsonLineEvent, Result, Turn} + + @fixtures Path.expand("../fixtures/turns", __DIR__) + @fields [ + :input_tokens, + :cached_input_tokens, + :cache_write_input_tokens, + :output_tokens, + :reasoning_output_tokens + ] + + defp result(events, code \\ 0) do + Result.from_cmd({Enum.map_join(events, "\n", &Jason.encode!/1), code}) + end + + defp event(type, data \\ %{}), do: Map.put(data, "type", type) + + defp message(text, type \\ "item.completed") do + event(type, %{"item" => %{"type" => "agent_message", "text" => text}}) + end + + test "projects the synthetic 0.165.0 fixture without changing the raw result" do + stdout = File.read!(Path.join(@fixtures, "0.165.0-completed.ndjson")) + result = Result.from_cmd({stdout, 0}) + turn = Turn.from_result(result) + + assert %Turn{ + status: :completed, + exit_code: 0, + thread_id: "019b1e96-6935-7a02-a61e-325504b0b832", + final_message: "done", + error: nil, + usage: %{ + input_tokens: 120, + cached_input_tokens: 100, + cache_write_input_tokens: 0, + output_tokens: 7, + reasoning_output_tokens: 3 + } + } = turn + + assert turn.events == JsonLineEvent.parse_lines(stdout) + assert turn.terminal_event == List.last(turn.events) + assert turn.terminal_event.data["usage"]["future_tokens"] == 99 + refute Map.has_key?(turn.usage, :future_tokens) + assert result == %Result{stdout: stdout, stderr: "", exit_code: 0, success: true} + end + + test "status follows terminal and error order, with nonzero exit overriding completion" do + completed = event("turn.completed") + failed = event("turn.failed", %{"error" => %{"message" => "failed"}}) + error = event("error", %{"message" => "stream error"}) + + for {events, code, status, native_error} <- [ + {[], 0, :no_terminal, nil}, + {[], 2, :failed, nil}, + {[completed], 0, :completed, nil}, + {[completed], 2, :failed, nil}, + {[failed], 0, :failed, failed["error"]}, + {[failed], 2, :failed, failed["error"]}, + {[failed, completed], 0, :completed, nil}, + {[completed, failed], 0, :failed, failed["error"]}, + {[error], 0, :failed, error}, + {[error, completed], 0, :completed, nil}, + {[completed, error], 0, :failed, error}, + {[completed, error], 2, :failed, error}, + {[completed, error, completed], 0, :completed, nil}, + {[error, failed], 0, :failed, failed["error"]} + ] do + turn = Turn.from_result(result(events, code)) + assert turn.status == status, inspect({events, code}) + assert turn.error == native_error + assert turn.exit_code == code + end + end + + test "uses exit_code even when the Result.success flag disagrees" do + assert %Turn{status: :failed} = + Turn.from_result(%{result([event("turn.completed")], 1) | success: true}) + + assert %Turn{status: :completed} = + Turn.from_result(%{result([event("turn.completed")]) | success: false}) + end + + test "a failed event without native error data is still failed" do + assert %Turn{status: :failed, error: nil, terminal_event: %JsonLineEvent{}} = + Turn.from_result(result([event("turn.failed")])) + end + + test "uses the last completed agent message from the last turn" do + events = [ + event("turn.started"), + message("previous turn"), + event("turn.completed"), + event("turn.started"), + message("draft"), + event("item.completed", %{"item" => %{"type" => "reasoning", "text" => "thinking"}}), + message("final"), + message("not completed", "item.updated"), + event("turn.completed") + ] + + turn = Turn.from_result(result(events)) + assert turn.final_message == "final" + assert length(turn.events) == length(events) + assert Turn.from_result(result(events ++ [event("turn.started")])).final_message == nil + end + + test "missing turn.started is allowed, and malformed final message does not reuse earlier text" do + assert Turn.from_result(result([message("hello")])).final_message == "hello" + + for text <- [nil, 123, %{}, []] do + assert Turn.from_result(result([message("earlier"), message(text)])).final_message == nil + end + + assert Turn.from_result(result([message("")])).final_message == "" + + assert Turn.from_result(result([event("item.completed", %{"item" => nil})])).final_message == + nil + end + + test "thread identity uses the first nonblank string on thread.started only" do + events = [ + event("result", %{"session_id" => "legacy"}), + event("item.completed", %{"thread_id" => "wrong"}), + event("thread.started", %{"thread_id" => 7}), + event("thread.started", %{"thread_id" => " \t"}), + event("thread.started", %{"thread_id" => "native"}), + event("thread.started", %{"thread_id" => "later"}) + ] + + assert Turn.from_result(result(events)).thread_id == "native" + assert Turn.from_result(result([])).thread_id == nil + end + + test "invalid synthetic usage remains in raw events but becomes nil in the projection" do + stdout = File.read!(Path.join(@fixtures, "0.165.0-invalid-usage.ndjson")) + turn = Turn.from_result(Result.from_cmd({stdout, 0})) + assert turn.usage == Map.new(@fields, &{&1, nil}) + assert turn.terminal_event.data["usage"]["input_tokens"] == -1 + assert turn.terminal_event.data["usage"]["cached_input_tokens"] == "100" + assert turn.terminal_event.data["usage"]["cache_write_input_tokens"] == 1.5 + assert turn.terminal_event.data["usage"]["output_tokens"] == true + assert turn.terminal_event.data["usage"]["future_tokens"] == 99 + end + + test "every known usage field accepts only nonnegative integers" do + for field <- @fields, value <- [0, 9, -1, 1.0, "9", true, false, nil, [], %{}] do + usage = %{Atom.to_string(field) => value} + turn = Turn.from_result(result([event("turn.completed", %{"usage" => usage})])) + expected = if is_integer(value) and value >= 0, do: value + assert turn.usage[field] == expected + assert Map.keys(turn.usage) |> Enum.sort() == Enum.sort(@fields) + end + end + + test "absent or malformed usage is unknown, and never leaks across turns or terminals" do + first = event("turn.completed", %{"usage" => %{"input_tokens" => 5}}) + + for usage <- [nil, [], 0, "unknown"] do + assert Turn.from_result(result([event("turn.completed", %{"usage" => usage})])).usage == nil + end + + assert Turn.from_result(result([first, event("turn.started")])).usage == nil + assert Turn.from_result(result([first, event("turn.failed")])).usage == nil + assert Turn.from_result(result([first, event("turn.completed")])).usage == nil + + assert Turn.from_result(result([event("turn.completed", %{"usage" => %{}})])).usage == + Map.new(@fields, &{&1, nil}) + end + + test "partial events project without inventing an exit code" do + events = JsonLineEvent.parse_lines(~s({"type":"turn.completed"})) + assert %Turn{status: :completed, exit_code: nil} = Turn.from_events(events) + assert %Turn{status: :failed, exit_code: 3} = Turn.from_events(events, 3) + assert %Turn{status: :no_terminal, exit_code: nil} = Turn.from_events([]) + end + + test "noise, malformed JSON and unknown event types do not obscure valid events" do + stdout = + "diagnostic\n{invalid\n[]\n" <> result([message("done"), event("turn.completed")]).stdout + + turn = Turn.from_result(Result.from_cmd({stdout, 0})) + assert turn.status == :completed + assert turn.final_message == "done" + assert length(turn.events) == 2 + + unknown = event("future.event", %{"new_field" => 123}) + turn = Turn.from_result(result([unknown])) + assert turn.status == :no_terminal + assert hd(turn.events).data == unknown + end +end diff --git a/test/fixtures/turns/0.165.0-completed.ndjson b/test/fixtures/turns/0.165.0-completed.ndjson new file mode 100644 index 0000000..68cdeb0 --- /dev/null +++ b/test/fixtures/turns/0.165.0-completed.ndjson @@ -0,0 +1,4 @@ +{"type":"thread.started","thread_id":"019b1e96-6935-7a02-a61e-325504b0b832"} +{"type":"turn.started"} +{"type":"item.completed","item":{"id":"item_0","type":"agent_message","text":"done"}} +{"type":"turn.completed","usage":{"input_tokens":120,"cached_input_tokens":100,"cache_write_input_tokens":0,"output_tokens":7,"reasoning_output_tokens":3,"future_tokens":99}} diff --git a/test/fixtures/turns/0.165.0-invalid-usage.ndjson b/test/fixtures/turns/0.165.0-invalid-usage.ndjson new file mode 100644 index 0000000..33c7f6f --- /dev/null +++ b/test/fixtures/turns/0.165.0-invalid-usage.ndjson @@ -0,0 +1,4 @@ +{"type":"thread.started","thread_id":"019b1e96-6935-7a02-a61e-325504b0b832"} +{"type":"turn.started"} +{"type":"item.completed","item":{"id":"item_0","type":"agent_message","text":"done"}} +{"type":"turn.completed","usage":{"input_tokens":-1,"cached_input_tokens":"100","cache_write_input_tokens":1.5,"output_tokens":true,"reasoning_output_tokens":null,"future_tokens":99}} diff --git a/test/fixtures/turns/README.md b/test/fixtures/turns/README.md new file mode 100644 index 0000000..01d170e --- /dev/null +++ b/test/fixtures/turns/README.md @@ -0,0 +1,13 @@ +# Turn fixtures + +Both NDJSON fixtures are **synthetic**, not captured from a provider run. +The five usage field names follow the codex-cli 0.165.0 binary's JSON event +fields recorded in issue #150. Event envelopes follow the synthetic +`fake_codex_structured` fixture based on @openai/codex-sdk 0.164.0. + +- `0.165.0-completed.ndjson`: ordinary text completion with all five usage + fields, including an explicit zero. `future_tokens` is an invented unknown + field used to verify that raw events retain fields outside the projection. +- `0.165.0-invalid-usage.ndjson`: deliberately invalid usage values to verify + that the typed projection rejects negatives, strings, floats, booleans, + and null without altering raw event data.