[assafelovic]

What this PR does

Journeys go from one sample to a test you can run deeply on your own site. ax journey can now run a set: the same goals at up to four levels of help, across up to four agents, repeated, on your Front account. It previews the run count and what is left of your plan, asks before launching, follows the runs until the platform grades them, and prints a matrix with gap lines that name the layer where agents lose your site; ax journey view|list|cancel pick a set up again, list sets and stop one. One goal with one agent still runs one journey and draws it live. ax journey is also now the one journey command: with no goal of its own it runs one of ora's curated tasks free, which ax deep-journey did, and ax deep-journey keeps working as a deprecated alias. Every command that sends a key now checks where it goes first.

Why

One journey says an agent failed on your site once, but not how often, with which agents, or whether a little help would have got it through. Experiment platforms for developer tools answer those questions for CLIs and SDKs; this brings the same depth to websites, adapted to ora's score layers:

Developers expect	ax after this PR
Runs across agents and models, repeated	A set: 1 to 5 goals x up to 4 levels x 1 to 4 agents x 1 to 10 repeats, at most 200 runs
A ladder of guidance, from a bare need to the exact command	Levels of help: Need only, Brand named, Docs linked, Exact target, each a fixed template
Rates with their run counts	A matrix per goal, level and agent: a rate from 5 graded runs, counts below that, and gap lines naming the score layer
A preview before anything is spent	The run count and the plan's runs left, a confirm, and a refusal when the plan cannot cover it
Infrastructure failures kept apart from results	Failed, timed out, canceled and lost runs are excluded and never counted against the site
Flags for runs that cannot be trusted	blocked and never visited, counted beside the verdicts
One next step on every result and error	Next: lines, one --json error envelope, and ax learn
Leave a long run and pick it up later	ax journey view, list and cancel; Ctrl-C detaches

Not in this release: checks beside the judge, sets in CI with a PR comment, a query over runs, and an MCP tool for sets.

How ax journey works now

flowchart TD
  Z["ax journey ..."] --> G{"A goal of your own?"}
  G -- "no, or --intent / --agent / --no-stream" --> FR["ora.ai's public journeys: a curated task,<br/>free, no key (was ax deep-journey)"]
  G -- yes --> A["ax journey 'goal' --domain site ..."]
  A --> M{"One goal, one agent,<br/>no --repeat, no levels?"}
  M -- yes --> O["One journey, drawn live as a graph<br/>(unchanged; exit 0, 1 or 2)"]
  M -- no --> V["Check the set: one site, 1 to 5 goals,<br/>up to 4 levels, 1 to 4 agents,<br/>1 to 10 repeats, at most 200 runs"]
  V --> P{"Preview: 'This set uses 30 runs ...'<br/>'Launch 30 runs? [y/N]'"}
  P -- "no, or a refusal" --> X["Nothing launched"]
  P -- "yes, or --yes" --> R["Write the record first:<br/>~/.ax/journeys/j-id.json"]
  R --> L["Create one journey per goal, level and repeat;<br/>each runs every agent"]
  L --> F["Follow until every run is graded,<br/>or --wait runs out"]
  F --> C["Count per goal, level and agent<br/>(the platform grades, ax only counts)"]
  C --> T["Report: the matrix plus gap lines,<br/>e.g. 'loses agents at the Discovery layer'"]
  T --> N["Next: ax journey view id while runs are going,<br/>otherwise ax audit site"]
  S[("Your Front account at app.ora.ai")]
  O -.-> S
  L -.-> S
  F -.-> S

Loading

You want	Command
One of ora's curated tasks, free, no key	ax journey --domain your-site.com [--intent pricing]
One journey on your own goal	ax journey "Find the API docs" --domain your-site.com
A set of journeys	Add --levels, --docs <url>, --exact <url> or --brand <name>, and/or --agents a,b and --repeat N (plus --yes, --wait <min>, --json)
Pick a set up again, from any terminal	ax journey view <id>
Sets recorded on this machine	ax journey list
Stop a set	ax journey cancel <id>

• Levels of help: Need only (the goal as written, skipped when it names the site), Brand named, Docs linked (with --docs) and Exact target (with --exact). Every level is a fixed prompt template; no model writes a prompt. Without levels, a set runs the goal as written.
• The report: levels down, agents across; a cell reads 60% (3 of 5) from 5 gradable runs and 3 of 4 below that. Excluded runs (never started, failed, timed out, canceled) never count against the site. Two flags are counted beside the verdicts and never change one: blocked (the rule ora.ai uses for its own runs, vendored under src/journey-set/vendor/) and never visited.
• Exit codes for a set: 0 when a report printed (complete or not), 2 usage or a refusal before launch, 3 the platform failed with no report, 130 Ctrl-C after printing the view line.
• The key: a Front API key in ORA_API_KEY with journeys:write, journeys:read, runs:read, runs:write, intents:read. The record never holds the key or a token.

The README's ## journey section is the full reference.

One journey command

There were two: ax deep-journey for ora's free public journeys and ax journey for your Front account. Which system runs a journey is now read from what you type:

• No goal: ax journey --domain stripe.com [--intent pricing] [--agent id] [--no-stream] [--json] runs a curated task on ora.ai's public journeys, free, exit codes 0 any verdict, 1 run failed, 2 usage, 3 API error, as before.
• A goal: runs on your Front account with ORA_API_KEY, unchanged. Without the key, the error now also names the free form.
• A goal with a partner key: a goal plus --agent, --intent or --no-stream, or a lone goal when only ORA_PARTNER_API_KEY is set, takes the public keyed tier, which ax deep-journey --task did.
• Both kinds of flag (--intent with --levels, say) is a usage error that names both and gives the call for your Front account.

ax deep-journey <url> still runs, with the same flags (--task included) and exit codes. It prints one line on stderr naming the ax journey call it now stands for, keeps its own error prefix, and is hidden from the help screen; it should be removed in a future major release. Next lines from either name print the ax journey form, and ax audit now ends on Next: ax journey --domain <site>. The README's #deep-journey anchor still lands on the free section.

Also fixed: --no-stream never reached ax deep-journey, because citty parses --no-stream as stream: false; both names now read it that way.

Keys go only where you chose

A set sends your Front API key on every call, so this branch also checks where every key goes, for every command that sends one (ax journey on both paths, ax audit and --tunnel ora, ax webmcp-audit):

• The default hosts always work. Any other host in ORA_API_URL or ORA_PLATFORM_URL gets a key only over https, or over plain http to this machine.
• A host that only a .env in the current directory names is refused before anything is sent, except localhost, 127.0.0.1 and [::1]. That file belongs to whatever repository you run in; to use another host, export the variable in your shell. ax journey exits 2 with KEY_DESTINATION_REFUSED.
• Names are matched the way Windows reads them, so a lowercase ora_platform_url in a .env is caught too.
• Server text in error messages is printed with control characters replaced, so a server cannot move the cursor or retitle your terminal. reportError's display is now a function that receives the cleaned message.
• The key exchange times out after 30 seconds.

The README says this under the env table (9b2e830, 932f671, 7278a72).

Other changes on this branch

• Every command ends on a Next line; --json errors share one envelope that passes the server body through raw; ax learn prints the whole workflow (c6fcb63, c7e68e0, 6aa88a0).
• The free journey renders the run's flags and ora's next action, shows the keyed retry when ora links a docs page, and points a free run at a set for your own goals (b286849, 33a05fb, 7cbeb0b).
• The Studio client types its failures and times out each request (3227090).

Testing

• pnpm lint, pnpm typecheck, pnpm build, pnpm smoke and pnpm verify:pack pass. The full suite passes: 928 tests, 2 skipped (both skips predate this branch). Lint's two warnings and one info are on lines that predate it.
• Smoke checks on the built CLI: ax deep-journey still runs, prints its new name and keeps its prefix and exit code; ax journey with no goal takes the free path and never asks for a Front key.
• Run by hand against a local ora.ai: ax journey --domain and ax deep-journey on an unknown intent (both list the intents, exit 2), a goal with no Front key (names the free form), ax audit, ax skill, ax generate-files, ax journey list, and one journey and a set without a key (both exit 2 with the key steps).
• The key checks, run with the built CLI, fake keys and a local listener: a .env naming a remote host, a *.localhost name or plain http is refused with nothing sent; an exported host and the default host get through; a server answering with escape codes leaves no raw escape bytes on stderr. Not run on Windows; the case rule is covered by tests that stub the platform.
• Not run yet: a real set on the Front platform. No Front API key was available, so the Studio fixtures are built from the API source rather than recorded, and the sentence the report prints under the Need only level (NEED_LEVEL_NOTE) is worded before any real need-level run. The first real set should be run before release.

Reviews

Reviewed for engineering (approved), protocol (approved) and security (approved on the third pass). The first two security passes blocked on where keys go, on server text reaching the terminal, and on a lowercase variable name getting past the check on Windows; all are fixed above.

Open for the reviewer

• AGENTS.md: a set departs from invariants 1, 4, 5, 7 and 9: ax counts verdicts and writes gap lines, prints its own --json document because no single platform payload exists, adds exit code 130, keeps a run ledger (not config) under ~/.ax/journeys/, and adds view|list|cancel under journey, which invariant 9 reserves. Invariant 9 also says new journey commands land as ax deep-journey; this PR goes the other way and folds deep-journey into journey, keeping the old name as an alias. A proposed invariant 16 amending them is ready; AGENTS.md is not edited in this PR.
• ax audit --tunnel ora and its errors still say "platform key" for ORA_API_KEY; the rest now says Front API key.
• Three error strings carried over from earlier PRs contain em dashes (src/api/deep-journey.ts, and the --task partner key and mutually exclusive messages, which ax deep-journey still prints).
• When to remove ax deep-journey: at the next major release. ora.ai never sends it in a response, so only old scripts call it.

Release

Merging lets release-please cut the next minor (0.11.0). ora.ai's docs, MCP page and journey launchpad teach ax journey sets and wait for that npm release before they merge.

[liady]

Thanks for this. The set logic is careful: pure functions, atomic record writes (temp file and rename, 0600), and server text cleaned everywhere it's printed. I ran the changed suites at cea36b5 (36 files, 796 tests pass, tsc --noEmit clean). I also probed the key destination check with localhost.evil.com, localhost., 0.0.0.0, [::ffff:127.0.0.1] and userinfo URLs; all are handled.

Requesting changes for three reasons.

1. It breaks easily against a real platform. The follow loop fails on a single bad read per round, and reads in flight ignore --wait and Ctrl-C (inline). Since no set has run against the real Front platform and the fixtures are hand-built, these are the paths most likely to be hit first. Please run one real set (even 2 goals x 2 agents) before release, and record its responses as fixtures in place of the hand-built ones.

2. Counting rules need real data. Excluding timeout may raise rates exactly on bad sites (inline on outcome.ts). This decides what the matrix tells a customer, so it needs evidence from Studio's source or a real run, not a guess.

3. Size and invariants.

• 16k lines in one PR mix three things that deserve separate reviews. I'd split into:
  • (a) the key destination check and terminal sanitizer: small and security sensitive, so it can merge first;
  • (b) Next: lines, the --json error envelope and ax learn;
  • (c) sets.
• The PR knowingly departs from AGENTS.md invariants 1, 4, 5, 7 and 9 but leaves AGENTS.md unchanged. The next agent working here will read the invariants and "fix" the code back. Please land the invariant 16 amendment in this PR, or drop the departures.
• Smaller: the vendored run-flags copy needs a check that catches drift (inline). There are three em dashes in error strings the PR touches (src/api/deep-journey.ts, the --task partner key message, the mutually exclusive message); please fix them while you're in these files.

A smaller v1 would also cut most of the fixture risk: goals x agents x repeat, without levels and gap lines, run against the real platform. Then add levels once there's data showing the gap lines say something true.

---

Generated by Claude Code

[liady]

Blocking: one bad read in each of three rounds ends the follow for the whole set.

readRound marks the round failed if any single journey() or insightLanded() read fails, and three failed rounds in a row (READ_FAILURE_LIMIT) return read_failed. There is no backoff and no special case for 429 or 404.

Two ways this fires on a healthy set:

• With up to 200 journeys read one after another every 10s, one rate limited or flaky GET per round is enough. After about 30s the set reports read_failed, and every journey that hasn't settled shows as in flight.
• One journey that always 404s does the same, every time. A likely way to get one: a journey recorded from Studio's no-agent 502 (launch.ts:139) whose row the server never kept.

Suggested shape:

• Count consecutive failures per journey (Map<string, number>). End with read_failed only when every journey that hasn't settled has hit the limit.
• On a 404 (StudioApiError with status === 404), stop reading that journey and report it as gone, rather than failing the round.
• On 429 or 5xx, wait before the next round (use Retry-After when present, else double the interval up to a cap).

Please add a test where 1 of 50 journeys 404s forever and the other 49 settle. Today that ends read_failed.

---

Generated by Claude Code

[liady]

--wait and Ctrl-C don't stop a read already in flight. The follow signal never reaches fetch; each request is bounded only by AbortSignal.timeout(REQUEST_TIMEOUT_MS). The wait deadline is checked between rounds, so against a hanging platform one round can run up to 200 x 60s, far past --wait. Ctrl-C can also sit for up to 60s before it detaches.

Engines is >=22.4.0, so AbortSignal.any is available. Let callers pass a signal and combine the two:

signal: options.signal
  ? AbortSignal.any([AbortSignal.timeout(REQUEST_TIMEOUT_MS), options.signal])
  : AbortSignal.timeout(REQUEST_TIMEOUT_MS),

Then thread the follow signal through journey() and insightLanded(), and bound each round by the remaining --wait time, not only the gaps between rounds.

---

Generated by Claude Code

[liady]

"You can close the terminal" prints before any journey is created. Launching up to 200 creates one after another takes a while. A user who takes the line at its word and closes the terminal sends SIGHUP to the process:

• The set is cut short partway through launching.
• markStopped never runs, so the record doesn't say it stopped.
• ax journey cancel from another terminal cancels only what's recorded so far, while the first process may still be creating journeys.

Print this line once launching has finished, from launchAndReport once launch returns. Until then, print something like "Launching N runs; keep this terminal open until they have all started." It would also be worth handling SIGHUP the same way as SIGINT during launch, so the record is marked stopped either way.

---

Generated by Claude Code

[liady]

Product question before this ships: should timeout really be excluded? On a site that is hostile to agents, the most likely way a run ends is that the agent wanders until its time runs out. If Studio stores that as timeout, the set drops exactly the runs that show the site's problem. The matrix then reports a higher rate than reality, and a set can look healthy because the site is bad.

The comment says a harness crash is stored as failed and the reaper writes lost. Those are clearly infrastructure. timeout is ambiguous. Can we confirm from Studio's source which code paths write timeout and timed_out? If any of them is "the agent ran out of time on the site", that path should count as unsatisfied (or at least show as its own column), not be excluded.

Related: any final status not in either list counts as in flight forever, so an unknown value like expired or aborted makes the set wait the full --wait. Treat unknown statuses as excluded with their raw value shown, so a new status from the platform is visible rather than silent.

---

Generated by Claude Code

[liady]

This copy has nothing that catches drift. The header says to re-copy whenever ora's file changes, but nothing enforces it: the tunnel vendor is tested against its upstream, and this one isn't.

It also matters right away: I'm asking for a change to the blocked first-request rule on eralabs-ai/ora#1699 (a single 401 on the first request can raise the flag). If that rule changes there, this copy is already stale on the day it lands.

Options, cheapest first:

1. Record the source file's hash in this header, and add a CI step that fetches src/lib/journey/run-flags.ts from ora main and fails when the hash differs.
2. Publish the rule in a small shared package and depend on it.

Either is fine for this PR; (1) is about 20 lines.

---

Generated by Claude Code

[liady]

CI is red on cea36b5, for two separate reasons.

1. Test (Node 22) / Test (Node 24): two tests in src/journey-set/report-flow.test.ts fail only in CI (lines 155 and 199). The cause: picocolors turns colour on when CI is set, and renderSetReport wraps the incomplete line in pc.yellow(...) (render-report.ts:232). So logged() returns the line with ANSI codes around it, and the exact-string toContain misses. It passes on a dev machine because colour is off there. To reproduce: CI=true npx vitest run src/journey-set/report-flow.test.ts.

Fix: strip the codes in the test helper (verified: 8/8 pass with and without CI=true):

 import { join } from "node:path";
+import { stripVTControlCharacters } from "node:util";
 import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
@@
-const logged = () => vi.mocked(console.log).mock.calls.map((c) => String(c[0]));
+const logged = () =>
+	vi.mocked(console.log).mock.calls.map((c) => stripVTControlCharacters(String(c[0])));

Worth checking whether other suites that assert on rendered lines rely on colour being off too. Setting NO_COLOR=1 in a vitest setup file would make the whole suite behave the same locally and in CI.

2. Contract drift vs production fails because BUILT_AGAINST is 1.42.0 and production still serves 1.41.0. That job stays red until eralabs-ai/ora#1699's contract change ships, and #1699 says to merge only after this PR releases. So the order as written is circular; details are on eralabs-ai/ora#1699. Shipping the ora contract change first breaks the loop.

---

Generated by Claude Code

[liady]

Re-checked at 0205173. None of the earlier review points are addressed yet, and this commit adds scope rather than shrinking it: deep-journey becomes a hidden, deprecated alias, and ax journey now covers two platforms.

State of this head

• Merge conflict with main in src/contract/types.gen.ts and src/contract/version.ts: #94 regenerated the types for 1.41.0. Because of it, Test, Lint, Build and the contract drift job didn't run on this commit; only CodeQL did.
• Same two CI-only failures: I ran the full suite with CI=true on 0205173. 926 pass, and the same 2 tests in report-flow.test.ts fail (the colour codes issue from my earlier comment). tsc --noEmit is clean.

Problems this commit adds

• Platform routing: which platform and key a run uses now depends on which environment variables are set and on a one-letter flag difference (inline).
• Cross-PR drift: eralabs-ai/ora#1699 still teaches ax deep-journey in /docs/cli and /docs/journeys, and its /cli page follows the old split. With this commit those docs teach a command that ax hides from --help and calls deprecated. The two PRs need to describe the same CLI.
• Em dashes: the via === "deep-journey" branches keep the em dash messages flagged earlier, so they will now live on as the deprecated path's text.

I'd hold further feature commits until the split and the blocking items from the first review land. Each new commit makes the split harder.

---

Generated by Claude Code

[liady]

The same command now runs on a different platform depending on which environment variables happen to be set. With no routing flags, ax journey "Find the API docs" --domain x.com:

• runs on the Front account (ORA_API_KEY, Front billing, Front's agents) when ORA_API_KEY is set;
• runs on ora.ai's public partner tier (a different key, a different agent roster, a single public run) when only ORA_PARTNER_API_KEY or ORA_SCAN_API_KEY is set.

The output doesn't say which one ran. A developer whose laptop has one key and whose CI has both gets two different products from the same line, billed to different accounts, and their results won't compare.

On top of that, routing also flips on a single flag, and two of those flags differ by one letter:

• --agent claude sends the run to ora.ai's public tier and needs the partner key;
• --agents claude-code sends it to Front and needs ORA_API_KEY.

A typo there doesn't fail; it quietly switches platform and key.

Suggestion: make the platform explicit rather than inferred, e.g. --on front|ora with no environment fallback, and print the platform on the first line of every run ("Running on your Front account" / "Running on ora.ai public journeys"). Then a missing key becomes a clear usage error rather than a silent reroute.

---

Generated by Claude Code

[liady]

present() re-parses rawArgs by hand next to citty's own parsing. Routing (and so which key gets sent where) now depends on two parsers agreeing about flag names, --no-* negation and -- handling. If citty already gives you the parsed args, deciding from which keys are defined there keeps one parser, one truth. If there's a reason it can't (citty filling in defaults), please add a test per flag that the raw parser and citty agree.

---

Generated by Claude Code