Computer-use for legacy back-office apps

Record a task once with a model, save it as a typed capability, then replay it without the model. Built as a take-home slice of the integration layer that would sit behind bank/credit-union agents when the core has no API.

The live target is Heritage Core, a local teller console: frameset layout, nested tables, no test ids. Observation and targeting go through the accessibility tree, not CSS.

Setup

Python 3.11+. From the repo root:

python -m venv .venv
.\.venv\Scripts\activate
pip install -e ".[dev]"
python -m playwright install chromium

On macOS/Linux, activate with source .venv/bin/activate.

Copy .env.example to .env. Discovery needs OPENAI_API_KEY. Replay, tests, and the teller app do not.

OPENAI_API_KEY=sk-...
CUSE_MODEL=gpt-4o-mini

Demo operator / PIN (OP4421 / demo) are only used if you sign on by hand. Automated runs attach an already-authenticated session via /dev/attach so credentials never land in artifacts or logs.

Demo path

The teller app is started automatically by these commands. You can also run it alone with cuse serve (http://127.0.0.1:8787).

1. Discover (LLM in the loop)

cuse discover --goal "look up member 10001 and read their current savings balance" --param member_id=10001 --out capabilities/lookup_member_savings.json --no-operator

Writes capabilities/lookup_member_savings.json (or --out path) and a run folder under evidence/.

Checked-in evidence from live Heritage Core sessions:

• evidence/lookup_member_savings.json — the capability artifact
• evidence/replay-success/ — deterministic replay, member_id=10001, outputs savings_balance=4250.18
• evidence/replay-not-found/ — same artifact, member_id=99999, business_outcome
• evidence/handoff/ — live session paused, controller flipped to human, resumed on the same Chromium page

Discovery logs land next to those after you run step 1 with a key (evidence/discover-*). Replay does not need a key.

2. Replay (no LLM)

cuse replay --artifact capabilities/lookup_member_savings.json --param member_id=10001

3. Replay a business outcome, not a crash

cuse replay --artifact capabilities/lookup_member_savings.json --param member_id=99999

Expect status: business_outcome, outcome_id: member_not_found. Same for --param member_id=12 (invalid_member_id).

4. Call it like an agent tool

cuse invoke lookup_member_savings --param member_id=10001

5. Human takes the live session

cuse handoff-demo --headed

Automation pauses on the same Chromium window, opens an operator desk at http://127.0.0.1:8788, and waits. Use that window (not a new browser), then Resume automation. To skip the wait in CI:

cuse handoff-demo --auto-resume-s 2

--headed is useful when you want to watch; default is headless.

Without live services

• cuse replay and cuse invoke only need the local teller app (auto-started) and Chromium. No model key.
• pytest covers schema, allowlist/redaction, handoff control transfer, and the three replay outcomes against Heritage Core.
• Discovery is the only path that calls OpenAI.

Layout

path	what
apps/heritage_core/	local teller console (proxy for a core/servicing screen)
src/cuse/	discover, artifact schema, replay, policy, handoff
capabilities/	saved artifacts
policy.yaml	allowlist, recoverable interstitials, hard failures, redaction
evidence/	run logs, a11y dumps, failure screenshots
REPORT.md	design write-up

Policy

policy.yaml is the guardrail file: allowed hosts/actions, irreversible control names (Confirm Open, …), recoveries (maintenance notice, please-wait), and hard stops (session expired, ABEND). Discovery and replay both go through it.