Export, import, and resume toolpath documents over object storage - #238
Export, import, and resume toolpath documents over object storage#238akesling wants to merge 2 commits into
Conversation
…folder
Adds object storage as a destination for toolpath documents, at the
plumbing layer:
path p export object --input claude-abc --to s3://my-bucket/traces
path p export object --input claude-abc --to ~/Dropbox/traces
path p import object s3://my-bucket/traces/<name>.json
path resume s3://my-bucket/traces/<name>.json
path resume ~/Dropbox/traces # lists it, pick one
Transport is the `object_store` crate, so one code path covers AWS S3,
any S3-compatible endpoint (R2, MinIO, Ceph, B2), and `file://` for a
plain folder. A folder is a first-class destination, not a testing
affordance — it needs no credentials at all — and it's what the tests
round-trip against, so export, import, and resume are covered end to end
with no network and no mock HTTP server.
Credentials come from wherever you already keep them. `object_store`
resolves only the server cases (EKS/IRSA, ECS, EC2 instance metadata);
it reads no `~/.aws` because it avoids the AWS SDK, which leaves out how
nearly every developer actually has S3 access. `aws_creds` fills that
in: static-key profiles are parsed from `~/.aws/credentials`, and SSO,
`role_arn` chains, and `credential_process` are delegated to `aws
configure export-credentials` — the AWS CLI's own resolver, so refresh
and future profile types stay its problem. Not `aws-config`: 31 crates,
and its family requires rustc 1.94.1 against a repo pinned to 1.94.0.
`path auth s3 login` is the fallback for endpoints AWS tooling doesn't
know about. It stores connection settings only — deliberately not a
destination, so one credential serves any number of buckets. `path auth
s3 status` reports which credential source actually won.
Objects are named `<date>-<topic>-<cache-id>.json`. Every component is a
pure function of the document, so re-exporting a session that grew
overwrites its own object instead of accumulating near-duplicates; the
date is the session's first step, so it doesn't move as the conversation
continues. That legibility is what makes `path resume <destination>`
cheap — it lists a bucket or folder and builds picker rows from names
alone, no downloads.
A scheme-less destination is a local path; a bucket is spelled `s3://`.
A bare relative value is rejected rather than quietly creating
`./my-bucket/traces` and reporting success. `memory://` is rejected: a
fresh per-process store, so anything written there is gone before the
command exits.
Split out of the configuration work (a default share target, `path
share --to`) deliberately: that surface overlaps main's `remote` model
and `config.toml`, and the ~20 in-flight `Config` PRs. This half has no
opinion about where the setting lives.
An expired SSO session is the one credential failure with an obvious
next step. Reporting it and stopping made the user go run the command
themselves and start over, for no reason.
`aws_creds` now detects it, offers to run `aws sso login --profile
<name>`, and retries once. Offer, not do: that command opens a browser
and waits, and that shouldn't happen because someone typed `path
resume`. With no terminal to ask — CI — it fails with the exact command
rather than blocking on a prompt nobody will answer.
Exactly one retry. If a fresh login still yields nothing, looping won't
help and the real error is whatever comes back the second time.
Detection matches the CLI's message, since it reports this as a plain
non-zero exit and the wording varies by version ("Token ... does not
exist", "has expired", "refresh failed", or a direct instruction to run
`aws sso login`). Requiring `sso` alongside keeps unrelated failures —
a denied AssumeRole, an unreachable endpoint — out of the offer; that's
covered by a test asserting both directions.
The login and the prompt sit behind `Env` seams alongside `aws_cli`, so
the policy (detect, ask, log in, retry once) is testable without a real
AWS CLI or a terminal.
|
Added: when an SSO session has expired, Offer, not do — that command opens a browser and waits, which shouldn't fire because someone typed Detection matches the CLI's message, since it reports this as a plain non-zero exit and the wording varies by version. Requiring The login and the prompt sit behind |
Adds object storage as a destination for toolpath documents — an S3
bucket, any S3-compatible endpoint (R2, MinIO, Ceph, B2), or a plain
folder.
Scope: transport only, deliberately
This is the half of #168 that doesn't collide with anything in flight.
The configuration surface from that PR — a default share target,
path share --to,path target— is not here: it overlapsben/share-repo-config'sremotemodel andconfig.toml, and the ~20open
robert/config-*PRs threadingConfigthrough every command.It'll be refitted onto that model once the
Configstack lands.Nothing here has an opinion about where a setting lives.
store.rstakes a destination as an argument; that's it.
Worth noting
crate::remote's own doc comment already anticipates this:"The URL scheme is the extension point for future backends (an
s3://bucket, say)." Happy to reshape toward that if you'd rather the two
models converge sooner — @benbarber121 your call.
Credentials come from wherever you already keep them
object_storeresolves static keys, EKS/IRSA web identity, ECS taskroles, and EC2 instance metadata — the server cases. It reads no
~/.aws/credentials, noAWS_PROFILE, no SSO, because it deliberatelyavoids depending on the AWS SDK. That leaves out how nearly every
developer actually has S3 access on a laptop, and the "fix" would have
been telling them to mint a long-lived IAM key and paste it into a
plaintext file — which most orgs disable by policy.
aws_credsresolves properly:~/.aws/credentials. Trivial ini, no new deps.role_arnchains,credential_processare delegated toaws configure export-credentials --format process, which runs theAWS CLI's own resolver. Anyone using SSO already has the CLI — it's
how they log in — and delegating keeps refresh, cache layout, and
every future profile type the CLI's problem rather than ours.
Precedence follows AWS's own with our stored settings on top:
path auth s3 login→--profile/$AWS_PROFILE→AWS_ACCESS_KEY_ID→
[default]→ object_store's instance chain. Region falls back to theprofile's.
path auth s3 statusreports which source won, becausethat's the first question when an upload fails.
Not
aws-config— measured, not assumed: 31 crates, and the wholeaws-*family currently requires rustc 1.94.1 whilerust-toolchain.tomlpins 1.94.0. That's an MSRV treadmill on anexactly-pinned toolchain, not a one-time cost.
path auth s3 loginis now documented as the fallback for endpoints AWStooling doesn't know about, where a scoped long-lived token is the right
answer. It stores connection settings only — deliberately not a
destination — so one credential serves any number of buckets.
Objects are named to be read
Documents land at
<date>-<topic>-<cache-id>.json. Every component is apure function of the document, so re-exporting a session that grew
overwrites its own object instead of accumulating near-duplicates; the
date is the session's first step, so it doesn't move as the
conversation continues.
That legibility is what makes browsing cheap:
path resume <destination>lists a bucket, prefix, or folder and builds picker rows from object
names alone — no downloads. A destination holding one document skips the
picker. Anything ending in
.jsonis still fetched directly.Two traps closed
my-bucket/traces) is rejected ratherthan quietly creating
./my-bucket/tracesand reporting success. It'soverwhelmingly a bucket name typed from memory;
./my-bucket/tracesopts in explicitly.
memory://is rejected: a fresh per-process store, so anythingwritten there is gone before the command exits.
Testing
scripts/quality_gates.sh8/8. 464 unit + 31 new integration tests.The folder backend isn't only a testing affordance, but it's what makes
the tests real: export, import, and resume round-trip through the actual
object_storecode path with no network and no mock server. Credentialresolution is tested against an in-memory
~/.awsand a stubbed AWSCLI, and the integration suite pins
AWS_SHARED_CREDENTIALS_FILE/AWS_CONFIG_FILEat nonexistent paths so a developer's real defaultprofile can't leak in and start making network calls.
wasm32-unknown-emscriptenstill builds — the new modules are cfg-gatedlike
cmd_pathbase.path-cli0.19.0,toolpath-clishim in lockstep.Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.