Skip to content

Export, import, and resume toolpath documents over object storage - #238

Open
akesling wants to merge 2 commits into
mainfrom
alex/object-storage
Open

Export, import, and resume toolpath documents over object storage#238
akesling wants to merge 2 commits into
mainfrom
alex/object-storage

Conversation

@akesling

@akesling akesling commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Adds object storage as a destination for toolpath documents — an S3
bucket, any S3-compatible endpoint (R2, MinIO, Ceph, B2), or a plain
folder.

path p export object --input claude-abc --to s3://my-bucket/traces
path p export object --input claude-abc --to ~/Dropbox/toolpath-traces
path p import object s3://my-bucket/traces/2026-08-07-fix-the-parser-claude-abc.json

path resume s3://my-bucket/traces/2026-08-07-fix-the-parser-claude-abc.json
path resume ~/Dropbox/toolpath-traces   # lists it, pick one

Scope: transport only, deliberately

This is the half of #168 that doesn't collide with anything in flight.
The configuration surface from that PR — a default share target,
path share --to, path target — is not here: it overlaps
ben/share-repo-config's remote model and config.toml, and the ~20
open robert/config-* PRs threading Config through every command.
It'll be refitted onto that model once the Config stack lands.

Nothing here has an opinion about where a setting lives. store.rs
takes a destination as an argument; that's it.

Worth noting crate::remote's own doc comment already anticipates this:
"The URL scheme is the extension point for future backends (an s3://
bucket, say)."
Happy to reshape toward that if you'd rather the two
models converge sooner — @benbarber121 your call.

Credentials come from wherever you already keep them

object_store resolves static keys, EKS/IRSA web identity, ECS task
roles, and EC2 instance metadata — the server cases. It reads no
~/.aws/credentials, no AWS_PROFILE, no SSO, because it deliberately
avoids depending on the AWS SDK. That leaves out how nearly every
developer actually has S3 access on a laptop, and the "fix" would have
been telling them to mint a long-lived IAM key and paste it into a
plaintext file — which most orgs disable by policy.

aws_creds resolves properly:

  • Static-key profiles are parsed straight out of
    ~/.aws/credentials. Trivial ini, no new deps.
  • SSO, role_arn chains, credential_process are delegated to
    aws configure export-credentials --format process, which runs the
    AWS CLI's own resolver. Anyone using SSO already has the CLI — it's
    how they log in — and delegating keeps refresh, cache layout, and
    every future profile type the CLI's problem rather than ours.

Precedence follows AWS's own with our stored settings on top:
path auth s3 login--profile/$AWS_PROFILEAWS_ACCESS_KEY_ID
[default] → object_store's instance chain. Region falls back to the
profile's. path auth s3 status reports which source won, because
that's the first question when an upload fails.

Not aws-config — measured, not assumed: 31 crates, and the whole
aws-* family currently requires rustc 1.94.1 while
rust-toolchain.toml pins 1.94.0. That's an MSRV treadmill on an
exactly-pinned toolchain, not a one-time cost.

path auth s3 login is now documented as the fallback for endpoints AWS
tooling doesn't know about, where a scoped long-lived token is the right
answer. It stores connection settings only — deliberately not a
destination — so one credential serves any number of buckets.

Objects are named to be read

Documents land at <date>-<topic>-<cache-id>.json. Every component is a
pure function of the document, so re-exporting a session that grew
overwrites its own object instead of accumulating near-duplicates; the
date is the session's first step, so it doesn't move as the
conversation continues.

That legibility is what makes browsing cheap: path resume <destination>
lists a bucket, prefix, or folder and builds picker rows from object
names alone — no downloads. A destination holding one document skips the
picker. Anything ending in .json is still fetched directly.

Two traps closed

  • A bare relative destination (my-bucket/traces) is rejected rather
    than quietly creating ./my-bucket/traces and reporting success. It's
    overwhelmingly a bucket name typed from memory; ./my-bucket/traces
    opts in explicitly.
  • memory:// is rejected: a fresh per-process store, so anything
    written there is gone before the command exits.

Testing

scripts/quality_gates.sh 8/8. 464 unit + 31 new integration tests.

The folder backend isn't only a testing affordance, but it's what makes
the tests real: export, import, and resume round-trip through the actual
object_store code path with no network and no mock server. Credential
resolution is tested against an in-memory ~/.aws and a stubbed AWS
CLI, and the integration suite pins AWS_SHARED_CREDENTIALS_FILE /
AWS_CONFIG_FILE at nonexistent paths so a developer's real default
profile can't leak in and start making network calls.

wasm32-unknown-emscripten still builds — the new modules are cfg-gated
like cmd_pathbase.

path-cli 0.19.0, toolpath-cli shim in lockstep.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

…folder

Adds object storage as a destination for toolpath documents, at the
plumbing layer:

    path p export object --input claude-abc --to s3://my-bucket/traces
    path p export object --input claude-abc --to ~/Dropbox/traces
    path p import object s3://my-bucket/traces/<name>.json
    path resume s3://my-bucket/traces/<name>.json
    path resume ~/Dropbox/traces          # lists it, pick one

Transport is the `object_store` crate, so one code path covers AWS S3,
any S3-compatible endpoint (R2, MinIO, Ceph, B2), and `file://` for a
plain folder. A folder is a first-class destination, not a testing
affordance — it needs no credentials at all — and it's what the tests
round-trip against, so export, import, and resume are covered end to end
with no network and no mock HTTP server.

Credentials come from wherever you already keep them. `object_store`
resolves only the server cases (EKS/IRSA, ECS, EC2 instance metadata);
it reads no `~/.aws` because it avoids the AWS SDK, which leaves out how
nearly every developer actually has S3 access. `aws_creds` fills that
in: static-key profiles are parsed from `~/.aws/credentials`, and SSO,
`role_arn` chains, and `credential_process` are delegated to `aws
configure export-credentials` — the AWS CLI's own resolver, so refresh
and future profile types stay its problem. Not `aws-config`: 31 crates,
and its family requires rustc 1.94.1 against a repo pinned to 1.94.0.

`path auth s3 login` is the fallback for endpoints AWS tooling doesn't
know about. It stores connection settings only — deliberately not a
destination, so one credential serves any number of buckets. `path auth
s3 status` reports which credential source actually won.

Objects are named `<date>-<topic>-<cache-id>.json`. Every component is a
pure function of the document, so re-exporting a session that grew
overwrites its own object instead of accumulating near-duplicates; the
date is the session's first step, so it doesn't move as the conversation
continues. That legibility is what makes `path resume <destination>`
cheap — it lists a bucket or folder and builds picker rows from names
alone, no downloads.

A scheme-less destination is a local path; a bucket is spelled `s3://`.
A bare relative value is rejected rather than quietly creating
`./my-bucket/traces` and reporting success. `memory://` is rejected: a
fresh per-process store, so anything written there is gone before the
command exits.

Split out of the configuration work (a default share target, `path
share --to`) deliberately: that surface overlaps main's `remote` model
and `config.toml`, and the ~20 in-flight `Config` PRs. This half has no
opinion about where the setting lives.
An expired SSO session is the one credential failure with an obvious
next step. Reporting it and stopping made the user go run the command
themselves and start over, for no reason.

`aws_creds` now detects it, offers to run `aws sso login --profile
<name>`, and retries once. Offer, not do: that command opens a browser
and waits, and that shouldn't happen because someone typed `path
resume`. With no terminal to ask — CI — it fails with the exact command
rather than blocking on a prompt nobody will answer.

Exactly one retry. If a fresh login still yields nothing, looping won't
help and the real error is whatever comes back the second time.

Detection matches the CLI's message, since it reports this as a plain
non-zero exit and the wording varies by version ("Token ... does not
exist", "has expired", "refresh failed", or a direct instruction to run
`aws sso login`). Requiring `sso` alongside keeps unrelated failures —
a denied AssumeRole, an unreachable endpoint — out of the offer; that's
covered by a test asserting both directions.

The login and the prompt sit behind `Env` seams alongside `aws_cli`, so
the policy (detect, ask, log in, retry once) is testable without a real
AWS CLI or a terminal.
@akesling

Copy link
Copy Markdown
Contributor Author

Added: when an SSO session has expired, path offers to run aws sso login --profile <name> and retries once, instead of reporting the failure and making you start the command over.

Offer, not do — that command opens a browser and waits, which shouldn't fire because someone typed path resume. With no terminal to ask (CI), it fails with the exact command rather than blocking on a prompt nobody will answer. Exactly one retry: if a fresh login still yields nothing, looping won't help.

Detection matches the CLI's message, since it reports this as a plain non-zero exit and the wording varies by version. Requiring sso alongside keeps unrelated failures (a denied AssumeRole, an unreachable endpoint) out of the offer — tested in both directions.

The login and the prompt sit behind Env seams alongside aws_cli, so the policy is testable without a real AWS CLI or a terminal.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant