Skip to content

Fix an s3 URL credential leak, a false-green test and the docs - #54

Merged
wudidapaopao merged 4 commits into
mainfrom
fix/doc-and-s3-url-hardening
Nov 16, 2026
Merged

wudidapaopao merged 4 commits into
mainfrom
fix/doc-and-s3-url-hardening

Conversation

@ShawnChen-Sirius

@ShawnChen-Sirius ShawnChen-Sirius commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

Three independent fixes, rebased onto main (v2.1.0). The first three commits are Shawn's original ones; the fourth is a follow-up made during the rebase.

1. Credentials in an s3:// namespace URL leaked (low severity)

s3_location took everything before the first / as the bucket, so s3://KEY:SECRET@bucket/prefix was accepted with a bucket literally named KEY:SECRET@bucket. The secret then showed up in Namespace::location(), the Debug output of Namespace and S3Backend, Backend::describe() (documented as never containing credentials), every error message built from it, and the outgoing request (the hostname on AWS, the path with MinIO or a custom endpoint).

Severity: low. It only happens with the optional durable-s3 feature and only when a user puts credentials in the URL. Those credentials were never used to sign anything, so the URL never worked. The harm is that the secret ends up in logs, error reports and the request line.

Fix: such a URL is now refused with an error that quotes neither the URL nor the credentials. It is refused rather than stripped, because a caller who wrote credentials into the URL believes they are being used. An @ inside the key prefix (s3://bucket/a@b) is still allowed.

The follow-up commit closes two gaps:

  • An AWS secret key can contain /. With s3://KEY:abc/def@bucket/p the bucket ends at KEY:abc, before the @, so checking for @ alone accepted it. The check now refuses a bucket that contains @ or :, neither of which can appear in a bucket name.
  • A build without durable-s3 quoted the whole s3:// URL, credentials included, in its "needs the durable-s3 feature" error. It no longer quotes the URL.

2. S3 tests passed when no bucket was configured

Without CHDB_DURABLE_S3_BUCKET, the eight real-bucket tests returned early and were reported as 8 passed. They are now #[ignore]d, so a normal run reports 8 ignored, and --ignored opts in. CI's S3 step now filters on durable:: instead of durable::backends, so the new URL test in durable::namespace actually runs there.

3. Docs

  • The Arrow "High-level helpers" snippet in docs/examples.md did not compile (wrong argument count, and an Arrow error that ? cannot convert). It is fixed, and the insert_record_batches / insert_record_batch_reader description is corrected.
  • durable::Backend is described as having seven methods, which is what the trait declares, instead of six.
  • README.md documents that a URL carrying credentials is refused and how to run the bucket tests.

The original PR also bumped the version in the dependency lines to 2.0.0. main already says 2.1, so those edits were dropped during the rebase.

Verification (Linux x86_64, engine v26.11.0-rc.2, rustc 1.99.1)

Before the fix, on main:

  • A throwaway unit test printed the secret in location(), Namespace Debug, describe(), S3Backend Debug and a request error ("GET head.json failed against s3://AKIAEXAMPLE:wJalrXUtnFEMIsecret@my-bucket/..."). The same happened with a secret containing /.
  • cargo test --features durable-s3 --test durable_s3 with no bucket: 8 passed.
  • The docs snippet, extracted into an example, failed to compile with E0061 and E0277.

After the fix, on this branch:

  • The same throwaway test: Namespace::new refuses both URLs and the error contains neither the key nor the secret.
  • The new unit test fails if the check is put back to @ only, and passes with this branch.
  • durable_s3 with no bucket: 0 passed; 8 ignored.
  • The corrected snippet compiles.
  • cargo test -- --test-threads=1 with default features, --features durable and --features durable-s3: all pass, including doc tests (64 with default features), and 16/16 runtime cases.
  • CI's S3 command cargo test --features durable-s3 --lib -- --test-threads=1 durable::: 74 passed, including the new test.
  • cargo fmt --check passes, and cargo clippy --all-targets -- --deny warnings is clean with default features and with --features durable-s3.

Not run: the bucket tests against a real bucket (--ignored), and the static-link jobs (left to CI).

Note

Fix S3 URL credential leak in Namespace.s3 and ignore real-bucket tests

  • durable::namespace.s3_location now rejects S3 namespace URLs whose parsed bucket component contains @ or :. The error no longer repeats the URL, so credentials never appear in error messages.
  • The feature-disabled Namespace.s3 handler also returns a generic Backend error instead of formatting the supplied URL into it.
  • All real-bucket integration tests in durable_s3.rs are now #[ignore]d by default and run with --ignored. The CI step in rust.yml was widened to run all durable:: library tests, which previously showed a false green by matching nothing.
  • Docs updated: README.md and docs/examples.md document credential refusal, the new durable::Backend method count of seven, and corrected Arrow ingestion examples using insert_record_batch with InsertOptions::default_bulk().
  • Behavioral Change: S3 namespace URLs with credentials in the authority (e.g. key:secret@bucket) now fail with a Backend error; an @ in the key prefix remains accepted.

Macroscope summarized 91e0c4f.

@CLAassistant

CLAassistant commented Sep 20, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@ShawnChen-Sirius
ShawnChen-Sirius force-pushed the fix/doc-and-s3-url-hardening branch from b867353 to 74205e5 Compare September 20, 2026 23:09
@ShawnChen-Sirius

Copy link
Copy Markdown
Contributor Author

@wudidapaopao, please reivew this PR

ShawnChen-Sirius and others added 4 commits November 11, 2026 13:33
The `High-level helpers` snippet in docs/examples.md has never compiled. It
was added in 40e01c7 calling `insert_record_batch` with three arguments and a
`&RecordBatch`, against a signature that already took four and the batch by
value, so it has shipped broken in v1.4.0 and v2.0.0. Neither README.md nor
docs/examples.md is wired into doctests, which is why 68 doctests pass while
the two files readers are pointed at first do not.

Also corrected, all found the same way:

- `insert_record_batches` was described as taking a `RecordBatchReader`. It
  takes a `Vec<RecordBatch>` and a `SchemaRef`; `insert_record_batch_reader`
  is the one that takes a reader.
- `durable::Backend` was described as six methods. It declares seven, none of
  them defaulted.
- The dependency line said `1.1.0` in README.md and `1.0.0` in
  docs/examples.md, two and three releases behind the crate.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`require_bucket!` printed a reason and returned, which libtest reports as a
pass — and it swallows the stderr of a passing test, so the reason was not
printed either. Without a bucket the suite summarised as `8 passed` in 0.00s,
and the run that found this read it as eight green tests.

That is the wrong failure mode here more than anywhere else: these are the
tests the local backend cannot stand in for, because only a real provider
settles whether `If-None-Match: *` and `If-Match: <etag>` behave the way the
protocol assumes, and the single-writer guarantee rests on that.

`#[ignore]` makes the same run report `8 ignored`, which is what happened.
`--ignored` opts in, so the invocation in the module docs and in README.md
gains it. `require_bucket!` stays, now covering only an `--ignored` run on a
machine that turns out to have no bucket.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`s3_location` split the authority on the first `/` and took everything before
it as the bucket, so `s3://key:secret@bucket/durable` parsed as a bucket named
`key:secret@bucket`. Nothing rejected it and nothing stripped it, and the
secret then travelled everywhere the location does: `Namespace::location`,
the `Debug` impl, `S3Backend::describe` — which `Backend::describe` documents
as never containing credentials — every error built from it, and the request
that goes to the provider.

An `@` cannot appear in a bucket name, so this is unambiguous. Refused rather
than stripped, because a caller who put credentials in the URL believes they
are being used, and reading different ones from the environment instead would
authenticate as somebody they did not choose. The message quotes neither the
URL nor the userinfo: refusing while echoing it back would put the secret in
the log it is meant to stay out of. An `@` below the authority is a legal S3
key character and stays one.

The CI filter moves from `durable::backends` to `durable::`, because the URL
parsing those tests rest on lives in `durable::namespace` and the new test
would otherwise never run.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…s3 URLs

A secret key may contain `/`, which ends the parsed bucket at
`user:sec` before the `@` is reached, so checking for `@` alone let
such a URL through with part of the secret in the bucket and the rest in
the prefix. Refuse a bucket containing `:` as well, which no bucket name
can contain either.

A build without `durable-s3` quoted the whole URL in its "needs the
feature" error, credentials included; it no longer quotes it.
@auxten
auxten force-pushed the fix/doc-and-s3-url-hardening branch from 74205e5 to 91e0c4f Compare November 11, 2026 13:39
@auxten
auxten requested a review from wudidapaopao November 11, 2026 13:39
@auxten

auxten commented Nov 11, 2026

Copy link
Copy Markdown
Member

Rebased onto main (v2.1.0). The only conflict was the version line in README.md and docs/examples.md: this PR bumped it to 2.0.0, but main already says 2.1, so I kept main's line and dropped that edit. The code, test and CI changes are unchanged.

I added one commit on top: the credential check now also refuses : in the bucket, because a secret key containing / (for example s3://KEY:abc/def@bucket) ends the bucket before the @ and got past the @-only check. The build without durable-s3 also no longer echoes the s3 URL in its error. Details and the verification I ran are in the description.

@ShawnChen-Sirius

Copy link
Copy Markdown
Contributor Author

@wudidapaopao, please review this PR

@wudidapaopao wudidapaopao left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@wudidapaopao
wudidapaopao merged commit ece0e32 into main Nov 16, 2026
11 checks passed
mwaites pushed a commit to mwaites/chdb that referenced this pull request Nov 20, 2026
make_backend took the whole URL authority as the bucket, so
s3://KEY:SECRET@bucket/prefix became a bucket named KEY:SECRET@bucket.
boto3 rejected it before sending anything, but its ParamValidationError
quotes the bucket, putting the secret into tracebacks and logs. The gcs
and azure branches built their client from the same value. A secret
containing / split the URL so the @ landed in the path instead.

Such a URL is now refused, both by make_backend and when a Namespace is
constructed, with an error that quotes no part of the URL and points to
the provider credential chain. An authority containing : is refused as
well, which covers the / split. An @ in the key prefix stays allowed.
This matches chdb-io/chdb-rust#54.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants