Repository navigation
Fix an s3 URL credential leak, a false-green test and the docs - #54
Merged
Merged
Conversation
ShawnChen-Sirius
force-pushed
the
fix/doc-and-s3-url-hardening
branch
from
September 20, 2026 23:09
b867353 to
74205e5
Compare
Contributor
Author
|
@wudidapaopao, please reivew this PR |
The `High-level helpers` snippet in docs/examples.md has never compiled. It was added in 40e01c7 calling `insert_record_batch` with three arguments and a `&RecordBatch`, against a signature that already took four and the batch by value, so it has shipped broken in v1.4.0 and v2.0.0. Neither README.md nor docs/examples.md is wired into doctests, which is why 68 doctests pass while the two files readers are pointed at first do not. Also corrected, all found the same way: - `insert_record_batches` was described as taking a `RecordBatchReader`. It takes a `Vec<RecordBatch>` and a `SchemaRef`; `insert_record_batch_reader` is the one that takes a reader. - `durable::Backend` was described as six methods. It declares seven, none of them defaulted. - The dependency line said `1.1.0` in README.md and `1.0.0` in docs/examples.md, two and three releases behind the crate. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`require_bucket!` printed a reason and returned, which libtest reports as a pass — and it swallows the stderr of a passing test, so the reason was not printed either. Without a bucket the suite summarised as `8 passed` in 0.00s, and the run that found this read it as eight green tests. That is the wrong failure mode here more than anywhere else: these are the tests the local backend cannot stand in for, because only a real provider settles whether `If-None-Match: *` and `If-Match: <etag>` behave the way the protocol assumes, and the single-writer guarantee rests on that. `#[ignore]` makes the same run report `8 ignored`, which is what happened. `--ignored` opts in, so the invocation in the module docs and in README.md gains it. `require_bucket!` stays, now covering only an `--ignored` run on a machine that turns out to have no bucket. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`s3_location` split the authority on the first `/` and took everything before it as the bucket, so `s3://key:secret@bucket/durable` parsed as a bucket named `key:secret@bucket`. Nothing rejected it and nothing stripped it, and the secret then travelled everywhere the location does: `Namespace::location`, the `Debug` impl, `S3Backend::describe` — which `Backend::describe` documents as never containing credentials — every error built from it, and the request that goes to the provider. An `@` cannot appear in a bucket name, so this is unambiguous. Refused rather than stripped, because a caller who put credentials in the URL believes they are being used, and reading different ones from the environment instead would authenticate as somebody they did not choose. The message quotes neither the URL nor the userinfo: refusing while echoing it back would put the secret in the log it is meant to stay out of. An `@` below the authority is a legal S3 key character and stays one. The CI filter moves from `durable::backends` to `durable::`, because the URL parsing those tests rest on lives in `durable::namespace` and the new test would otherwise never run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…s3 URLs A secret key may contain `/`, which ends the parsed bucket at `user:sec` before the `@` is reached, so checking for `@` alone let such a URL through with part of the secret in the bucket and the rest in the prefix. Refuse a bucket containing `:` as well, which no bucket name can contain either. A build without `durable-s3` quoted the whole URL in its "needs the feature" error, credentials included; it no longer quotes it.
auxten
force-pushed
the
fix/doc-and-s3-url-hardening
branch
from
November 11, 2026 13:39
74205e5 to
91e0c4f
Compare
Member
|
Rebased onto main (v2.1.0). The only conflict was the version line in I added one commit on top: the credential check now also refuses |
This was referenced Nov 11, 2026
Contributor
Author
|
@wudidapaopao, please review this PR |
mwaites
pushed a commit
to mwaites/chdb
that referenced
this pull request
Nov 20, 2026
make_backend took the whole URL authority as the bucket, so s3://KEY:SECRET@bucket/prefix became a bucket named KEY:SECRET@bucket. boto3 rejected it before sending anything, but its ParamValidationError quotes the bucket, putting the secret into tracebacks and logs. The gcs and azure branches built their client from the same value. A secret containing / split the URL so the @ landed in the path instead. Such a URL is now refused, both by make_backend and when a Namespace is constructed, with an error that quotes no part of the URL and points to the provider credential chain. An authority containing : is refused as well, which covers the / split. An @ in the key prefix stays allowed. This matches chdb-io/chdb-rust#54.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three independent fixes, rebased onto
main(v2.1.0). The first three commits are Shawn's original ones; the fourth is a follow-up made during the rebase.1. Credentials in an
s3://namespace URL leaked (low severity)s3_locationtook everything before the first/as the bucket, sos3://KEY:SECRET@bucket/prefixwas accepted with a bucket literally namedKEY:SECRET@bucket. The secret then showed up inNamespace::location(), theDebugoutput ofNamespaceandS3Backend,Backend::describe()(documented as never containing credentials), every error message built from it, and the outgoing request (the hostname on AWS, the path with MinIO or a custom endpoint).Severity: low. It only happens with the optional
durable-s3feature and only when a user puts credentials in the URL. Those credentials were never used to sign anything, so the URL never worked. The harm is that the secret ends up in logs, error reports and the request line.Fix: such a URL is now refused with an error that quotes neither the URL nor the credentials. It is refused rather than stripped, because a caller who wrote credentials into the URL believes they are being used. An
@inside the key prefix (s3://bucket/a@b) is still allowed.The follow-up commit closes two gaps:
/. Withs3://KEY:abc/def@bucket/pthe bucket ends atKEY:abc, before the@, so checking for@alone accepted it. The check now refuses a bucket that contains@or:, neither of which can appear in a bucket name.durable-s3quoted the wholes3://URL, credentials included, in its "needs thedurable-s3feature" error. It no longer quotes the URL.2. S3 tests passed when no bucket was configured
Without
CHDB_DURABLE_S3_BUCKET, the eight real-bucket tests returned early and were reported as8 passed. They are now#[ignore]d, so a normal run reports8 ignored, and--ignoredopts in. CI's S3 step now filters ondurable::instead ofdurable::backends, so the new URL test indurable::namespaceactually runs there.3. Docs
docs/examples.mddid not compile (wrong argument count, and an Arrow error that?cannot convert). It is fixed, and theinsert_record_batches/insert_record_batch_readerdescription is corrected.durable::Backendis described as having seven methods, which is what the trait declares, instead of six.README.mddocuments that a URL carrying credentials is refused and how to run the bucket tests.The original PR also bumped the version in the dependency lines to
2.0.0.mainalready says2.1, so those edits were dropped during the rebase.Verification (Linux x86_64, engine v26.11.0-rc.2, rustc 1.99.1)
Before the fix, on
main:location(),NamespaceDebug,describe(),S3BackendDebugand a request error ("GET head.json failed against s3://AKIAEXAMPLE:wJalrXUtnFEMIsecret@my-bucket/..."). The same happened with a secret containing/.cargo test --features durable-s3 --test durable_s3with no bucket:8 passed.After the fix, on this branch:
Namespace::newrefuses both URLs and the error contains neither the key nor the secret.@only, and passes with this branch.durable_s3with no bucket:0 passed; 8 ignored.cargo test -- --test-threads=1with default features,--features durableand--features durable-s3: all pass, including doc tests (64 with default features), and 16/16 runtime cases.cargo test --features durable-s3 --lib -- --test-threads=1 durable::: 74 passed, including the new test.cargo fmt --checkpasses, andcargo clippy --all-targets -- --deny warningsis clean with default features and with--features durable-s3.Not run: the bucket tests against a real bucket (
--ignored), and the static-link jobs (left to CI).Note
Fix S3 URL credential leak in
Namespace.s3and ignore real-bucket testsdurable::namespace.s3_locationnow rejects S3 namespace URLs whose parsed bucket component contains@or:. The error no longer repeats the URL, so credentials never appear in error messages.Namespace.s3handler also returns a genericBackenderror instead of formatting the supplied URL into it.#[ignore]d by default and run with--ignored. The CI step in rust.yml was widened to run alldurable::library tests, which previously showed a false green by matching nothing.durable::Backendmethod count of seven, and corrected Arrow ingestion examples usinginsert_record_batchwithInsertOptions::default_bulk().key:secret@bucket) now fail with aBackenderror; an@in the key prefix remains accepted.Macroscope summarized 91e0c4f.