Repository navigation
Refuse durable namespace URLs that carry credentials - #746
Merged
Merged
Conversation
make_backend took the whole URL authority as the bucket, so s3://KEY:SECRET@bucket/prefix became a bucket named KEY:SECRET@bucket. boto3 rejected it before sending anything, but its ParamValidationError quotes the bucket, putting the secret into tracebacks and logs. The gcs and azure branches built their client from the same value. A secret containing / split the URL so the @ landed in the path instead. Such a URL is now refused, both by make_backend and when a Namespace is constructed, with an error that quotes no part of the URL and points to the provider credential chain. An authority containing : is refused as well, which covers the / split. An @ in the key prefix stays allowed. This matches chdb-io/chdb-rust#54.
wudidapaopao
approved these changes
Nov 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What leaked, and where
make_backendinchdb/durable/backends/__init__.pyused the whole URL authority (urlparse(url).netloc) as the bucket or container name. A namespace URL with credentials in it, such astherefore produced a bucket named
AKIAEXAMPLE:wJalrSECRET@my-bucket. boto3 rejects that name locally, before sending any request, but itsParamValidationErrorquotes the bucket, so the secret ended up in the traceback and in any log that recorded it. Thegcs://andazure://branches passed the same value to their clients.An AWS secret key can contain
/. Withs3://AKIAEXAMPLE:wJalr/SECRET@my-bucket/durable, the authority ends atAKIAEXAMPLE:wJalr, so the bucket is that string (still leaking the key and the first part of the secret) and the rest of the secret goes into the object prefix.Severity: low. It only happens when a user writes credentials into the URL. Those credentials were never used to sign anything, so such a URL never worked. The harm is that the secret could end up in error output and logs.
The fix
refuse_credentials(url)check runs at the start ofmake_backend, and also inNamespace.__init__so a bad URL fails when the namespace is constructed instead of on the firstopen().@or:. Checking:covers the case where a/in the secret moves the@into the path; neither character can appear in an S3 bucket, GCS bucket or Azure container name.ValueErrorquotes no part of the URL. It tells the user to remove theuser:secret@part and use the provider's usual credential chain (AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEYor~/.aws/credentials, Application Default Credentials orCHDB_DURABLE_GCS_TOKEN,CHDB_DURABLE_AZURE_CONN).@below the authority (s3://my-bucket/durable/a@b) is still allowed, since it is a legal key character.u.netlocrather thanu.hostname. Once the check has passed,netlocis a bare name, andhostnamewould lowercase it.I also checked whether anything else in
chdb/durablerepeats the namespace URL. Nothing does:Namespacehas no customreprand there is no logging of the URL. The other error messages quote only the object id, the key, or the provider's own error, and with this change the provider never receives a bucket name that contains credentials.Tests
A new case,
test_url_carrying_credentials_is_refused_without_repeating_them, intests/test_durable.pycovers:s3://,gcs://andazure://URLs withkey:secret@, with a secret containing/, and with a username only. Each one goes through bothmake_backendandNamespace(...). The test checks that no part of the key, the secret or the bucket appears instr()orrepr()of the error, that the error has no chained cause, and that no backend was constructed.gs://andaz://spellings.s3://,gcs://andazure://URLs, pluss3://my-bucket/durable/a@b, still resolve to the expected bucket and prefix.The provider backend classes are replaced with a stand-in, so the test needs no SDK, no credentials and no network, and runs in the existing durable CI job.
Validation (Linux x86_64, Python 3.12, chdb-core 26.9.0)
mainwithout network access. The twos3://URLs raised boto3'sParamValidationErrorquotingAKIAEXAMPLE:wJalrSECRET@my-bucketandAKIAEXAMPLE:wJalr. Thegcs://andazure://URLs were accepted with the credentials as the container name (checked with the backend classes mocked).ValueErrorfor all five URLs, with no part of the credentials in the message. The normal URLs behave as before.main. It also fails if the:check is removed, so it covers the/-in-secret case.pytest tests/test_durable.py: 51 passed.python tests/test_durable.py(script mode):ALL PASS (51 checks).pytest tests/: 108 passed, 6 skipped.cd datastore && pytest tests/: 11546 passed, 4 failed. None of the failures involve code this PR touches:test_arrow_interop.pytests (polarsandduckdbconsuming a DataStore) fail on this machine onmainas well.tzdata. After installing it, both files pass (130 passed).Note
Refuse durable namespace URLs that carry credentials
Adds a
refuse_credentialsvalidator in backends/init.py that raisesValueErrorwhen the URL authority contains a username, password,@, or:. It runs in bothmake_backendandNamespace.__init__before any backend is created./, plus accepted URLs like a path containing@.Namespaceor callingmake_backendwith credentials in the URL authority now raisesValueErrorimmediately.Macroscope summarized be6849e.