Skip to content

Refuse durable namespace URLs that carry credentials - #746

Merged
wudidapaopao merged 1 commit into
mainfrom
fix-durable-url-credentials
Nov 12, 2026
Merged

wudidapaopao merged 1 commit into
mainfrom
fix-durable-url-credentials

Conversation

@auxten

@auxten auxten commented Nov 11, 2026 •

Copy link
Copy Markdown
Member

What leaked, and where

make_backend in chdb/durable/backends/__init__.py used the whole URL authority (urlparse(url).netloc) as the bucket or container name. A namespace URL with credentials in it, such as

s3://AKIAEXAMPLE:wJalrSECRET@my-bucket/durable

therefore produced a bucket named AKIAEXAMPLE:wJalrSECRET@my-bucket. boto3 rejects that name locally, before sending any request, but its ParamValidationError quotes the bucket, so the secret ended up in the traceback and in any log that recorded it. The gcs:// and azure:// branches passed the same value to their clients.

An AWS secret key can contain /. With s3://AKIAEXAMPLE:wJalr/SECRET@my-bucket/durable, the authority ends at AKIAEXAMPLE:wJalr, so the bucket is that string (still leaking the key and the first part of the secret) and the rest of the secret goes into the object prefix.

Severity: low. It only happens when a user writes credentials into the URL. Those credentials were never used to sign anything, so such a URL never worked. The harm is that the secret could end up in error output and logs.

The fix

  • A new refuse_credentials(url) check runs at the start of make_backend, and also in Namespace.__init__ so a bad URL fails when the namespace is constructed instead of on the first open().
  • It refuses any URL whose authority has a username or password, or contains @ or :. Checking : covers the case where a / in the secret moves the @ into the path; neither character can appear in an S3 bucket, GCS bucket or Azure container name.
  • The ValueError quotes no part of the URL. It tells the user to remove the user:secret@ part and use the provider's usual credential chain (AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY or ~/.aws/credentials, Application Default Credentials or CHDB_DURABLE_GCS_TOKEN, CHDB_DURABLE_AZURE_CONN).
  • The URL is refused rather than stripped, because a user who put credentials there thinks they are being used. This matches the Rust binding's fix in Fix an s3 URL credential leak, a false-green test and the docs chdb-rust#54.
  • An @ below the authority (s3://my-bucket/durable/a@b) is still allowed, since it is a legal key character.
  • The bucket is still taken from u.netloc rather than u.hostname. Once the check has passed, netloc is a bare name, and hostname would lowercase it.

I also checked whether anything else in chdb/durable repeats the namespace URL. Nothing does: Namespace has no custom repr and there is no logging of the URL. The other error messages quote only the object id, the key, or the provider's own error, and with this change the provider never receives a bucket name that contains credentials.

Tests

A new case, test_url_carrying_credentials_is_refused_without_repeating_them, in tests/test_durable.py covers:

  • s3://, gcs:// and azure:// URLs with key:secret@, with a secret containing /, and with a username only. Each one goes through both make_backend and Namespace(...). The test checks that no part of the key, the secret or the bucket appears in str() or repr() of the error, that the error has no chained cause, and that no backend was constructed.
  • The same credential URLs with the unsupported gs:// and az:// spellings.
  • Normal s3://, gcs:// and azure:// URLs, plus s3://my-bucket/durable/a@b, still resolve to the expected bucket and prefix.

The provider backend classes are replaced with a stand-in, so the test needs no SDK, no credentials and no network, and runs in the existing durable CI job.

Validation (Linux x86_64, Python 3.12, chdb-core 26.9.0)

  • Reproduced on main without network access. The two s3:// URLs raised boto3's ParamValidationError quoting AKIAEXAMPLE:wJalrSECRET@my-bucket and AKIAEXAMPLE:wJalr. The gcs:// and azure:// URLs were accepted with the credentials as the container name (checked with the backend classes mocked).
  • On this branch, the same script gets the new ValueError for all five URLs, with no part of the credentials in the message. The normal URLs behave as before.
  • The new test fails on main. It also fails if the : check is removed, so it covers the /-in-secret case.
  • pytest tests/test_durable.py: 51 passed. python tests/test_durable.py (script mode): ALL PASS (51 checks).
  • pytest tests/: 108 passed, 6 skipped.
  • cd datastore && pytest tests/: 11546 passed, 4 failed. None of the failures involve code this PR touches:
    • Two test_arrow_interop.py tests (polars and duckdb consuming a DataStore) fail on this machine on main as well.
    • Two timezone tests failed because the fresh venv had no tzdata. After installing it, both files pass (130 passed).

Note

Refuse durable namespace URLs that carry credentials

Adds a refuse_credentials validator in backends/init.py that raises ValueError when the URL authority contains a username, password, @, or :. It runs in both make_backend and Namespace.__init__ before any backend is created.

  • The error message never quotes the URL. It lists credential sources for S3, GCS, and Azure (environment variables or provider credential chains) instead.
  • Adds tests covering credential URLs across the three cloud aliases, including a secret containing /, plus accepted URLs like a path containing @.
  • Behavioral Change: constructing a Namespace or calling make_backend with credentials in the URL authority now raises ValueError immediately.

Macroscope summarized be6849e.

make_backend took the whole URL authority as the bucket, so
s3://KEY:SECRET@bucket/prefix became a bucket named KEY:SECRET@bucket.
boto3 rejected it before sending anything, but its ParamValidationError
quotes the bucket, putting the secret into tracebacks and logs. The gcs
and azure branches built their client from the same value. A secret
containing / split the URL so the @ landed in the path instead.

Such a URL is now refused, both by make_backend and when a Namespace is
constructed, with an error that quotes no part of the URL and points to
the provider credential chain. An authority containing : is refused as
well, which covers the / split. An @ in the key prefix stays allowed.
This matches chdb-io/chdb-rust#54.
@auxten
auxten requested a review from wudidapaopao November 11, 2026 14:32
@wudidapaopao
wudidapaopao merged commit c8bd956 into main Nov 12, 2026
8 of 11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants