PyArrow: Keep the storage account out of ADLS paths in parse_location

[krishnakaanchan-png]

Rationale for this change

parse_location builds the path as f"{uri.netloc}{uri.path}". That is correct for S3, where the netloc is the bucket. For Azure the netloc is container@account.dfs.core.windows.net, so the account ends up inside the path and PyArrow reads that whole first segment as the container name. Every PyArrowFileIO call on a canonical ABFS location is therefore pointed at a container that cannot exist, since container names allow only lowercase letters, numbers and hyphens.

PyArrow itself handles these locations correctly. FileSystem.from_uri("abfss://myfs@myacct.dfs.core.windows.net/wh/d.parquet") returns myfs/wh/d.parquet, so the problem is only on our side. After this change parse_location returns the same path that PyArrow does.

The full trace and the Azure documentation references are in #2698.

This is the path half only. The other half is that _initialize_azure_fs takes no netloc, so the account can come only from adls.account-name and never from the location itself. I will send that separately, it needs a call on precedence when the property and the location disagree.

Related to #2698. Not closing it here, since the account derivation is still pending.

Are these changes tested?

Yes. Two new parametrised tests over abfs, abfss, wasb and wasbs, one for the account qualified form and one for the container only form.

I also added an S3 case to test_parse_location. That function has to keep serving both shapes and there was nothing pinning the S3 behaviour that the new branch has to preserve.

make lint is clean and tests/io/test_pyarrow.py passes.

Are there any user-facing changes?

Yes. Locations of the form abfs[s]://<container>@<account>.dfs.core.windows.net/<path> now resolve to the correct container under PyArrowFileIO. Earlier they resolved to a container named after the entire netloc. S3, HDFS and the container only Azure form are unchanged.

[Fokko]

Thanks @krishnakaanchan-png for fixing this, as far as my Azure knowledge goes, this looks good to me 👍

[krishnakaanchan-png]

The three required checks here were never reporting. markdown-link-check, python-ci-required and python-integration-required were all sitting at "Expected, waiting for status to be reported" and the commit had no statuses at all, so auto merge had nothing to wait on. The head commit was from 31 Aug and predates those jobs becoming required, so they never ran for it.

I have rebased on main and pushed. All three report now, and markdown-link-check and python-ci-required are green.

python-integration-required is red, but only because integration-test failed while building the hive image:

target hive: failed to solve: process "/bin/sh -c apt-get update -qq &&
  apt-get -qq -y install --no-install-recommends curl ..." did not complete
  successfully: exit code: 100

That is an apt failure in the container build, nothing from the test suite ran. A re-run should clear it.

Could someone re-run integration-test, and the force push has also turned off auto merge so that needs enabling again. Sorry for the churn.

[rambleraptor]

This looks great. Thanks for doing this!

@Fokko could you put this in the merge queue?