PyArrow: Derive the ADLS account from the location when it is not configured - #3936
Open
krishnakaanchan-png wants to merge 2 commits into
Open
PyArrow: Derive the ADLS account from the location when it is not configured#3936krishnakaanchan-png wants to merge 2 commits into
krishnakaanchan-png wants to merge 2 commits into
Conversation
For Azure the netloc is <container>@<account>.<host>, so building the path as netloc + path put the account inside the path and PyArrow then read the whole first segment as the container name. Return only the container instead, which matches what PyArrow's own from_uri produces for the same location.
…figured _initialize_azure_fs took no netloc, unlike the S3 and HDFS initialisers next to it, so the account could only come from adls.account-name and the account in an abfss location was dropped. Pass the netloc through and fall back to the account in its host part. An explicit adls.account-name still wins, same precedence as FsspecFileIO. The fallback is gated on the userinfo part being present, since a container only netloc carries no account.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2698
Builds on #3884, that one needs to go in first. Only the second commit here is new.
Rationale for this change
This is the other half of #2698. #3884 stops the account leaking into the path. This one lets the account be read from the location at all.
_initialize_azure_fstook no netloc, unlike_initialize_s3_fs(netloc)and_initialize_hdfs_fs(scheme, netloc)next to it. Soaccount_namecould only come fromadls.account-name, and the account in anabfss://<container>@<account>.dfs.core.windows.net/...location was dropped. This leads to two things. A location without the property set failed withArrowInvalid: AzureOptions doesn't contain a valid account name, and one FileIO could not serve two storage accounts.Now the netloc is passed through and the account comes from its host part when the property is not set.
On precedence, an explicit
adls.account-namestill wins over the account in the location. I asked about this on #2698 and did not get a reply, so I have kept it the same asFsspecFileIO. That is the conservative choice, existing single account setups behave exactly as before. If you would rather it raised on a mismatch, tell me and I will change it.The fallback is gated on the userinfo part being present, not on the hostname. On the container only form
abfss://warehouse/f.parquetthe netloc is the container, so inferring an account from it would givewarehouseand break the existing ADLS tests.Taking the account as the first label of the host holds for standard endpoints, private endpoints, sovereign clouds and the DNS zone endpoints. It does not hold for a custom domain on the blob endpoint.
FsspecFileIOhas the same limitation, so I have not tried to solve it here.Are these changes tested?
Yes. I have added 4 new tests. The account coming from the location across all four ADLS schemes, an explicit property winning over the location, a container only netloc giving no account, and one FileIO serving two accounts and getting a separate filesystem for each.
make lintis clean andmake testpasses.Are there any user-facing changes?
Yes.
abfs[s]andwasb[s]locations that carry the account now work underPyArrowFileIOwithoutadls.account-namebeing set, where earlier they failed withAzureOptions doesn't contain a valid account name. A single FileIO can also serve more than one storage account now.Nothing changes when
adls.account-nameis set, or for the container only form.