Skip to content

feat(io): add ResolvingFileIO to resolve FileIO by location scheme and forward vended credentials#828

Open
plusplusjiajia wants to merge 2 commits into
apache:mainfrom
plusplusjiajia:feat-resolving-fileio
Open

feat(io): add ResolvingFileIO to resolve FileIO by location scheme and forward vended credentials#828
plusplusjiajia wants to merge 2 commits into
apache:mainfrom
plusplusjiajia:feat-resolving-fileio

Conversation

@plusplusjiajia

Copy link
Copy Markdown
Member

Follow-up to #719: replaces the temporary warehouse-based FileIO detection
with a scheme-based ResolvingFileIO, mirroring Java.

Why

The FileIO impl was picked from DetectBuiltinFileIO(warehouse), so any
non-s3:// warehouse resolved to arrow-fs-local and binding vended
credentials failed with Configured FileIO does not support vended storage credentials. Credential-vending catalogs typically use a logical warehouse
identifier (bucket ARN / catalog name), not a storage URI, so the feature
only worked when warehouse was literally s3://….

What

  1. ResolvingFileIO (core) — picks the impl per file-path scheme
    (s3/s3a/s3n/ossarrow-fs-s3; file/no scheme → local; else
    NotSupported), lazily loaded via FileIORegistry and cached; forwards
    the full credential list to impls that support it (Java
    ResolvingFileIO semantics). Registered as resolving-file-io.
  2. REST defaultMakeCatalogFileIO defaults to resolving-file-io
    when io-impl is unset (Java DEFAULT_FILE_IO_IMPL likewise); the
    warehouse detection and the io-impl-vs-warehouse check are removed.
    Explicit io-impl is honored as before.
  3. ArrowS3FileIOSetStorageCredentials skips non-S3 credential
    prefixes instead of rejecting the set (Java S3FileIO filters by "s3");
    servers may vend credentials for several storage systems at once.

@MisterRaindrop

Copy link
Copy Markdown
Contributor

Could we add an end-to-end test using the actual Arrow FileIO implementations? The current tests use mocks and don’t cover the full REST → ResolvingFileIO → FileIORegistry → Arrow FileIO path.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants