feat(format): add generic metadata API by lidavidm · Pull Request #4694 · apache/arrow-adbc

lidavidm · 2026-08-17T06:05:04Z

Add a generic API for fetching metadata, modeled on ADO.NET's GetSchema
Define the standard metadata collections
Revert the current revisions to GetObjects in favor of not changing it

format: provide way to escape wildcards in GetObjects and other functions #3997: define a way to escape search pattern arguments
discussion: ConnectionGetObjects vs ConnectionGetTableSchema #1704 (comment): define a way to provide pagination tokens

Closes #4400.

lidavidm · 2026-08-17T06:05:21Z

I'm thinking we don't run-length-encode by default to simplify things further.

CurtHagenlocher

Exciting! I've left a few comments and questions resulting from a first, casual review of the code.

CurtHagenlocher · 2026-08-27T20:52:31Z

+/// |--------------------------|------------------------------|----------|
+/// | collection_name          | utf8 not null                |          |
+/// | collection_description   | utf8                         |          |
+/// | collection_schema        | extension<arrow.schema_json> |          |


The proposal for this type seems to have stalled :(.

I'm talking to Kent, hopefully we can keep it going. Otherwise, I may propose an arrow.schema_ipc (even if that's a bit ugly)

Having it not be an extension type would probably make it simpler (when there's a JSON option, regular arrow.json I think would suffice, given that the name of the field contains schema). Until then binary is probably fine, and IPC, though sort of gross, at least doesn't require most implementations to write an additional parser.

Sort of off-topic then but I think it's kind of an implementation flaw that extension types are so hard to work with in many Arrow implementations 😅

Hmm, I suppose in practice the extension type is unnecessary for these fields. In theory I'm thinking that databow and other UIs around ADBC could use the extension type to know to render it in a more user-friendly way. In practice hardcoding it may be fine just because there's only a few cases/contexts where you're returning a schema in a table?

The rendering is a good point, although if you use a functional-style matcher you can detect the parent struct based on the field names and print the whole thing nicely (probably better than you'd get by strictly keying on extension name).

Closes apache#4400.

amoeba · 2026-08-31T16:12:52Z

+/// defined by the collection. For example, a client may request a list of
+/// tables in the database, or a list of supported data types.
+///
+/// All drivers must implement a collection called "meta" (which is aliased to


Suggested change

/// All drivers must implement a collection called "meta" (which is aliased to

/// All drivers that implement this API must implement a collection called "meta" (which is aliased to

Isn't that kind of implicit?

I don't think so but I'll defer to you on the language.

amoeba · 2026-08-31T16:28:21Z

+/// \brief Filter the collection on the literal catalog name of the foreign
+///   key.


This was hard to understand for me as written. Does this option let me filter to tables that are referenced by a table whose name I provide?

My confusion goes for this and the other options related to foreign keys. Maybe we mean something more like,

Filter the collection on the literal catalog name of the referencing table.

NB: I didn't catch this on my first read through but gpt-5.6-sol flagged it and I found I wasn't able to explain it one way or the other.

It's ~directly ported from JDBC https://docs.oracle.com/javase/8/docs/api/java/sql/DatabaseMetaData.html#getCrossReference-java.lang.String-java.lang.String-java.lang.String-java.lang.String-java.lang.String-java.lang.String-

Though I noticed below I flubbed the filters list so will fix that

The way I understood it, I thought it was an option to filter by external catalogs that are non-native (i.e foreign) to the platform. For example external Iceberg catalogs in Databricks, reading AWS Glue from Snowflake etc.

kentkwu · 2026-09-01T18:27:43Z

+/// use a vendor-specific prefix (e.g. `postgresql.`) to avoid conflicts with
+/// future standardized collections. Drivers must not use the `adbc.` prefix.
+///
+/// Drivers may not necessarily accept filter options or other options before


"May not necessarily accept" feels a bit ambiguous. It seems there's an implicit state machine here with regards to ADBC_METADATA_* options relative to ADBC_METADATA_COLLECTION both in the lifecycle and what the valid/invalid states should be. Should the spec make that more explicit?

Yes, there's also bulk-ingest as part of the state machine.

I agree it needs to be made more explicit. If we're broadly OK with this approach to metadata, then I'll document it better.

kentkwu · 2026-09-01T18:52:16Z

+/// fixed.
+///
+/// Drivers may implement collections beyond those defined by ADBC, but must
+/// use a vendor-specific prefix (e.g. `postgresql.`) to avoid conflicts with


prefix field names with the vendor/driver name to differentiate them (e.g. 'POSTGRESQL:owner', not just 'owner').

vendor-specific prefix (e.g. postgresql.)

Could these two suggestions be made consistent? Or is this intentional?

I'd say that most of the time we prefer prefix. in ADBC, but for whatever reason when the metadata is in Arrow field/schema metadata we've gone with PREFIX: (a convention started upstream). It does feel arbitrary, though.

Ah, it's actually specified upstream: https://arrow.apache.org/docs/format/Columnar.html#custom-application-metadata

kentkwu · 2026-09-08T21:40:30Z

+/// vendor-specific metadata. Applications must access these by name or using
+/// an offset from the end of the schema and cannot assume that the index of
+/// the field will remain stable.  Hence, it is discouraged to access these
+/// fields by ordinal as this is brittle.  Drivers must add the fields at the


What's the value of mentioning offset-from-the-end here, as opposed to just specifying that field access must be by name?

It seems only stable for a single driver at a single version, since you'd need to know how many fields that driver appends and in what order within its block.

The paragraph also calls ordinal access brittle two sentences later, which seems to agree. Is there a scenario I'm missing?

It's to clarify that you can't assume we won't add more fields in the spec, so if you do use an ordinal, then it needs to be from the end.

serramatutu · 2026-09-09T10:03:24Z

+/// \brief Filter the collection on the literal routine name.
+///
+/// The type is char*.
+#define ADBC_METADATA_FILTER_ROUTINE "adbc.metadata.filter.routine"


To make sure I understand this right: we're defining anything that can run as user-defined code as a "routine", so for example Snowflake has SQL UDFs, Snowpark UDFS (Python/JS) and Stored Procedures, and all of those would fall under the same "routine" category?

And the way to filter those out if I wanted only Python UDFs for example us by using the routine_types filter, correct? The values of those filters will be platform dependent, so for Snowflake it'd be sql_udf, python_udf, js_udf, stored_procedure. And each driver would need a list of platform-specific supported routine_types.

Is it worth to standardize common routine types, or do you think that's a little too much? For example stored_procedure is a pretty standard thing across a lot of different databases, so maybe it would be nice to have a shared key for all of those?

Now that I think of it, maybe it's too early to standardize that... Databricks alone has like 3 different kinds of Python UDFs. And the definition of a stored_procedure is fuzzy...

serramatutu · 2026-09-09T10:18:05Z

+/// | constraint_deferrability | int16                   | (4)      |
+/// | constraint_match_type    | int16                   | (5)      |
+///
+/// 1. The 1-based index of the column pair within the foreign key (1 => first


Is there a reason why we're using 1-based indices?

In general, for most of your questions: we are directly copying JDBC/ODBC for these definitions for consistency.

serramatutu · 2026-09-09T10:43:32Z

+/// Some drivers may support this to allow fetching a large metadata
+/// collection in multiple calls. If the pagination token could not be used,
+/// the driver should return an error.
+#define ADBC_METADATA_OPTION_PAGINATION_TOKEN "adbc.metadata.pagination_token"


Two things:

I assume this is an "internal" key that will be used by the driver manager implementations to return stateful RecordBatchReaders that paginate under the hood when you pull a new batch, right? I think it would be nice if applications didn't need to worry about setting this option and keeping track of cursors across multiple calls. As an application I'd like to just consume record batches and the pagination cursors and state are handled by the driver manager.

Is this intended to be used as both an input option as well as an output metadata key? If my understanding is correct, the lifecycle is:

The app (driver manager) would call AdbcConnectionGetInfo without a adbc.metadata.pagination_token (the first call), then the driver returns a record batch with a adbc.metadata.pagination_token in the schema metadata.

The next call, the app passes in the same adbc.metadata.pagination_token. I don't understand where, since AdbcConnectionGetInfo has no explicit support for options. Does this get set as a connection-level parameter that is set before the call and unset after it returns? Does this get added together with the filters?

This repeats until the last page, where the driver returns no adbc.metadata.pagination_token, signifying it's done listing.

I think it would be nice to document this lifecycle somewhere. Not sure the right place for it is here, though.

You don't need this in normal operation. The driver manager also isn't involved. This is meant to let you resume an operation across different fetches; a regular fetch should already paginate.

lidavidm · 2026-09-14T04:20:11Z

@zeroshade can I get a look here? 🙏

lidavidm mentioned this pull request Aug 17, 2026

feat(format): add generic metadata API #4544

Closed

4 tasks

lidavidm force-pushed the spec-1.2.0 branch from dce2b2f to 1698b78 Compare August 17, 2026 08:05

lidavidm force-pushed the gh-4400-sandbox branch 3 times, most recently from e222af1 to 1a39168 Compare August 18, 2026 01:27

lidavidm requested review from CurtHagenlocher, Mandukhai-Alimaa, amoeba and paleolimbot August 18, 2026 01:28

lidavidm marked this pull request as ready for review August 27, 2026 02:32

lidavidm requested a review from zeroshade as a code owner August 27, 2026 02:32

This was referenced Aug 27, 2026

feat(format): add table_definition to GetObjects #4050

Closed

feat(format): expose functions/procedures in catalog metadata #4031

Closed

kentkwu mentioned this pull request Aug 27, 2026

feat(go): support updating table column descriptions adbc-drivers/bigquery#279

Closed

CurtHagenlocher reviewed Aug 27, 2026

View reviewed changes

feat(format): add generic metadata API

2d08fe1

Closes apache#4400.

lidavidm force-pushed the gh-4400-sandbox branch from a7ce389 to 2d08fe1 Compare August 31, 2026 07:57

clarifications

ed2c607

amoeba reviewed Aug 31, 2026

View reviewed changes

kentkwu reviewed Sep 1, 2026

View reviewed changes

kentkwu reviewed Sep 4, 2026

View reviewed changes

Comment thread c/include/arrow-adbc/adbc.h Outdated

lidavidm added 2 commits September 7, 2026 13:30

comments

076a2a5

clarify schema/catalog behavior

a15d036

kentkwu reviewed Sep 8, 2026

View reviewed changes

serramatutu reviewed Sep 9, 2026

View reviewed changes

lidavidm requested a review from amoeba September 14, 2026 04:19

	/// All drivers must implement a collection called "meta" (which is aliased to
	/// All drivers that implement this API must implement a collection called "meta" (which is aliased to

		/// \brief Filter the collection on the literal catalog name of the foreign
		/// key.

Conversation

lidavidm commented Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lidavidm commented Aug 17, 2026

Uh oh!

CurtHagenlocher left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

serramatutu Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

serramatutu Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

serramatutu Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lidavidm commented Sep 14, 2026

Uh oh!

Reviewers

Assignees

Labels

lidavidm commented Aug 17, 2026 •

edited

Loading

serramatutu Sep 9, 2026 •

edited

Loading

serramatutu Sep 9, 2026 •

edited

Loading

serramatutu Sep 9, 2026 •

edited

Loading