Skip to content

Column stats pruning during read planning issues its requests one after another #1007

Open

Description

@yihua

Feature Description

When a filtered plan prunes files with the metadata table's column_stats partition, the lookup lists the metadata timeline again although the planner already holds the metadata table with that timeline loaded, sends a HEAD before each whole-file log read, and reads the files one after another. On object storage each of those requests waits on the previous one, so planning time grows with the number of files in the partition.

Proposal: fence the base files with the timeline the metadata table handle already loaded, read each file with a single GET, and fetch the log files concurrently, bounded by the file-slice read concurrency.

Additional context

No response

Activity

  1. yihua commented on Nov 9, 2026

    @yihua
    ContributorAuthor

    Fixed in #1010.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions