Skip to content

Skip zero-byte files in read_mpfile globs and lists (skip_empty_files) - #2

Closed
MatthewMooreZA wants to merge 35 commits into
mainfrom
duckdb-2.0-readiness
Closed

MatthewMooreZA wants to merge 35 commits into
mainfrom
duckdb-2.0-readiness

Conversation

@MatthewMooreZA

Copy link
Copy Markdown
Owner

Summary

  • Add a skip_empty_files option to read_mpfile (default true). Zero-byte files in a glob or list are dropped at bind time, for both union_by_name modes and for FROM 'x*.rpt', so one empty file no longer fails the whole read. skip_empty_files = false restores the previous behaviour.
  • The check is one open and size lookup per file (IsZeroByteMPFile), with no reading, and uses the raw file size via the MPFileSystem bypass. A single literal path is never filtered, and a non-empty file with no header row is not covered.
  • Tests cover the empty file first and not first, with union_by_name both false and true, plus globs, skip_empty_files = false, and a non-boolean value. Fixtures are in test/data/skip_empty/. README and CLAUDE.md are updated.

Test plan

  • ./build/release/test/unittest "test/sql/mpduck.test" passes (446 assertions) on the merged tree
  • CI green

Generated by Claude Code

MatthewMooreZA and others added 30 commits September 19, 2026 14:46
Write path:
- Format FLOAT/DOUBLE with shortest round-trip output; std::to_string
  wrote 6 decimal places and silently turned small values into 0.000000
- Map BIGINT to N (was I, which reads back as a 32-bit INTEGER)
- Quote fallback types (DATE, LIST, ...) written as text
- Reject column names containing commas, quotes or line breaks
- Close the output handle explicitly in finalize

Docs/CI:
- Rewrite stale CLAUDE.md (described placeholder functions and OpenSSL)
- Fix UPDATING.md references to a nonexistent deploy job
- Add non-blocking build job against DuckDB main for early warning of
  C++ API breakage ahead of v2.0

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XeGfNHb8c25dWFJFZyRXNf
- bench/memory_bench.sh: single-file peak RSS/time for read_mpfile vs
  read_csv (skip= past the metadata rows) on a byte-identical footerless copy
- bench/glob_bench.sh: same comparison as the matched file count grows, with
  either constant total rows or constant rows per file
- bench/rss_profile.py: RSS-over-time sampler for a single query

All default to small data, <=2 threads, a memory_limit, and nice/ionice.
The plain read_csv glob (no union_by_name) is excluded unless
INCLUDE_PLAIN=1: it used 4-7 GB of RAM on 55 MB of data.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
read_mpfile always passed union_by_name=true to read_csv_auto. That makes
DuckDB bind and sniff every file up front, retains an extra scan buffer,
and reports a placeholder cardinality of 42 rows to the planner.

Now union_by_name is only added when the matched files do not all have the
same VARIABLE_TYPES columns and types (or any file lacks a schema row). A
single file never needs it.

Measured on a 542 MB single file, 1 thread: peak RSS 117 MB -> 89 MB, equal
to read_csv (89-90 MB); plan cardinality estimate 42 -> ~12.6M rows. Globs
of 2-20 identical-schema files match the equivalent read_csv call at 1, 2
and 4 threads.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Reading every matched file's header and VARIABLE_TYPES rows at bind time
costs an open and a read per file, serially, which is expensive with many
files and worse on remote storage.

union_by_name=false (new default): globs and lists are only expanded. The
dtypes come from the first file that has data and the other files are
opened lazily by the scan, so bind cost no longer grows with file count.
Opens for a 20-file glob drop from 40 (previously) / 60 (union) to 21.
Files must share columns; differing files, empty files and missing files
are errors, as with read_csv.

union_by_name=true: previous behaviour. Every schema is parsed, files with
no data are dropped, schemas are merged and union_by_name is forwarded.

BREAKING: globs/lists of files with differing columns or types, or that
include empty files, now need union_by_name=true.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
ParseMPFileSchema swallowed every exception when opening a file and returned
an empty schema. For a single file that was harmless (read_csv then raised
the same error), but for globs and lists the empty schema made the file look
like it had no data, so it was filtered out and the query returned partial
results with no error. On remote storage this would turn a throttling or
credentials error into silently incomplete data.

Open errors now propagate in every mode. A file that opens but has no header
row is still treated as "no data" (dropped only with union_by_name=true).

The implicit FROM 'glob.rpt' scan passed the glob pattern itself to
ParseMPFileSchema and relied on the swallowed error; it now expands the glob
like read_mpfile does, which also gives it VARIABLE_TYPES column types.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
ignore_errors defaults to false, like read_csv. When true, rows that fail to
convert (e.g. a non-numeric value in an S/I/N column) are skipped instead of
raising an error. It is forwarded unchanged to read_csv_auto, so it behaves
exactly as read_csv does: it skips rows, not files (an unreadable file is
still an error), and only the columns a query reads are converted.

Accepted by both the string/glob and list overloads and combines with
union_by_name and filename. The named parameters that are forwarded now
travel together in an MPReadOptions struct instead of growing the helper
signatures.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Call out the union_by_name default change as a breaking change in mpduck 2.0
and add an upgrading table showing what versions before 2.0 did, what the
default does now, and how to keep the old behaviour (union_by_name = true).
Note that an unreadable file is now an error, and that ignore_errors is
available from mpduck 2.0.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
DuckDB's C++ extension API is internal and changed in 2.0. The differences
mpduck touches are isolated in src/include/mpduck_compat.hpp, selected by
the presence of duckdb/common/identifier.hpp (2.0 only):

- Identifier vs string for column names, aliases and COPY option keys
- ConstantExpression(const Value&) deleted -> ConstantExpression::FromValue
- StarExpression::exclude_list private -> ExcludeListMutable()
- Vector::Flatten(count) -> Flatten()

The rest of the code calls MPName/MPMakeName/MPNameString/MPConstant/
MPExcludeColumn/MPFlatten and reads the same on both versions.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
DuckDB 2.0 probes the COPY target path with DirectoryExists (and may call
RemoveFile when overwriting). MPFileSystem claims .rpt/.prn/.fac paths, so the
base-class default threw "DirectoryExists is not implemented!". Forward both
to the underlying filesystem like FileExists and MoveFile already do.

Harmless on 1.5.x, where the calls are simply not made for COPY targets.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
named_parameters is keyed by Identifier in 2.0, so a string name must go
through MPMakeName. No change on 1.5.x.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
DuckDB 2.0.0 removed TableFunction::named_parameters; options are typed
kwargs on the function signature. Add MPAddNamedParameters to the compat
header (WithTypedKwargs on 2.0, named_parameters on 1.5.x) and use it for
read_mpfile's filename, union_by_name and ignore_errors.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ScTRuBeUFVBV8GM7JAazCL
Pin duckdb to v2.0.0 and extension-ci-tools to main (no v2.0.0 branch
exists yet; v2.0-cyanoptera was the pre-release branch). The stable and
code-quality jobs build v2.0.0; a new duckdb-1-5-build job keeps the 1.5.x
line building from the same source. Update docs/UPDATING.md to match.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ScTRuBeUFVBV8GM7JAazCL
Runs a matrix of generated mpfile shapes (narrow, wide, string-heavy,
sparse, footer, many small files) against read_csv with skip=3, with and
without explicit dtypes, recording wall time, CPU time and peak RSS.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QmFaNFyKJdqhEbv1LqvNzZ
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QmFaNFyKJdqhEbv1LqvNzZ
The release shell does not preload mpduck, and /usr/bin/time is not
always installed, so load the extension explicitly and take ru_maxrss
from wait4.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QmFaNFyKJdqhEbv1LqvNzZ
GitHub rejects continue-on-error on jobs that call a reusable workflow,
so the whole run failed at startup with no jobs. Remove the DuckDB main
job and pin the 2.0 jobs to the v2.0-cyanoptera extension-ci-tools
branch, as duckdb/extension-template does.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JdGwgmyotDmadSabixNBeH
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JdGwgmyotDmadSabixNBeH
identifier.hpp exists in DuckDB 1.5.6, so the compat header picked the 2.0
Identifier API there and the 1.5.x build failed. Switch on a header that
only exists in 2.0. The missing-file test matched the Linux OS error text;
match the file name so it passes on Windows too.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JdGwgmyotDmadSabixNBeH
Most real mpfiles carry no VARIABLE_TYPES row, so read_mpfile passes no
dtypes to read_csv_auto. Every dataset now has a twin written without
types, with the read_csv baseline skipping one row fewer.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QmFaNFyKJdqhEbv1LqvNzZ
read_mpfile left dialect detection to the CSV sniffer. It enumerates
delimiter/quote/escape candidates and re-scans later chunks when they tie,
which made wide files and globs slow (and, when dtypes are passed, repeats
the search for every file in a glob).

Always forward delim, quote, escape and new_line, defaulting to the Prophet
dialect: ',' '"' '"' and CRLF. They can be overridden with read_csv's
parameter names; sep is accepted as an alias for delim, and only ',' is
valid because MPFileSystem rebuilds the header rows on commas.

Files with LF line endings now need new_line='\n'; the three LF tests pass
it. Adds tests and a single-quote fixture for the overrides.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QmFaNFyKJdqhEbv1LqvNzZ
make runs the repo formatter, which reflows the source list in CMakeLists.txt
on every build; commit its output so the tree stays clean. .cache/ is build
output.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QmFaNFyKJdqhEbv1LqvNzZ
Pinning delim, quote and escape removes the sniffer's dialect search; also
pinning new_line measured as no change (within +/-9% either way), but it made
pure-LF files fail. Forward new_line only when the user sets it, and put the
LF tests back to reading without options.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QmFaNFyKJdqhEbv1LqvNzZ
Windows reports 'The system cannot find the file specified', so the
Linux wording failed the Windows test job.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JdGwgmyotDmadSabixNBeH
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QmFaNFyKJdqhEbv1LqvNzZ
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QmFaNFyKJdqhEbv1LqvNzZ

# Conflicts:
#	src/mpduck_extension.cpp
The tests read fixtures that are deliberately CRLF or LF, and a checkout with
core.autocrlf (the Git for Windows default) would rewrite them.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QmFaNFyKJdqhEbv1LqvNzZ
The release is versioned 1.0.0, the first stable release after 0.0.x, so the
upgrade notes and the 'available from' labels say 1.0 instead of 2.0. List the
pinned CSV dialect in the 1.0 upgrade notes, and state that the mpduck version
is independent of the DuckDB versions the same source builds against.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QmFaNFyKJdqhEbv1LqvNzZ
MatthewMooreZA and others added 5 commits October 24, 2026 11:36
…_csv_auto

Pin delim, quote and escape (overridable, new_line left to auto-detection), add the read_mpfile vs read_csv benchmark harness and results, mark fixtures -text, and document the upcoming mpduck 1.0.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QmFaNFyKJdqhEbv1LqvNzZ
Add a skip_empty_files option (default true) that drops zero-byte files from a
glob or list at bind time, for both union_by_name modes. skip_empty_files=false
restores the previous behaviour. A single literal path is never filtered.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VdnqtS7GQEMhLC5UCDk66a
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VdnqtS7GQEMhLC5UCDk66a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants