Repository navigation
Skip zero-byte files in read_mpfile globs and lists (skip_empty_files) - #2
Closed
MatthewMooreZA wants to merge 35 commits into
Closed
MatthewMooreZA wants to merge 35 commits into
MatthewMooreZA wants to merge 35 commits into
Conversation
Write path: - Format FLOAT/DOUBLE with shortest round-trip output; std::to_string wrote 6 decimal places and silently turned small values into 0.000000 - Map BIGINT to N (was I, which reads back as a 32-bit INTEGER) - Quote fallback types (DATE, LIST, ...) written as text - Reject column names containing commas, quotes or line breaks - Close the output handle explicitly in finalize Docs/CI: - Rewrite stale CLAUDE.md (described placeholder functions and OpenSSL) - Fix UPDATING.md references to a nonexistent deploy job - Add non-blocking build job against DuckDB main for early warning of C++ API breakage ahead of v2.0 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XeGfNHb8c25dWFJFZyRXNf
- bench/memory_bench.sh: single-file peak RSS/time for read_mpfile vs read_csv (skip= past the metadata rows) on a byte-identical footerless copy - bench/glob_bench.sh: same comparison as the matched file count grows, with either constant total rows or constant rows per file - bench/rss_profile.py: RSS-over-time sampler for a single query All default to small data, <=2 threads, a memory_limit, and nice/ionice. The plain read_csv glob (no union_by_name) is excluded unless INCLUDE_PLAIN=1: it used 4-7 GB of RAM on 55 MB of data. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
read_mpfile always passed union_by_name=true to read_csv_auto. That makes DuckDB bind and sniff every file up front, retains an extra scan buffer, and reports a placeholder cardinality of 42 rows to the planner. Now union_by_name is only added when the matched files do not all have the same VARIABLE_TYPES columns and types (or any file lacks a schema row). A single file never needs it. Measured on a 542 MB single file, 1 thread: peak RSS 117 MB -> 89 MB, equal to read_csv (89-90 MB); plan cardinality estimate 42 -> ~12.6M rows. Globs of 2-20 identical-schema files match the equivalent read_csv call at 1, 2 and 4 threads. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Reading every matched file's header and VARIABLE_TYPES rows at bind time costs an open and a read per file, serially, which is expensive with many files and worse on remote storage. union_by_name=false (new default): globs and lists are only expanded. The dtypes come from the first file that has data and the other files are opened lazily by the scan, so bind cost no longer grows with file count. Opens for a 20-file glob drop from 40 (previously) / 60 (union) to 21. Files must share columns; differing files, empty files and missing files are errors, as with read_csv. union_by_name=true: previous behaviour. Every schema is parsed, files with no data are dropped, schemas are merged and union_by_name is forwarded. BREAKING: globs/lists of files with differing columns or types, or that include empty files, now need union_by_name=true. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
ParseMPFileSchema swallowed every exception when opening a file and returned an empty schema. For a single file that was harmless (read_csv then raised the same error), but for globs and lists the empty schema made the file look like it had no data, so it was filtered out and the query returned partial results with no error. On remote storage this would turn a throttling or credentials error into silently incomplete data. Open errors now propagate in every mode. A file that opens but has no header row is still treated as "no data" (dropped only with union_by_name=true). The implicit FROM 'glob.rpt' scan passed the glob pattern itself to ParseMPFileSchema and relied on the swallowed error; it now expands the glob like read_mpfile does, which also gives it VARIABLE_TYPES column types. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
ignore_errors defaults to false, like read_csv. When true, rows that fail to convert (e.g. a non-numeric value in an S/I/N column) are skipped instead of raising an error. It is forwarded unchanged to read_csv_auto, so it behaves exactly as read_csv does: it skips rows, not files (an unreadable file is still an error), and only the columns a query reads are converted. Accepted by both the string/glob and list overloads and combines with union_by_name and filename. The named parameters that are forwarded now travel together in an MPReadOptions struct instead of growing the helper signatures. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Call out the union_by_name default change as a breaking change in mpduck 2.0 and add an upgrading table showing what versions before 2.0 did, what the default does now, and how to keep the old behaviour (union_by_name = true). Note that an unreadable file is now an error, and that ignore_errors is available from mpduck 2.0. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
DuckDB's C++ extension API is internal and changed in 2.0. The differences mpduck touches are isolated in src/include/mpduck_compat.hpp, selected by the presence of duckdb/common/identifier.hpp (2.0 only): - Identifier vs string for column names, aliases and COPY option keys - ConstantExpression(const Value&) deleted -> ConstantExpression::FromValue - StarExpression::exclude_list private -> ExcludeListMutable() - Vector::Flatten(count) -> Flatten() The rest of the code calls MPName/MPMakeName/MPNameString/MPConstant/ MPExcludeColumn/MPFlatten and reads the same on both versions. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
DuckDB 2.0 probes the COPY target path with DirectoryExists (and may call RemoveFile when overwriting). MPFileSystem claims .rpt/.prn/.fac paths, so the base-class default threw "DirectoryExists is not implemented!". Forward both to the underlying filesystem like FileExists and MoveFile already do. Harmless on 1.5.x, where the calls are simply not made for COPY targets. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
named_parameters is keyed by Identifier in 2.0, so a string name must go through MPMakeName. No change on 1.5.x. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
DuckDB 2.0.0 removed TableFunction::named_parameters; options are typed kwargs on the function signature. Add MPAddNamedParameters to the compat header (WithTypedKwargs on 2.0, named_parameters on 1.5.x) and use it for read_mpfile's filename, union_by_name and ignore_errors. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ScTRuBeUFVBV8GM7JAazCL
Pin duckdb to v2.0.0 and extension-ci-tools to main (no v2.0.0 branch exists yet; v2.0-cyanoptera was the pre-release branch). The stable and code-quality jobs build v2.0.0; a new duckdb-1-5-build job keeps the 1.5.x line building from the same source. Update docs/UPDATING.md to match. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ScTRuBeUFVBV8GM7JAazCL
Runs a matrix of generated mpfile shapes (narrow, wide, string-heavy, sparse, footer, many small files) against read_csv with skip=3, with and without explicit dtypes, recording wall time, CPU time and peak RSS. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QmFaNFyKJdqhEbv1LqvNzZ
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QmFaNFyKJdqhEbv1LqvNzZ
The release shell does not preload mpduck, and /usr/bin/time is not always installed, so load the extension explicitly and take ru_maxrss from wait4. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QmFaNFyKJdqhEbv1LqvNzZ
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QmFaNFyKJdqhEbv1LqvNzZ
GitHub rejects continue-on-error on jobs that call a reusable workflow, so the whole run failed at startup with no jobs. Remove the DuckDB main job and pin the 2.0 jobs to the v2.0-cyanoptera extension-ci-tools branch, as duckdb/extension-template does. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JdGwgmyotDmadSabixNBeH
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JdGwgmyotDmadSabixNBeH
identifier.hpp exists in DuckDB 1.5.6, so the compat header picked the 2.0 Identifier API there and the 1.5.x build failed. Switch on a header that only exists in 2.0. The missing-file test matched the Linux OS error text; match the file name so it passes on Windows too. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JdGwgmyotDmadSabixNBeH
Most real mpfiles carry no VARIABLE_TYPES row, so read_mpfile passes no dtypes to read_csv_auto. Every dataset now has a twin written without types, with the read_csv baseline skipping one row fewer. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QmFaNFyKJdqhEbv1LqvNzZ
read_mpfile left dialect detection to the CSV sniffer. It enumerates delimiter/quote/escape candidates and re-scans later chunks when they tie, which made wide files and globs slow (and, when dtypes are passed, repeats the search for every file in a glob). Always forward delim, quote, escape and new_line, defaulting to the Prophet dialect: ',' '"' '"' and CRLF. They can be overridden with read_csv's parameter names; sep is accepted as an alias for delim, and only ',' is valid because MPFileSystem rebuilds the header rows on commas. Files with LF line endings now need new_line='\n'; the three LF tests pass it. Adds tests and a single-quote fixture for the overrides. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QmFaNFyKJdqhEbv1LqvNzZ
make runs the repo formatter, which reflows the source list in CMakeLists.txt on every build; commit its output so the tree stays clean. .cache/ is build output. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QmFaNFyKJdqhEbv1LqvNzZ
Pinning delim, quote and escape removes the sniffer's dialect search; also pinning new_line measured as no change (within +/-9% either way), but it made pure-LF files fail. Forward new_line only when the user sets it, and put the LF tests back to reading without options. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QmFaNFyKJdqhEbv1LqvNzZ
Windows reports 'The system cannot find the file specified', so the Linux wording failed the Windows test job. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JdGwgmyotDmadSabixNBeH
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QmFaNFyKJdqhEbv1LqvNzZ
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QmFaNFyKJdqhEbv1LqvNzZ
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QmFaNFyKJdqhEbv1LqvNzZ # Conflicts: # src/mpduck_extension.cpp
The tests read fixtures that are deliberately CRLF or LF, and a checkout with core.autocrlf (the Git for Windows default) would rewrite them. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QmFaNFyKJdqhEbv1LqvNzZ
The release is versioned 1.0.0, the first stable release after 0.0.x, so the upgrade notes and the 'available from' labels say 1.0 instead of 2.0. List the pinned CSV dialect in the 1.0 upgrade notes, and state that the mpduck version is independent of the DuckDB versions the same source builds against. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QmFaNFyKJdqhEbv1LqvNzZ
…_csv_auto Pin delim, quote and escape (overridable, new_line left to auto-detection), add the read_mpfile vs read_csv benchmark harness and results, mark fixtures -text, and document the upcoming mpduck 1.0. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QmFaNFyKJdqhEbv1LqvNzZ
Add a skip_empty_files option (default true) that drops zero-byte files from a glob or list at bind time, for both union_by_name modes. skip_empty_files=false restores the previous behaviour. A single literal path is never filtered. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VdnqtS7GQEMhLC5UCDk66a
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VdnqtS7GQEMhLC5UCDk66a
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VdnqtS7GQEMhLC5UCDk66a
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VdnqtS7GQEMhLC5UCDk66a
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
skip_empty_filesoption toread_mpfile(defaulttrue). Zero-byte files in a glob or list are dropped at bind time, for bothunion_by_namemodes and forFROM 'x*.rpt', so one empty file no longer fails the whole read.skip_empty_files = falserestores the previous behaviour.IsZeroByteMPFile), with no reading, and uses the raw file size via theMPFileSystembypass. A single literal path is never filtered, and a non-empty file with no header row is not covered.union_by_nameboth false and true, plus globs,skip_empty_files = false, and a non-boolean value. Fixtures are intest/data/skip_empty/. README andCLAUDE.mdare updated.Test plan
./build/release/test/unittest "test/sql/mpduck.test"passes (446 assertions) on the merged treeGenerated by Claude Code