Repository navigation
Pin the CSV dialect read_mpfile passes to read_csv_auto; add read_mpfile vs read_csv benchmarks - #1
Merged
Conversation
Runs a matrix of generated mpfile shapes (narrow, wide, string-heavy, sparse, footer, many small files) against read_csv with skip=3, with and without explicit dtypes, recording wall time, CPU time and peak RSS. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QmFaNFyKJdqhEbv1LqvNzZ
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QmFaNFyKJdqhEbv1LqvNzZ
The release shell does not preload mpduck, and /usr/bin/time is not always installed, so load the extension explicitly and take ru_maxrss from wait4. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QmFaNFyKJdqhEbv1LqvNzZ
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QmFaNFyKJdqhEbv1LqvNzZ
Most real mpfiles carry no VARIABLE_TYPES row, so read_mpfile passes no dtypes to read_csv_auto. Every dataset now has a twin written without types, with the read_csv baseline skipping one row fewer. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QmFaNFyKJdqhEbv1LqvNzZ
read_mpfile left dialect detection to the CSV sniffer. It enumerates delimiter/quote/escape candidates and re-scans later chunks when they tie, which made wide files and globs slow (and, when dtypes are passed, repeats the search for every file in a glob). Always forward delim, quote, escape and new_line, defaulting to the Prophet dialect: ',' '"' '"' and CRLF. They can be overridden with read_csv's parameter names; sep is accepted as an alias for delim, and only ',' is valid because MPFileSystem rebuilds the header rows on commas. Files with LF line endings now need new_line='\n'; the three LF tests pass it. Adds tests and a single-quote fixture for the overrides. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QmFaNFyKJdqhEbv1LqvNzZ
make runs the repo formatter, which reflows the source list in CMakeLists.txt on every build; commit its output so the tree stays clean. .cache/ is build output. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QmFaNFyKJdqhEbv1LqvNzZ
Pinning delim, quote and escape removes the sniffer's dialect search; also pinning new_line measured as no change (within +/-9% either way), but it made pure-LF files fail. Forward new_line only when the user sets it, and put the LF tests back to reading without options. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QmFaNFyKJdqhEbv1LqvNzZ
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QmFaNFyKJdqhEbv1LqvNzZ
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QmFaNFyKJdqhEbv1LqvNzZ
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QmFaNFyKJdqhEbv1LqvNzZ # Conflicts: # src/mpduck_extension.cpp
The tests read fixtures that are deliberately CRLF or LF, and a checkout with core.autocrlf (the Git for Windows default) would rewrite them. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QmFaNFyKJdqhEbv1LqvNzZ
The release is versioned 1.0.0, the first stable release after 0.0.x, so the upgrade notes and the 'available from' labels say 1.0 instead of 2.0. List the pinned CSV dialect in the 1.0 upgrade notes, and state that the mpduck version is independent of the DuckDB versions the same source builds against. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QmFaNFyKJdqhEbv1LqvNzZ
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
read_mpfileleft CSV dialect detection to DuckDB's sniffer. This pins the dialect, exposes it as parameters, and adds a benchmark harness with a write-up of the results.Dialect parameters (
src/mpduck_extension.cpp)delim/sep,quoteandescapeare always forwarded toread_csv_auto, defaulting to,,"and". Each can be overridden with theread_csvparameter of the same name.,is valid fordelim/sep(any other value raises an error):MPFileSystemsplits and rebuilds the!header rows on commas.new_lineis forwarded only when the user sets it. Pinning it measured as no change (-8% to +9%, both directions) and would make pure-LF files fail, so CRLF and LF files both read with no options.read_mpfileoverloads share oneMPReadParameters()list.Why. With the dialect left to the sniffer,
callgrindshowed the extra bind cost was all inCSVSniffer::DetectDialect -> RefineCandidates: delimiter x quote x escape candidates tie and later chunks are re-scanned for each, at a cost that grows with row width. Passingdtypesalso sends later files in a glob through the full sniffer.Benchmarks (
bench/perf_matrix.py,bench/RESULTS.md): a matrix of generated mpfile shapes (narrow, wide, string-heavy, sparse, footer, 500 small files, 100 x ~100 MB glob), each with and without aVARIABLE_TYPESrow, againstread_csv(skip=...).Results (
count(*), best of 3,read_mpfileonly)Against
read_csvnow: level on ordinary files, 20-30% faster on 300-column files, 10-20x faster on the 500-file glob, and on a glob of 100 x ~100 MB filesread_csvis OOM-killed at ~13.6 GB on every query whileread_mpfilecompletes in ~2 GB. Part of the glob advantage is thatread_csvneedsskip, which is what makes it slow and memory-hungry on multi-file reads;read_mpfilestrips the metadata rows inMPFileSysteminstead. Caveats (single machine, ~5-10% noise, generated data only, 10 GB runs are best-of-2 and page-cache dependent) are inbench/RESULTS.md.Tests and docs
make test), up from 388: the new ones cover the overrides,sep/delim, thedelimrestriction, NULL handling,new_linemismatch and validation, and a single-quote fixture (test/data/quote_single.rpt).CLAUDE.mdarchitecture note updated.duckdb-2.0-readinesshas been merged into this branch. The one conflict was insrc/mpduck_extension.cpp, where the base had reflowed the twoMPAddNamedParameterscalls that this PR replaces with the shared helper; I kept this branch's side.Notes for review
quote/escapeoverrides.Generated by Claude Code