Skip to content

Pin the CSV dialect read_mpfile passes to read_csv_auto; add read_mpfile vs read_csv benchmarks - #1

Merged
MatthewMooreZA merged 13 commits into
duckdb-2.0-readinessfrom
ccr-8f10315d-um6l8w
Oct 24, 2026
Merged

MatthewMooreZA merged 13 commits into
duckdb-2.0-readinessfrom
ccr-8f10315d-um6l8w

Conversation

@MatthewMooreZA

Copy link
Copy Markdown
Owner

What

read_mpfile left CSV dialect detection to DuckDB's sniffer. This pins the dialect, exposes it as parameters, and adds a benchmark harness with a write-up of the results.

Dialect parameters (src/mpduck_extension.cpp)

  • delim/sep, quote and escape are always forwarded to read_csv_auto, defaulting to ,, " and ". Each can be overridden with the read_csv parameter of the same name.
  • Only , is valid for delim/sep (any other value raises an error): MPFileSystem splits and rebuilds the ! header rows on commas.
  • new_line is forwarded only when the user sets it. Pinning it measured as no change (-8% to +9%, both directions) and would make pure-LF files fail, so CRLF and LF files both read with no options.
  • Both read_mpfile overloads share one MPReadParameters() list.

Why. With the dialect left to the sniffer, callgrind showed the extra bind cost was all in CSVSniffer::DetectDialect -> RefineCandidates: delimiter x quote x escape candidates tie and later chunks are re-scanned for each, at a cost that grows with row width. Passing dtypes also sends later files in a glob through the full sniffer.

Benchmarks (bench/perf_matrix.py, bench/RESULTS.md): a matrix of generated mpfile shapes (narrow, wide, string-heavy, sparse, footer, 500 small files, 100 x ~100 MB glob), each with and without a VARIABLE_TYPES row, against read_csv(skip=...).

Results (count(*), best of 3, read_mpfile only)

Scenario Before After
wide 300 cols, typed, 1 / 4 threads 1.94 / 1.63 s 0.97 / 0.74 s
wide 300 cols, no types, 1 / 4 threads 1.81 / 1.41 s 1.01 / 0.64 s
500 small files, typed, 1 thread 3.51 s 0.77 s
narrow 100k, 1 thread 0.13 s 0.08 s

Against read_csv now: level on ordinary files, 20-30% faster on 300-column files, 10-20x faster on the 500-file glob, and on a glob of 100 x ~100 MB files read_csv is OOM-killed at ~13.6 GB on every query while read_mpfile completes in ~2 GB. Part of the glob advantage is that read_csv needs skip, which is what makes it slow and memory-hungry on multi-file reads; read_mpfile strips the metadata rows in MPFileSystem instead. Caveats (single machine, ~5-10% noise, generated data only, 10 GB runs are best-of-2 and page-cache dependent) are in bench/RESULTS.md.

Tests and docs

  • 427 SQL logic test assertions pass on the merged tree (make test), up from 388: the new ones cover the overrides, sep/delim, the delim restriction, NULL handling, new_line mismatch and validation, and a single-quote fixture (test/data/quote_single.rpt).
  • README section documenting the defaults; CLAUDE.md architecture note updated.
  • duckdb-2.0-readiness has been merged into this branch. The one conflict was in src/mpduck_extension.cpp, where the base had reflowed the two MPAddNamedParameters calls that this PR replaces with the shared helper; I kept this branch's side.

Notes for review

  • The defaults assume double-quote (RFC 4180) quoting. I had no real Prophet output, only the repo fixtures and generated data; files using another convention need quote/escape overrides.
  • CI has not run on this branch.

Generated by Claude Code

claude added 13 commits October 23, 2026 19:27
Runs a matrix of generated mpfile shapes (narrow, wide, string-heavy,
sparse, footer, many small files) against read_csv with skip=3, with and
without explicit dtypes, recording wall time, CPU time and peak RSS.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QmFaNFyKJdqhEbv1LqvNzZ
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QmFaNFyKJdqhEbv1LqvNzZ
The release shell does not preload mpduck, and /usr/bin/time is not
always installed, so load the extension explicitly and take ru_maxrss
from wait4.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QmFaNFyKJdqhEbv1LqvNzZ
Most real mpfiles carry no VARIABLE_TYPES row, so read_mpfile passes no
dtypes to read_csv_auto. Every dataset now has a twin written without
types, with the read_csv baseline skipping one row fewer.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QmFaNFyKJdqhEbv1LqvNzZ
read_mpfile left dialect detection to the CSV sniffer. It enumerates
delimiter/quote/escape candidates and re-scans later chunks when they tie,
which made wide files and globs slow (and, when dtypes are passed, repeats
the search for every file in a glob).

Always forward delim, quote, escape and new_line, defaulting to the Prophet
dialect: ',' '"' '"' and CRLF. They can be overridden with read_csv's
parameter names; sep is accepted as an alias for delim, and only ',' is
valid because MPFileSystem rebuilds the header rows on commas.

Files with LF line endings now need new_line='\n'; the three LF tests pass
it. Adds tests and a single-quote fixture for the overrides.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QmFaNFyKJdqhEbv1LqvNzZ
make runs the repo formatter, which reflows the source list in CMakeLists.txt
on every build; commit its output so the tree stays clean. .cache/ is build
output.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QmFaNFyKJdqhEbv1LqvNzZ
Pinning delim, quote and escape removes the sniffer's dialect search; also
pinning new_line measured as no change (within +/-9% either way), but it made
pure-LF files fail. Forward new_line only when the user sets it, and put the
LF tests back to reading without options.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QmFaNFyKJdqhEbv1LqvNzZ
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QmFaNFyKJdqhEbv1LqvNzZ
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QmFaNFyKJdqhEbv1LqvNzZ

# Conflicts:
#	src/mpduck_extension.cpp
The tests read fixtures that are deliberately CRLF or LF, and a checkout with
core.autocrlf (the Git for Windows default) would rewrite them.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QmFaNFyKJdqhEbv1LqvNzZ
The release is versioned 1.0.0, the first stable release after 0.0.x, so the
upgrade notes and the 'available from' labels say 1.0 instead of 2.0. List the
pinned CSV dialect in the 1.0 upgrade notes, and state that the mpduck version
is independent of the DuckDB versions the same source builds against.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QmFaNFyKJdqhEbv1LqvNzZ
@MatthewMooreZA
MatthewMooreZA merged commit f6e0798 into duckdb-2.0-readiness Oct 24, 2026
38 checks passed
@MatthewMooreZA
MatthewMooreZA deleted the ccr-8f10315d-um6l8w branch October 25, 2026 10:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants