Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
9cc5c1b
Fix mpfile write defects and prepare for DuckDB 2.0
MatthewMooreZA Sep 19, 2026
7d5df32
Add gentle memory benchmarks: read_mpfile vs read_csv
MatthewMooreZA Sep 20, 2026
4d7557b
Drop union_by_name when matched files share one schema
MatthewMooreZA Sep 20, 2026
f162a75
Add union_by_name to read_mpfile, defaulting to false like read_csv
MatthewMooreZA Sep 20, 2026
5b03acb
Stop silently dropping unreadable files from read_mpfile globs
MatthewMooreZA Sep 20, 2026
3017cf6
Add ignore_errors to read_mpfile, forwarded to read_csv
MatthewMooreZA Sep 20, 2026
9f6133a
README: document union_by_name and ignore_errors as mpduck 2.0 changes
MatthewMooreZA Sep 20, 2026
41cfcf9
Build against both DuckDB 1.5.x and 2.0 from one source tree
MatthewMooreZA Sep 20, 2026
33cd7dd
Forward DirectoryExists and RemoveFile in MPFileSystem
MatthewMooreZA Sep 20, 2026
4213111
Look up named parameters by Identifier on DuckDB 2.0
MatthewMooreZA Sep 20, 2026
8d40d75
Document the single-source 1.5.x/2.0 build
MatthewMooreZA Sep 20, 2026
9fd1a6e
Declare named parameters through the signature on DuckDB 2.0.0
MatthewMooreZA Oct 23, 2026
29b98b6
Target DuckDB v2.0.0 in CI and submodules
MatthewMooreZA Oct 23, 2026
768fdfc
Add perf_matrix.py benchmark comparing read_mpfile with read_csv
claude Oct 23, 2026
a385350
Add 100 x ~100MB glob scenario to perf_matrix
claude Oct 23, 2026
c1bb12c
perf_matrix: LOAD the extension and measure RSS via wait4
claude Oct 23, 2026
82248f8
perf_matrix: record OOM kills as results instead of aborting
claude Oct 23, 2026
e57da2f
Fix CI: drop invalid continue-on-error job, use v2.0-cyanoptera tools
claude Oct 23, 2026
9454eac
Ignore clangd cache
claude Oct 23, 2026
e6bb93f
Fix 1.5.x build switch and Windows missing-file test
claude Oct 24, 2026
e5bbb7d
perf_matrix: add _notypes twins written with INCLUDE_TYPES false
claude Oct 24, 2026
4cc44e2
Pin the CSV dialect read_mpfile hands to read_csv_auto
claude Oct 24, 2026
ceea035
Commit the formatter's CMakeLists layout and ignore .cache
claude Oct 24, 2026
564a51a
Do not pin new_line by default
claude Oct 24, 2026
8debb30
Match file name, not OS error text, in remaining missing-file tests
claude Oct 24, 2026
2819849
Add bench/RESULTS.md with the read_mpfile vs read_csv results
claude Oct 24, 2026
8c55771
README: document the dialect defaults as a table
claude Oct 24, 2026
5f6a28d
Merge duckdb-2.0-readiness into ccr-8f10315d-um6l8w
claude Oct 24, 2026
91dfb33
Mark fixtures and mpfiles -text so Git never converts their line endings
claude Oct 24, 2026
2aea57a
README: name the upcoming release mpduck 1.0
claude Oct 24, 2026
f6e0798
Merge pull request #1: pin the CSV dialect read_mpfile passes to read…
MatthewMooreZA Oct 24, 2026
6389a8e
Skip zero-byte files in read_mpfile globs and lists by default
claude Oct 24, 2026
5a6997a
Merge origin/duckdb-2.0-readiness into skip_empty_files branch
claude Oct 24, 2026
26f5213
README: drop version note from skip_empty_files
claude Oct 24, 2026
9ae5f1f
README: drop inline version notes from union_by_name and ignore_errors
claude Oct 24, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# Test fixtures and mpfiles depend on their exact line endings (most are CRLF, a few are deliberately LF), so never
# let Git convert them, whatever core.autocrlf is set to on the machine doing the checkout.
test/data/** -text
*.rpt -text
*.prn -text
*.fac -text
21 changes: 15 additions & 6 deletions .github/workflows/MainDistributionPipeline.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,17 +14,26 @@ concurrency:
jobs:
duckdb-stable-build:
name: Build extension binaries
uses: duckdb/extension-ci-tools/.github/workflows/_extension_distribution.yml@v1.5.1
uses: duckdb/extension-ci-tools/.github/workflows/_extension_distribution.yml@v2.0-cyanoptera
with:
duckdb_version: v1.5.2
ci_tools_version: v1.5.1
duckdb_version: v2.0.0
ci_tools_version: v2.0-cyanoptera
extension_name: mpduck

# The same source tree still supports DuckDB 1.5.x (see docs/UPDATING.md).
duckdb-1-5-build:
name: Build extension binaries (DuckDB 1.5.x)
uses: duckdb/extension-ci-tools/.github/workflows/_extension_distribution.yml@v1.5.6
with:
duckdb_version: v1.5.6
ci_tools_version: v1.5.6
extension_name: mpduck

code-quality-check:
name: Code Quality Check
uses: duckdb/extension-ci-tools/.github/workflows/_extension_code_quality.yml@v1.5.1
uses: duckdb/extension-ci-tools/.github/workflows/_extension_code_quality.yml@v2.0-cyanoptera
with:
duckdb_version: v1.5.2
ci_tools_version: v1.5.1
duckdb_version: v2.0.0
ci_tools_version: v2.0-cyanoptera
extension_name: mpduck
format_checks: 'format;tidy'
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -6,3 +6,4 @@ duckdb_unittest_tempdir/
testext
test/python/__pycache__/
.Rhistory
.cache/
23 changes: 9 additions & 14 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,12 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co

## What this is

`mpduck` is a DuckDB extension for reading and writing **model point files (mpfiles)** produced by the FIS Prophet actuarial application. These files use `.rpt` or `.prn` extensions and are CSV-like, but use a first-column indicator to distinguish between header rows, data rows, and schema definition rows — which standard CSV readers cannot handle.

The current scalar functions (`mpduck`, `mpduck_openssl_version`) are placeholders from the extension template. The real functionality will be table functions or COPY support for reading/writing `.rpt`/`.prn` files.
`mpduck` is a DuckDB extension for reading and writing **model point files (mpfiles)** produced by the FIS Prophet actuarial application. These files use `.rpt`, `.prn` or `.fac` extensions and are CSV-like, but use a first-column indicator to distinguish between header rows (`!`), data rows (`*`), comment rows (`&`) and schema rows (`VARIABLE_TYPES`) — which standard CSV readers cannot handle.

## Build commands

```bash
git submodule update --init --recursive # first time only
make # Build release
make test # Run SQL logic tests (release)
make test_debug # Run SQL logic tests (debug)
Expand All @@ -21,29 +20,25 @@ Build outputs:
- `./build/release/test/unittest` — Test runner with extension linked
- `./build/release/extension/mpduck/mpduck.duckdb_extension` — Loadable extension binary

To run the extension interactively:
```bash
./build/release/duckdb
```

To run a single test:
```bash
./build/release/test/unittest "test/sql/mpduck.test"
```

## Architecture

The extension follows the standard DuckDB extension pattern:
The extension is built against DuckDB's internal C++ API (not the stable C API). One source tree builds against both DuckDB 1.5.x and 2.0: the API differences (`Identifier` vs `string` for SQL names, `ConstantExpression::FromValue`, `StarExpression::ExcludeListMutable()`, `Vector::Flatten()`) live in **`src/include/mpduck_compat.hpp`**, selected by whether `duckdb/common/enums/identifier_case_mode.hpp` exists (2.0 only; `identifier.hpp` also exists in 1.5). Route any new API-sensitive call through it rather than adding `#if`s elsewhere.

- **`src/include/mpduck_extension.hpp`** — Declares `MpduckExtension` inheriting from `Extension`, implementing `Load()`, `Name()`, and `Version()`.
- **`src/mpduck_extension.cpp`** — Registers scalar functions in `Load()` using `UnaryExecutor` and `StringVector::AddString()` for result strings. Exports the C entry point via `DUCKDB_CPP_EXTENSION_ENTRY`.
- **`src/mpduck_extension.cpp`** — `LoadInternal()` registers everything: the `MPFileSystem` sub-system, the `read_mpfile` table function (VARCHAR and LIST(VARCHAR) overloads, both via `bind_replace`), a replacement scan for `FROM 'x.rpt'`, and the COPY function. Reading works by building a parser AST that rewrites to `SELECT * EXCLUDE ("!") FROM read_csv_auto([...], dtypes=..., filename=...)`. `union_by_name` defaults to false like `read_csv`: only the first file with data is parsed for its schema and the rest are opened lazily. With `union_by_name=true` every file's schema is parsed, empty files are dropped, schemas are merged (`MergeSchemas`) and `union_by_name=true` is forwarded (`ReadMPFileMultiRef`). `ignore_errors` (default false) is forwarded to `read_csv_auto` unchanged. `skip_empty_files` (default true) drops zero-byte files from a glob or list at bind time (`IsZeroByteMPFile`, one open and size lookup per file) in both modes; it is not forwarded and does not apply to a single literal path. All options travel in `MPReadOptions`, together with the dialect (`delim`/`sep`, `quote`, `escape` defaulting to `,` `"` `"`, always forwarded so the CSV sniffer never has to search for them, which is what makes wide files and globs fast; `new_line` is forwarded only when the user sets it, because pinning it gains nothing measurable and would break LF files). All are overridable with `read_csv`'s parameter names. Only `,` is valid for `delim` because `MPFileSystem` rebuilds the `!` rows on commas.
- **`src/mpfile_filesystem.cpp` / `src/include/mpfile_filesystem.hpp`** — `MPFileSystem` claims `.rpt`/`.prn`/`.fac` paths (`IsManuallySet` gives it priority) and serves a filtered stream: a normalised in-memory header built from the `!` rows, followed by the raw `*` data block, with any footer cut off (`ScanPreDataSection`, `FindDataEnd`). A thread-local bypass flag (`ScopedMPFSBypass`) stops it from intercepting its own raw opens. Also holds `ParseMPFileSchema` (header + `VARIABLE_TYPES` → DuckDB types) and `MergeSchemas` (type widening across multiple files).
- **`src/write_mpfile.cpp`** — `COPY ... TO 'x.rpt'` / `(FORMAT mpfile)` with `KV_METADATA` and `INCLUDE_TYPES` options. Registered under the names `mpfile`, `rpt`, `prn`, `fac`.
- **`extension_config.cmake`** — Registers the extension with DuckDB's build system and enables test loading.
- **`CMakeLists.txt`** — Links OpenSSL (SSL + Crypto) to both the static and loadable extension targets. OpenSSL is sourced via vcpkg (`vcpkg.json`).
- **`CMakeLists.txt`** — Builds the static and loadable extension targets. No external dependencies (`vcpkg.json` has none).

## Testing

Tests use DuckDB's SQLLogicTest format (`.test` files in `test/sql/`). Each test file uses `require mpduck` to load the extension before testing functions.
Tests use DuckDB's SQLLogicTest format (`test/sql/mpduck.test`, fixtures in `test/data/`). The test file uses `require mpduck` to load the extension. The fixtures depend on their exact line endings (most are CRLF, `lf.rpt` and `schema.fac` are LF), so `.gitattributes` marks `test/data/**` and the mpfile extensions `-text`: keep it that way so a Windows checkout with `core.autocrlf` cannot rewrite them.

## Updating DuckDB version

See `docs/UPDATING.md` — involves bumping the `duckdb` and `extension-ci-tools` submodules and updating version references in GitHub workflow files.
See `docs/UPDATING.md` — involves bumping the `duckdb` and `extension-ci-tools` submodules and updating version references in `.github/workflows/MainDistributionPipeline.yml`.
6 changes: 2 additions & 4 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,8 @@ set(LOADABLE_EXTENSION_NAME ${TARGET_NAME}_loadable_extension)
project(${TARGET_NAME})
include_directories(src/include)

set(EXTENSION_SOURCES
src/mpduck_extension.cpp
src/mpfile_filesystem.cpp
src/write_mpfile.cpp)
set(EXTENSION_SOURCES src/mpduck_extension.cpp src/mpfile_filesystem.cpp
src/write_mpfile.cpp)

build_static_extension(${TARGET_NAME} ${EXTENSION_SOURCES})
build_loadable_extension(${TARGET_NAME} " " ${EXTENSION_SOURCES})
Expand Down
121 changes: 121 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,12 @@ INSTALL mpduck FROM community;
LOAD mpduck;
```

## Versions

mpduck follows [semantic versioning](https://semver.org). The mpduck version is independent of the DuckDB version:
the same source builds against DuckDB 2.0 and DuckDB 1.5.x (CI builds v2.0.0 and v1.5.6), and an extension binary
has to match the DuckDB version it is loaded into.

## Reading files

Reference a `.rpt`, `.prn`, or `.fac` file directly in a `FROM` clause:
Expand All @@ -31,6 +37,121 @@ SELECT * FROM read_mpfile(['mp_2024.rpt', 'mp_2025.rpt']);
SELECT * FROM read_mpfile(['region_a*.rpt', 'region_b*.rpt']);
```

### union_by_name

> **Breaking change in mpduck 1.0.** Versions before 1.0 always combined multiple files as if
> `union_by_name = true`. From 1.0, `read_mpfile` takes a `union_by_name` parameter that **defaults to `false`**,
> matching `read_csv`, so reads that relied on the old behaviour now need `union_by_name = true`.
> See [Upgrading to mpduck 1.0](#upgrading-to-mpduck-10).

Like `read_csv`, `read_mpfile` takes `union_by_name`, which defaults to `false`.

- **`false` (default):** the files are expected to have the same columns. The column types come from the
`VARIABLE_TYPES` row of the first file that has data, and the remaining files are opened lazily by the scan, so
the cost of binding does not grow with the number of files. This matters most on remote storage such as S3.
Files with different columns fail with an error. Zero-byte files are skipped (see
[skip_empty_files](#skip_empty_files)), but a non-empty file with no header row is not.
- **`true`:** every file's header and `VARIABLE_TYPES` rows are read up front. Files with no data are skipped,
columns are matched by name (missing columns become `NULL`), and types that differ between files are widened
(`SMALLINT` < `INTEGER` < `DOUBLE`, anything else becomes `VARCHAR`). This costs an extra open and read per file.

In both modes a file that cannot be opened (missing, permission denied, network or credential errors) is an
error; it is never silently skipped.

```sql
SELECT * FROM read_mpfile('data/*.rpt', union_by_name = true);
```

`FROM 'data/*.rpt'` behaves like the default (`union_by_name = false`).

#### Upgrading to mpduck 1.0

When moving from a version before 1.0, add `union_by_name = true` if a multi-file read (a glob, or a list of files) does any of these:

| Your files | Before 1.0 | 1.0, by default | To keep the old behaviour |
|---|---|---|---|
| have different columns | combined by name, missing columns `NULL` | error: `COLUMN_TYPES error: Columns with names ... do not exist in the CSV File` | `union_by_name = true` |
| have the same columns with different types (e.g. `SMALLINT` in one, `INTEGER` in another) | types widened | types taken from the first file that has data (a value that does not fit is an error) | `union_by_name = true` |
| include a zero-byte file | file skipped | file skipped (`skip_empty_files`, on by default) | nothing to change |
| include a non-empty file with no header row | file skipped | error | `union_by_name = true` |
| are read with `FROM 'data/*.rpt'` | combined by name | same as `read_mpfile` with the default | use `read_mpfile('data/*.rpt', union_by_name = true)` |

Reads of a single file, and of several files that share the same columns and types, are unaffected.

Two further changes in 1.0 apply in both modes:

- An unreadable file (for example a missing path, or a permission or credentials error) used to be silently dropped
from a multi-file read, returning partial results. It is now an error.
- The CSV dialect is no longer sniffed. `read_mpfile` now reads with `delim = ','`, `quote = '"'` and
`escape = '"'`, which is faster (see [delim, quote, escape and new_line](#delim-quote-escape-and-new_line)). A file
that uses a different quote or escape character, which the sniffer used to detect, needs `quote` and `escape` passed
explicitly. Line endings are still detected.

### ignore_errors

Like `read_csv`, `read_mpfile` takes `ignore_errors`, which defaults to `false`. When
it is `true`, rows that fail to parse (for example a non-numeric value in a `S`, `I` or `N` column) are skipped
instead of raising an error.

```sql
SELECT * FROM read_mpfile('model_points.rpt', ignore_errors = true);
```

It behaves exactly as it does in `read_csv`: it skips rows, not files, so a file that cannot be opened is still an
error, and only the columns a query actually reads are converted, so a bad value in a column the query does not
use does not drop the row.

### delim, quote, escape and new_line

`read_mpfile` passes the CSV dialect of an mpfile to `read_csv` instead of leaving it to be sniffed. The defaults are
the standard CSV conventions (RFC 4180), which is also what `COPY ... TO` writes. Each can be overridden with the
parameter of the same name that `read_csv` takes:

| Parameter | Default | Notes |
|---|---|---|
| `delim` (alias `sep`) | `','` | Only `','` is accepted, see below. Give `delim` or `sep`, not both. |
| `quote` | `'"'` | Quoting is optional: unquoted fields read as they are. `quote = ''` turns quote handling off. |
| `escape` | `'"'` | A quote inside a quoted field is written doubled. |
| `new_line` | not set | Detected from the file, so CRLF and LF files both read without options. |

```sql
-- Single-quoted fields, as in 'O''Brien'
SELECT * FROM read_mpfile('model_points.rpt', quote = '''', escape = '''');

-- Force a line ending (the text \r\n, \n or \r, as in read_csv, not the control characters)
SELECT * FROM read_mpfile('lf_endings.rpt', new_line = '\n');
```

A `new_line` that does not match the file is an error. A NULL value leaves the default in place. The settings apply to
every file in a glob or list, and `FROM 'file.rpt'` always uses the defaults, since it takes no parameters.

`delim` and `sep` are accepted for parity with `read_csv`, but only `','` is valid: the header (`!`) rows are split and
rebuilt on commas, so any other delimiter is rejected with an error.

Fixing the dialect makes reads faster, most of all for very wide files and for globs of many files, because the CSV
sniffer no longer has to search for it. Measurements are in [`bench/RESULTS.md`](bench/RESULTS.md).

### skip_empty_files

`read_mpfile` takes `skip_empty_files`, which defaults to `true`. When it is `true`,
zero-byte files in a glob or a list of files are dropped before anything is read, so one empty file does not fail
the whole read. This applies with `union_by_name` either `false` or `true`, and to `FROM 'data/*.rpt'`.

```sql
-- Fail on empty files instead of skipping them
SELECT * FROM read_mpfile('data/*.rpt', skip_empty_files = false);
```

- It costs one open and size lookup per file at bind time, with no reading. On remote storage that is one extra
request per file, so set `skip_empty_files = false` if you read a very large number of files you know are not empty.
- It only covers files of exactly zero bytes. A non-empty file with no header row is still an error with the default
`union_by_name = false`, and is skipped with `union_by_name = true`.
- A single literal path, `read_mpfile('file.rpt')`, is never filtered.
- With `skip_empty_files = false` and `union_by_name = false`, an empty file fails the read with
`Columns with names ... do not exist in the CSV File`. With `union_by_name = true` it is still skipped, because
every file's header is checked.
- If every matched file is empty, the read fails with a `no files ... matched` error rather than returning no rows.

## Writing files

```sql
Expand Down
Loading
Loading