Skip to content

Harden 0.4.0: fix the [bracket] DDL crash, make the CLI scriptable, add CI - #5

Merged
dridk merged 8 commits into
mainfrom
dev
Aug 30, 2026
Merged

Harden 0.4.0: fix the [bracket] DDL crash, make the CLI scriptable, add CI#5
dridk merged 8 commits into
mainfrom
dev

Conversation

@dridk

@dridk dridk commented Aug 30, 2026

Copy link
Copy Markdown
Owner

Why

python -m sqlite2duckdb chinook.db out.db crashed with Parser Error: syntax error at or near "[". Chasing that down surfaced a handful of other problems: the CLI could not run unattended, the Python floor was never applied, and there was no CI to catch any of it.

The crash

duckdb replays sqlite's index DDL verbatim through its own parser, which rejects the [bracket] quoting that chinook and MS Access exports use:

CREATE INDEX [IFK_AlbumArtistId] ON "albums" ([ArtistId])

The attached catalog is parsed on bind, so both the SCHEMA and the DATA halves of COPY FROM DATABASE fail, and the sqlite extension exposes no setting to turn index parsing off. duckdb_indexes() on the attached database raises the same error.

So COPY FROM DATABASE is gone. Each table is now recreated from the DDL duckdb derives for the attached catalog (duckdb_tables().sql), which always parses and still carries the primary keys and NOT NULL constraints that a CREATE TABLE AS SELECT would lose. The indexes are then read back from sqlite_master and their quoting translated.

This is a regression fix: the pre-0bc556a loop never touched indexes, so chinook worked in 0.3.x.

Also in here

Bugs. requres-python was a typo, so the >=3.8 floor never reached the wheel metadata. importlib.metadata.version() raised PackageNotFoundError on an uninstalled checkout. A failed conversion left a half-written .duckdb behind, which then tripped the "already exists" guard on the next run. Path arguments crashed. db_name came from an unfiltered duckdb_databases row and was interpolated without escaping.

API. sqlite_to_duckdb(src, dst, *, overwrite=False) returns a ConversionResult, raises FileNotFoundError / FileExistsError instead of bare Exception, accepts PathLike, and reports through logging rather than printing to stdout.

CLI. --force, --quiet, --verbose. The overwrite prompt now only appears on a tty; it used to hang or raise EOFError under CI, in a pipe, or under uvx ... < /dev/null.

Packaging. Everything moves to uv: dev dependency group (faker was imported by the tests but declared nowhere, so a fresh clone could not run them), committed uv.lock, requirements.txt dropped. The sdist gets an allow list, taking it from 592K back to 13K.

CI. ruff and pytest on 3.9 through 3.13, plus publish-on-tag through PyPI trusted publishing.

Issues

Closes #3 — that traceback died in SHOW tables on a northwind export, the same [bracket] cause. There is now a regression test for it.

Refs #1 — the reporter's exact information_schema.table_constraints query returns 12 rows on a converted chinook instead of none. UNIQUE, FOREIGN KEY and CHECK are still not copied, so this may or may not be enough to close it.

Refs #2COPY FROM DATABASE was adopted in 0bc556a and is removed again here, for the reason above. The simplification still stands: the USE / SHOW tables dance the issue pointed at is long gone.

Verification

8 tests to 32, including value fidelity rather than row counts alone, the error paths, the CLI end to end, and five integration tests against the real chinook now shipped in examples/. They compare against the sqlite source rather than hardcoding: same table set, same row counts, same values, every named index present.

CI is green on all five Python versions.

🤖 Generated with Claude Code

https://claude.ai/code/session_01XtZtVXPQqXvdHLQ98bT2gF

sacha and others added 8 commits August 30, 2026 15:25
The manual loop built `CREATE TABLE {table} AS SELECT * FROM __other.{table}`
without quoting the identifiers, so any table needing quotes ("my table",
"order") aborted the conversion. It also relied on `SHOW tables`, which
enumerates views too: the sqlite extension re-parses each view's SQL with the
duckdb parser, and MS Access style `[bracket]` quoting is a syntax error there.

`COPY FROM DATABASE` copies tables, data, constraints and indexes in one
statement, and listing tables through `duckdb_tables` skips views entirely.
`ATTACH` now states `(TYPE SQLITE, READ_ONLY)` rather than relying on magic
byte detection, which fails on an empty sqlite file.

Copying indexes needs duckdb >= 1.1.0 (1.0.0 raises NotImplementedException),
so the dependency floor moves up. Views are still not converted.

Fixes #2
Fixes #3

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XtZtVXPQqXvdHLQ98bT2gF
`uvx sqlite2duckdb source.db target.db` runs the tool without installing it.
pip and `uv tool install` stay documented for a persistent install.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XtZtVXPQqXvdHLQ98bT2gF
Ships the COPY FROM DATABASE conversion: quoted identifiers and [bracket]
quoted views now convert, constraints and indexes are preserved, and duckdb
>= 1.1.0 is required.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XtZtVXPQqXvdHLQ98bT2gF
Attaching a sqlite database only exposes PRIMARY KEY and NOT NULL to duckdb;
UNIQUE, FOREIGN KEY and CHECK are dropped at the ATTACH layer, before
COPY FROM DATABASE runs. Views are not copied either. Claiming "relation and
constraint" as done was too broad.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XtZtVXPQqXvdHLQ98bT2gF
chinook is the canonical sqlite demo database, and it quotes its DDL with
[brackets], which duckdb's own parser rejects. Shipping it gives the test
suite a real database that nobody wrote for the tests.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XtZtVXPQqXvdHLQ98bT2gF
pyproject declared `requres-python` instead of `requires-python`, so the floor
was silently dropped from the wheel metadata and pip would install on any
version. Fix the typo and raise it to 3.9, duckdb 1.1's own floor.

Declare pytest and faker as a dev dependency group. tests/utils.py has always
imported faker but nothing listed it, so a fresh clone could not run the suite.
Drop requirements.txt, a stale duplicate of dependencies missing the >=1.1.0
floor that nothing consumed.

Restrict the sdist to an allow list. Hatchling ships everything git does not
ignore, which had put the 864K example database in it: 592K down to 13K.

Add GitHub Actions running ruff and pytest on 3.9 through 3.13, and publishing
on tags through PyPI trusted publishing rather than a manual twine upload.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XtZtVXPQqXvdHLQ98bT2gF
duckdb replays sqlite's index DDL verbatim through its own parser, which
rejects the [bracket] quoting that chinook.db and MS Access exports use, so
`python -m sqlite2duckdb chinook.db out.db` died with

    Parser Error: syntax error at or near "["

The attached catalog is parsed on bind, so both the SCHEMA and the DATA halves
of COPY FROM DATABASE fail, and the sqlite extension exposes no setting to turn
index parsing off.

Recreate each table from the DDL duckdb derives for the attached catalog
instead. Unlike sqlite's own it always parses, and it still carries the primary
keys and the NOT NULL constraints that a CREATE TABLE AS SELECT would lose.
Then read the indexes back from sqlite_master and translate their quoting.

Along the way, make the function usable as a library: raise FileNotFoundError
and FileExistsError rather than a bare Exception, return a ConversionResult,
report progress through logging instead of printing to stdout, accept PathLike,
and delete a partially written target instead of leaving one behind that trips
the "already exists" guard on the next run.

The CLI grows --force, --quiet and --verbose, and only prompts when stdin is a
tty. It used to hang or raise EOFError under CI, in a pipe, or under
`uvx ... < /dev/null`.

Tests go from 8 to 32: value fidelity rather than row counts alone, the error
paths, the CLI end to end, the bracket quoted index, and chinook itself.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XtZtVXPQqXvdHLQ98bT2gF
The overwrite prompt, the duckdb >=1.1.0 requirement and the supported Python
versions were undocumented, and views were only mentioned inside the
constraints bullet despite being dropped silently.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XtZtVXPQqXvdHLQ98bT2gF
@dridk
dridk merged commit afa6d01 into main Aug 30, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Get Syntax Error On Command - syntax error at or near "["

1 participant