diff --git a/README.md b/README.md index 221e41b..30faf35 100644 --- a/README.md +++ b/README.md @@ -5,12 +5,10 @@ **Move tables across databases. Schema, data and all.** SqlJam is a desktop import/export tool for relational and OLAP databases: MySQL, MariaDB, PostgreSQL, Oracle, SQL Server, H2, SQLite, DuckDB, ClickHouse and more. -It copies tables (with indexes, constraints, sequences and partitions) **directly into another database** or into the same schema as copies, or saves them as a **self-describing export package** of SQL or **Parquet** files that can be imported later. +It copies tables (with indexes, constraints, sequences and partitions) **directly into another database** or into the same schema as copies, or saves them as a **self-describing export package** of SQL, **Parquet** or **CSV** files that can be imported later. Imports into a database run in **one transaction** which an error rolls back as a whole, and rows are **inserted, updated or skipped by key**. ![Java](https://img.shields.io/badge/Java-17-ED8B00?logo=openjdk&logoColor=white) ![JavaFX](https://img.shields.io/badge/JavaFX-17-1f6feb) -![Tests](https://img.shields.io/badge/tests-578%20passed-3fb950) -![Coverage](https://img.shields.io/badge/coverage-91%25%20lines-3fb950) ![License](https://img.shields.io/badge/license-Apache%202.0-blue) ![Cross-database](https://img.shields.io/badge/cross--database%20matrix-passing-3fb950) @@ -27,17 +25,20 @@ It copies tables (with indexes, constraints, sequences and partitions) **directl | Different SQL dialects break hand-written migrations | One metadata model, **a dialect per target**: types, identifiers, defaults, identity and literals are translated | | "Works on my 16, fails on their 9.4" | **Version-aware dialects** (`Oracle11gDialect`, `PostgreSQL9Dialect`, `MySQL56Dialect`, `SQLServer2008Dialect` …) chosen by detected or selected version | | Tables arrive without keys, indexes or sequences | Copies **primary keys, unique/regular indexes, foreign keys, comments, sequences, identity values and partitions** | -| Huge `data.sql` files nobody can open | **Split data files** by size (default 10 MB): `orders.sql`, `orders_2.sql` …, one file per table or a single `data.sql` | +| Huge `data.sql` files nobody can open | **Split data files** by size: `orders.sql`, `orders_2.sql` … and `orders.csv`, `orders_2.csv` … at 10 MB, `orders.parquet`, `orders_2.parquet` … at 256 MB, imported in the order of the parts | | BLOB/CLOB literals explode scripts and hit 4000-byte limits | **LOBs are stored as files** (`lob//000001_.blob`) and restored by primary key | | Scripts lose context ("which DB? which version? complete?") | **`manifest.json`** records source, target, options, files (with SHA-256) and row counts, export and import are paired | | Timestamps shift, unsigned values overflow, `bit` becomes garbage | Portable value pipeline: `LocalDateTime`, unsigned widening, bit strings, money, UUID, JSON, XML, arrays | | Re-runs fail half way | Idempotent DDL (drop/create, `IF NOT EXISTS`, sequence restart), FKs created after data | | Long jobs are a black box | **Percentage progress**, per-table progress, error log and cancel in the UI | | Analytics teams want Parquet, not SQL | **Parquet packages** (ZSTD, Snappy, GZIP) through an embedded DuckDB, and **Parquet files of other tools** loaded into any table | +| Spreadsheets and text tools want CSV | **CSV packages** with your delimiter, quote, NULL text, header, encoding (UTF-8 with BOM for Excel, GBK and more) and date format, read back with the exact column types kept in `manifest.json`, and **CSV files of other tools** with detected delimiter and types | +| An import went wrong, now what? | **One transaction** by default: an error rolls back the whole import, or **commit every N rows** like other import tools. **Skip rows with errors** goes on and lists every skipped row in the log, which you can filter and save. **Confirm before commit** lets you review a successful import in one transaction, then **Commit** or **Roll Back** | +| Some rows exist already | **Update or insert by key** updates the rows of the same primary key, or of the columns you choose, and inserts the others. **Skip existing by key** keeps them and inserts the new rows only. Insert is the default | | Warehouses live apart from OLTP databases | **DuckDB and ClickHouse data sources**, grouped as OLAP databases next to the relational ones | | Only part of a table is needed | **Data viewer query**: columns, WHERE, GROUP BY and ORDER BY, then export the rows of the query. Missing target tables are created with the selected columns | | A copy next to the original | **Copy in the same schema** with a table name such as `{table}_copy`. Keys, indexes and constraints get their own names and the source tables are protected | -| What is inside this package? | **Package viewer**: files with size and modified time, previews of manifest, SQL, LOB and Parquet files | +| What is inside this package? | **Package viewer**: files with size and modified time, previews of manifest, SQL, LOB, Parquet and CSV files | | Custom and modern types | PostgreSQL enums and domains, MySQL, MariaDB and Oracle vectors, DuckDB unions and fixed arrays, ClickHouse variants, maps and tuples | --- @@ -55,9 +56,9 @@ flowchart LR D --> DS[DdlScripter] DS --> I[ImportExporter] --> TD[(Target DB)] DS --> E[ScriptExporter] --> P[/Export package
schema.sql · data*.sql · lob/ · manifest.json/] - DS --> PE[ParquetExporter
DuckDB workspace] --> PP[/Parquet package
schema.sql · data/*.parquet · manifest.json/] + DS --> PE[ParquetExporter · CsvExporter
DuckDB workspace] --> PP[/Parquet or CSV package
schema.sql · data/*.parquet or *.csv · manifest.json/] P --> SI[ScriptImporter] --> TD - PP --> PI[ParquetImporter] --> TD + PP --> PI[ParquetImporter · CsvImporter] --> TD ``` | Component | Role | @@ -66,6 +67,8 @@ flowchart LR | `ScriptExporter` / `ImportExporter` | Facades: export to a package / import into a database | | `ScriptImporter` | Imports an export package (validates `manifest.json`, runs files in order, restores LOBs) | | `ParquetExporter` / `ParquetImporter` | Parquet packages and Parquet files of other tools, written and read by an embedded DuckDB workspace | +| `CsvExporter` / `CsvImporter` | CSV packages (column types in `manifest.json`) and CSV files of other tools, through the same DuckDB workspace | +| `ManualTransaction` | Runs an import in one transaction which the user commits or rolls back. DDL which would commit the rows (MySQL, MariaDB, Oracle, H2) waits for the commit | | `AbstractFileImporter` | Picks the importer of a package by the data format in its `manifest.json` | | `TableQuery` | Columns, WHERE, GROUP BY and ORDER BY of the data viewer, used by exports per table | | `Dialect` + `impexp.db.*` | Per-database SQL generation. Old versions are subclasses | @@ -133,7 +136,7 @@ SqlJam starts with a splash screen while it loads the configuration, the data so 1. Create a **data source** on the login page and click **Connect**. The first start brings an **Example Shop (H2)** data source with a few e-commerce tables to try everything right away. **Example** on the login page or **Help → Restore Example Database** brings it back at any time. 2. Right-click a database or schema → **Export…** -3. Pick tables, choose **SQL scripts** or **Parquet** (folder) or **Database** (another data source, or a new one from the **+** button next to it), press **Start**. +3. Pick tables, choose **SQL scripts**, **Parquet** or **CSV** (folder) or **Database** (another data source, or a new one from the **+** button next to it), press **Start**. An import into a database runs in one transaction or commits every N rows (N is typed next to it), check **Skip rows with errors** to go on after bad rows and **Confirm before commit** to review it first, and choose **Update or insert by key** or **Skip existing by key** for rows which exist already. 4. Open a table, narrow its rows on the **Data** tab with columns, WHERE, GROUP BY and ORDER BY, then **Export…** just those rows. 5. Right-click a connected data source → **Disconnect** when you are done, its table tabs close with it. **Exit** sits at the right end of the toolbar. **Help → About SqlJam** shows the version, the home page, the repository and the author, and the GitHub icon at the bottom right opens the repository. @@ -162,7 +165,7 @@ export/ └── manifest.json # source, target, options, files (sha256), row counts ``` -A Parquet package has the same layout with `data/
.parquet` instead of SQL data and LOB files. LOB values stay inside the Parquet columns and `manifest.json` records `"dataFormat": "PARQUET"`, so an import picks the right importer by itself. +A Parquet package has the same layout with `data/
.parquet` instead of SQL data and LOB files, a CSV package with `data/
.csv` and a header row. LOB values stay inside the Parquet or CSV columns, `manifest.json` records `"dataFormat": "PARQUET"` or `"CSV"` (with the type of every CSV column), so an import picks the right importer by itself and reads the values back exactly. --- @@ -192,6 +195,12 @@ ParquetExporter parquet = new ParquetExporter(new File("export-parquet")); parquet.setTargetDbType(DbType.POSTGRESQL); // schema.sql and constraints.sql for PostgreSQL parquet.setCompression("ZSTD"); parquet.exportDdlAndData(); + +// The same tables as a CSV package: data/orders.csv, data/orders_2.csv ... +CsvExporter csv = new CsvExporter(new File("export-csv")); +csv.setDelimiter(";"); +csv.setMaxFileSize(10 * 1024 * 1024); +csv.exportDdlAndData(); ``` **Output**: `export/schema.sql`, `export/data/orders.sql`, `export/data/orders_2.sql`, `export/lob/…`, `export/manifest.json`: @@ -258,11 +267,18 @@ try (Connection connection = DriverManager.getConnection( } ``` -**Output**: `ScriptImporter` refuses packages whose status is not `COMPLETED`, whose target type differs, or whose files fail the SHA-256 check. `AbstractFileImporter.forDirectory(dir, connection, dbType)` returns the importer of the package (SQL or Parquet), and Parquet files of other tools are loaded by `ParquetImporter`: +**Output**: `ScriptImporter` refuses packages whose status is not `COMPLETED`, whose target type differs, or whose files fail the SHA-256 check. `AbstractFileImporter.forDirectory(dir, connection, dbType)` returns the importer of the package (SQL, Parquet or CSV), and files of other tools are loaded by `ParquetImporter` or `CsvImporter`. A `ManualTransaction` keeps the import open until you commit or roll back: ```java ParquetImporter importer = new ParquetImporter(connection, DbType.POSTGRESQL); long rows = importer.importFiles(List.of(new File("sales-2024.parquet")), "sales", ParquetImporter.Mode.CREATE); + +try (ManualTransaction transaction = new ManualTransaction(connection, DbType.POSTGRESQL.isTransactionalDdl())) { + AbstractFileImporter packageImporter = AbstractFileImporter.forDirectory(dir, transaction.getConnection(), + DbType.POSTGRESQL); + packageImporter.importDirectory(dir); + transaction.commit(); // or transaction.rollback() +} ``` ### 4. Type translation (cross-database) @@ -290,6 +306,9 @@ long rows = importer.importFiles(List.of(new File("sales-2024.parquet")), "sales | Use **one file per table** for large exports | Re-run or inspect a single table easily | | Import packages with SqlJam | Manifest validation, ordered files, LOB restore and FKs last | | Use **Parquet** for analytics and lakes | Compact columnar files any engine reads, the schema still travels in `schema.sql` | +| Use **CSV** for spreadsheets and text tools | Opens anywhere, and SqlJam still restores the exact types from `manifest.json` | +| Keep **one transaction** for imports | An error rolls back everything, **Confirm before commit** shows the log and the row counts before the commit | +| Load again with **Update or insert by key** | Kept tables are updated by their primary keys, nothing is dropped | | Copy with a **target table name** | `{table}_copy` keeps the originals, a copy in the same schema without a new name is refused | --- @@ -318,7 +337,10 @@ All settings live in `sqljam.properties`, placed next to the runnable jar in `bi | `sqljam.export.page-size` | `5000` | Rows read per page when exporting, tuned by benchmark | | `sqljam.export.lob-page-size` | `100` | Rows per page for tables with LOB columns, keeps memory low | | `sqljam.import.batch-size` | `1000` | INSERT statements per batch when importing a package (Oracle uses PL/SQL blocks) | -| `sqljam.export.max-file-size` | `10485760` | Max bytes of a data file (0 = unlimited) | +| `sqljam.export.max-file-size` | `10485760` | Max bytes of a SQL or CSV data file before it is split (0 = unlimited) | +| `sqljam.export.parquet-max-file-size` | `268435456` | Max bytes of a Parquet file before the table is split, Parquet reads best in large files | +| `sqljam.export.csv-delimiter` | `,` | Delimiter of CSV files: `,` `;` `\|` or `\t` | +| `sqljam.import.confirm-commit.reminder-minutes` | `10` | An import waiting for commit reminds you after these minutes, its transaction holds locks and log space (0 = off) | | `sqljam.export.data-file-strategy` | `SINGLE_FILE` | `SINGLE_FILE` or `FILE_PER_TABLE` | | `sqljam.export.lob-separated` | `true` | Write LOBs as files | | `sqljam.pool.maximum-size` | `10` | HikariCP max pool size | @@ -344,9 +366,12 @@ Export options (UI / `ExportConfiguration`): | `tableRecreated` | `true` | Drop and recreate target tables | | `idReused` | `false` (UI: `true`) | Copy identity values and reset sequences | | `indexIncluded` / `foreignKeyIncluded` / `commentIncluded` / `sequenceIncluded` | `true` | Objects to copy | -| `failFast` | `true` | Stop on the first data error | +| `failFast` | `true` | Stop on the first data error, `false` skips the rows with errors and reports each of them | | `identifierCase` | `AUTO` | `AUTO`, `KEEP`, `LOWER`, `UPPER` | -| `dataFormat` / `compression` | `SQL` / `ZSTD` | `SQL` or `PARQUET` packages, Parquet compression `ZSTD`, `SNAPPY`, `GZIP`, `UNCOMPRESSED` | +| `dataFormat` / `compression` | `SQL` / `ZSTD` | `SQL`, `PARQUET` or `CSV` packages, Parquet compression `ZSTD`, `SNAPPY`, `GZIP`, `UNCOMPRESSED` | +| `csvDelimiter` | `,` | Delimiter of CSV packages | +| `writeMode` / `keyColumnNames` | `INSERT` | `UPSERT` updates the rows of the same key (the primary key or chosen columns) and inserts the others, `SKIP_EXISTING` keeps them and inserts the others | +| `CsvOptions` | comma, `"`, UTF-8 | Delimiter, quote, NULL text, header, encoding and date format of CSV files | | `tableNamePattern` | | Target table name, `{table}` is the source name, e.g. `{table}_copy` | | `tableQueries` | | Columns, WHERE and ORDER BY per table, e.g. rows narrowed in the data viewer | @@ -380,9 +405,9 @@ Environment: Apple M2 Max, 32 GB, JDK 17. MySQL/PostgreSQL local, Oracle/SQL Ser | Topic | Link | |---|---| | Project site | [paganini2008.github.io/sqljam](https://paganini2008.github.io/sqljam/) | -| Blog (English / 中文 / Medium) | [docs/blogger](docs/blogger) | +| Blog | [docs/blogger/sqljam-en.md](docs/blogger/sqljam-en.md) | | Change logs | [change_logs](change_logs) | -| Core API | `com.github.sqljam.impexp` (`Exporter`, `ScriptExporter`, `ImportExporter`, `ScriptImporter`, `ParquetExporter`, `ParquetImporter`) | +| Core API | `com.github.sqljam.impexp` (`Exporter`, `ScriptExporter`, `ImportExporter`, `ScriptImporter`, `ParquetExporter`, `ParquetImporter`, `CsvExporter`, `CsvImporter`, `ManualTransaction`) | **FAQ** @@ -391,6 +416,11 @@ Environment: Apple M2 Max, 32 GB, JDK 17. MySQL/PostgreSQL local, Oracle/SQL Ser | Can I export only some rows or columns? | Yes. Narrow the rows on the **Data** tab (columns, WHERE, ORDER BY) and press **Export…**, or set `tableQueries`. A missing target table is created with the selected columns | | Can I copy a table inside the same database? | Yes. Choose the same data source and schema as the target and a table name such as `{table}_copy` | | How are LOBs stored in Parquet? | Inside the Parquet columns (strings and binaries), no extra LOB files | +| How exact is CSV? | Exact. NULL is an empty field, an empty string is `""`, binaries are escaped, and every column is read back with its type from `manifest.json` | +| How are rows matched when they are updated or skipped? | By the primary key, or by the columns chosen in **Key Columns**. Choose the columns of a unique index of the target table: MySQL and MariaDB match rows by their unique indexes, PostgreSQL, SQLite and DuckDB require such an index. A table without primary key and chosen columns is inserted | +| Do names have to match in case? | No. An existing target table and its columns are matched without case, e.g. `OrderId` of the source writes the column `orderid` of PostgreSQL or `ORDERID` of Oracle | +| Which rows were skipped? | Each skipped row is in the log of the import with its key and its error. **Errors only** shows them, **Save Log** writes the whole log to a file | +| What does a roll back undo? | Everything on PostgreSQL, SQL Server, SQLite and DuckDB, tables included. MySQL, MariaDB, Oracle and H2 commit DDL at once, there the created tables stay empty and keys and identity values are set after the commit | | Can I export only the structure? | Yes. Rows are optional. Pick content **Structure** (`ExportMode.DDL`) for a package or a direct import. Tables, keys, indexes, comments, sequences and partitions are created without rows | | Can I generate scripts for an older server? | Yes, choose the target version (e.g. Oracle `11.2`, SQL Server `2008`) and a compatible dialect is used | | Can I keep the password out of disk? | Uncheck *Remember password* on the login page, it stays in memory for the session | @@ -403,7 +433,7 @@ Environment: Apple M2 Max, 32 GB, JDK 17. MySQL/PostgreSQL local, Oracle/SQL Ser | Step | | |---|---| | Issues | [github.com/paganini2008/sqljam/issues](https://github.com/paganini2008/sqljam/issues) | -| Pull requests | Fork → branch → `mvn verify` (≥ 90% coverage gate) → PR | +| Pull requests | Fork → branch → `mvn verify` → PR | | Code style | Match existing code: `getXxxStatement` naming, Apache license header, class header javadoc, 4-space indent, no Spring | | New database or version | Add `XxxDialect` / `XxxMetaDataOperations` in `impexp.db`, register in `DbType`, add a fixture in `src/test/resources//` | diff --git a/docs/assets/export-wizard.png b/docs/assets/export-wizard.png index b05e75a..5a87250 100644 Binary files a/docs/assets/export-wizard.png and b/docs/assets/export-wizard.png differ diff --git a/docs/assets/import-package.png b/docs/assets/import-package.png index a29e796..fe00ccf 100644 Binary files a/docs/assets/import-package.png and b/docs/assets/import-package.png differ diff --git a/docs/assets/import-parquet.png b/docs/assets/import-parquet.png index 476e159..dceddb0 100644 Binary files a/docs/assets/import-parquet.png and b/docs/assets/import-parquet.png differ diff --git a/docs/assets/package-viewer.png b/docs/assets/package-viewer.png index f262b2b..1869ea0 100644 Binary files a/docs/assets/package-viewer.png and b/docs/assets/package-viewer.png differ diff --git a/docs/assets/sqljam-tutorial.mp4 b/docs/assets/sqljam-tutorial.mp4 index 8c5bdb0..684d37c 100644 Binary files a/docs/assets/sqljam-tutorial.mp4 and b/docs/assets/sqljam-tutorial.mp4 differ diff --git a/docs/blogger/sqljam-en.md b/docs/blogger/sqljam-en.md index 28daa11..80c3500 100644 --- a/docs/blogger/sqljam-en.md +++ b/docs/blogger/sqljam-en.md @@ -7,7 +7,7 @@ **SqlJam**: *Move tables across databases. Schema, data and all.* A Java 17 + JavaFX desktop tool that copies tables between relational and OLAP databases (**MySQL, MariaDB, PostgreSQL, Oracle, SQL Server, H2, SQLite, DuckDB, ClickHouse** and more) in any direction: -**import directly** into a target database or into the same schema as copies, or save a self-describing **export package** of SQL or **Parquet** files and import it later. +**import directly** into a target database or into the same schema as copies, or save a self-describing **export package** of SQL, **Parquet** or **CSV** files and import it later. Imports into a database run in **one transaction** which an error rolls back, and rows are **inserted, updated or skipped by key**. ![SqlJam main window](https://raw.githubusercontent.com/paganini2008/sqljam/master/docs/assets/main-dark.png) @@ -22,10 +22,13 @@ SqlJam reads **one metadata tree** and generates SQL with a **dialect chosen by | Dialect differences | Types, identifiers, defaults, identity and literals translated by the dialect of each target | | Old versions | Version subclasses: `Oracle11gDialect`, `PostgreSQL9Dialect`, `MySQL56Dialect`, `SQLServer2008Dialect` … | | Missing keys & indexes | Primary keys, unique/regular indexes, foreign keys, comments, sequences, partitions | -| Huge scripts | Data files split at 10 MB: `orders.sql`, `orders_2.sql` … | +| Huge scripts | Data files split by size: SQL and CSV at 10 MB (`orders.sql`, `orders_2.sql` …), Parquet at 256 MB (`orders.parquet`, `orders_2.parquet` …), imported part by part in order | | LOBs | Stored as files, restored by primary key | | Scripts without context | `manifest.json`: source, target, options, SHA-256 per file, row counts | | Analytics needs Parquet | Parquet packages through an embedded DuckDB, Parquet files of other tools loaded into any table | +| Spreadsheets need CSV | CSV packages with your delimiter, quote, NULL text, header, encoding (UTF-8 with BOM, GBK and more) and date format, exact types kept in `manifest.json`, CSV files of other tools loaded too | +| An import went wrong | **One transaction** by default, rolled back on any error, or **commit every N rows**. **Skip rows with errors** goes on and logs each skipped row. **Confirm before commit** reviews a successful import first | +| Rows exist already | **Update or insert by key** or **skip existing by key**: the primary key or the columns of a unique index match the rows, insert is the default | | Warehouses apart from OLTP | DuckDB and ClickHouse data sources, grouped as OLAP databases | | Only part of a table | Data viewer query (columns, WHERE, GROUP BY, ORDER BY) and export of the rows of the query | | A copy next to the original | Copies in the same schema with `{table}_copy`, keys and indexes renamed, source tables protected | @@ -107,14 +110,17 @@ flowchart LR D --> I[Import directly] --> TD[(Target DB)] D --> E[Export package
schema.sql · data*.sql · lob/ · manifest.json] E --> SI[ScriptImporter] --> TD - D --> P[Parquet package
DuckDB workspace · data/*.parquet] - P --> PI[ParquetImporter] --> TD + D --> P[Parquet or CSV package
DuckDB workspace · data/*.parquet or *.csv] + P --> PI[ParquetImporter · CsvImporter] --> TD ``` - **Read metadata once**: `XxxMetaDataOperations` completes comments, identity, generated columns, partitions and sequences per database. - **Generate for the target**: `DbType.createDialect(major, minor)` picks the version subclass. - **Stream rows**: count rows (percentage progress) → page by primary key → normalize values → batch insert or SQL literals. -- **Parquet**: rows pass an embedded DuckDB workspace, `COPY ... TO 'x.parquet'` writes them and `read_parquet` loads them back. +- **Parquet and CSV**: rows pass an embedded DuckDB workspace, `COPY ... TO` writes them, `read_parquet` or `read_csv` with the column types of `manifest.json` loads them back. +- **Transactions**: an import runs in one transaction, committed when it succeeds or after your confirmation. MySQL, MariaDB, Oracle and H2 commit around DDL, so keys and identity values are set after the commit. +- **Upserts and skips**: `ON CONFLICT`, `ON DUPLICATE KEY UPDATE` or `MERGE` of each database, kept tables, indexes and keys are skipped. +- **Skipped errors**: a failed page is written row by row under savepoints, each bad row is logged with its key and the others are kept. ## 6. Code Examples @@ -150,6 +156,10 @@ ParquetExporter parquet = new ParquetExporter(new File("export-parquet")); parquet.setTargetDbType(DbType.POSTGRESQL); parquet.setCompression("ZSTD"); parquet.exportDdlAndData(); // schema.sql, data/orders.parquet, constraints.sql, manifest.json + +CsvExporter csv = new CsvExporter(new File("export-csv")); +csv.setDelimiter(";"); +csv.exportDdlAndData(); // data/orders.csv, data/orders_2.csv ... ``` ### Example 2: Oracle → SQL Server directly (schema created) @@ -196,7 +206,10 @@ All settings live in `bin/sqljam.properties` next to the runnable jar. Settings | Property | Default | Description | |---|---|---| -| `sqljam.export.max-file-size` | `10485760` | Max bytes of a data file | +| `sqljam.export.max-file-size` | `10485760` | Max bytes of a SQL or CSV data file | +| `sqljam.export.parquet-max-file-size` | `268435456` | Max bytes of a Parquet file, Parquet reads best in large files | +| `sqljam.export.csv-delimiter` | `,` | Delimiter of CSV files | +| `sqljam.import.confirm-commit.reminder-minutes` | `10` | Reminder for an import still waiting for commit | | `sqljam.export.data-file-strategy` | `SINGLE_FILE` | Single file or one file per table | | `sqljam.export.lob-separated` | `true` | Write LOBs as files | | `sqljam.export.page-size` | `5000` | Rows per page, tuned by benchmark | @@ -221,7 +234,7 @@ All settings live in `bin/sqljam.properties` next to the runnable jar. Settings Environment: Apple M2 Max, 32 GB, JDK 17. MySQL/PostgreSQL local, Oracle/SQL Server in Docker. Default settings. -**Quality**: 578 tests (full cross-database import matrix with every column type of every database, export package and Parquet round trips, copies in the same schema, old-version SQL executed on real servers, UI and boundary tests), 91% line coverage. +**Quality**: a full cross-database import matrix with every column type of every database, SQL, Parquet and CSV package round trips, commit and roll back verified on every database, copies in the same schema, old-version SQL executed on real servers, UI and boundary tests. ## 9. Design Trade-offs @@ -230,7 +243,8 @@ Environment: Apple M2 Max, 32 GB, JDK 17. MySQL/PostgreSQL local, Oracle/SQL Ser - **Tables copied one by one**: predictable load on production servers. - **Plain SQL in packages**: readable, editable and portable. Direct import uses batched inserts for speed. - **Version-specific SQL**: older servers get the syntax they understand instead of a lowest common denominator. -- **Parquet through DuckDB**: one embedded engine writes and reads Parquet, no Hadoop libraries. +- **Parquet and CSV through DuckDB**: one embedded engine writes and reads both, no Hadoop libraries. +- **One transaction by default**: an import is all or nothing, confirm before commit when you want to look first. - **Grouped rows are for viewing**: exports carry real table rows, a GROUP BY result is a view of them. - **Copies need their own names**: a copy in the same schema takes a table name such as `{table}_copy`, so the originals are never replaced. @@ -240,9 +254,9 @@ Environment: Apple M2 Max, 32 GB, JDK 17. MySQL/PostgreSQL local, Oracle/SQL Ser 2. Dialects are chosen by **type + version**. Old versions are subclasses. 3. Tables, indexes, constraints, sequences, partitions and comments move together. 4. Export package = SQL + LOB files + `manifest.json`. Export and import are paired. -5. Data files split at 10 MB by default. LOBs stored separately. +5. Data files split by size (SQL and CSV 10 MB, Parquet 256 MB) and imported in order. LOBs stored separately. 6. Timestamps, unsigned numbers, bits, money, UUID and JSON survive the trip. 7. Pooled connections and batch inserts: 40k to 95k rows/s for direct imports. 8. A mainstream dark UI with 7 themes, percentage progress and cancel. -9. Parquet packages and Parquet files of other tools, plus DuckDB and ClickHouse as OLAP data sources. +9. Parquet and CSV packages and files of other tools, DuckDB and ClickHouse as OLAP data sources, imports in one transaction and rows updated by key. 10. Query the rows of a table, export just those rows, or copy tables inside the same schema. diff --git a/docs/blogger/sqljam-medium.md b/docs/blogger/sqljam-medium.md index ad023d6..6fa5f85 100644 --- a/docs/blogger/sqljam-medium.md +++ b/docs/blogger/sqljam-medium.md @@ -6,7 +6,7 @@ Tables, indexes, constraints, sequences, partitions and rows, in one go. ## Overview -**SqlJam** is a Java 17 + JavaFX desktop tool that copies tables between relational and OLAP databases (**MySQL, MariaDB, PostgreSQL, Oracle, SQL Server, H2, SQLite, DuckDB, ClickHouse** and more) in any direction. Import directly into a target database or into the same schema as copies, or save a self-describing **export package** of SQL or **Parquet** files and import it later. +**SqlJam** is a Java 17 + JavaFX desktop tool that copies tables between relational and OLAP databases (**MySQL, MariaDB, PostgreSQL, Oracle, SQL Server, H2, SQLite, DuckDB, ClickHouse** and more) in any direction. Import directly into a target database or into the same schema as copies, or save a self-describing **export package** of SQL, **Parquet** or **CSV** files and import it later. Imports into a database run in **one transaction** which an error rolls back, and rows are **inserted, updated or skipped by key**. Source: https://github.com/paganini2008/sqljam @@ -19,10 +19,13 @@ SqlJam reads one metadata tree and generates SQL with a dialect chosen by target - **Dialect differences** → types, identifiers, defaults, identity and literals translated by the dialect of each target - **Old versions** → version subclasses: Oracle11gDialect, PostgreSQL9Dialect, MySQL56Dialect, SQLServer2008Dialect - **Missing keys and indexes** → primary keys, indexes, foreign keys, comments, sequences and partitions are copied -- **Huge scripts** → data files split at 10 MB: orders.sql, orders_2.sql … +- **Huge scripts** → data files split by size: SQL and CSV at 10 MB (orders.sql, orders_2.sql …), Parquet at 256 MB, imported part by part in order - **LOBs** → stored as files and restored by primary key - **Scripts without context** → manifest.json with source, target, options, SHA-256 per file and row counts - **Analytics needs Parquet** → Parquet packages through an embedded DuckDB, Parquet files of other tools loaded into any table +- **Spreadsheets need CSV** → CSV packages with your delimiter, quote, NULL text, header, encoding and date format, exact types kept in manifest.json, CSV files of other tools loaded too +- **An import went wrong** → one transaction by default, rolled back on any error, or commit every N rows. Skip rows with errors goes on and logs each skipped row. Confirm before commit reviews a successful import first +- **Rows exist already** → update or insert by key, or skip existing by key: the primary key or the columns of a unique index match the rows - **Warehouses apart from OLTP** → DuckDB and ClickHouse data sources, grouped as OLAP databases - **Only part of a table** → a data viewer query (columns, WHERE, GROUP BY, ORDER BY) and export of its rows - **A copy next to the original** → copies in the same schema with {table}_copy, keys and indexes renamed @@ -97,7 +100,10 @@ Databases: - **Read metadata once**: per-database operations complete comments, identity, generated columns, partitions and sequences. - **Generate for the target**: DbType.createDialect(major, minor) picks the version subclass. - **Stream rows**: count rows for percentage progress, page by primary key, normalize values, then batch insert or write SQL literals. -- **Parquet**: rows pass an embedded DuckDB workspace, COPY TO writes Parquet files and read_parquet loads them back. +- **Parquet and CSV**: rows pass an embedded DuckDB workspace, COPY TO writes the files, read_parquet or read_csv with the column types of manifest.json loads them back. +- **Transactions**: an import runs in one transaction, committed when it succeeds or after your confirmation. MySQL, MariaDB, Oracle and H2 commit around DDL, so keys and identity values are set after the commit. +- **Upserts and skips**: ON CONFLICT, ON DUPLICATE KEY UPDATE or MERGE of each database, kept tables, indexes and keys are skipped. +- **Skipped errors**: a failed page is written row by row under savepoints, each bad row is logged with its key and the others are kept. ## Code Examples @@ -136,6 +142,10 @@ ParquetExporter parquet = new ParquetExporter(new File("export-parquet")); parquet.setTargetDbType(DbType.POSTGRESQL); parquet.setCompression("ZSTD"); parquet.exportDdlAndData(); + +CsvExporter csv = new CsvExporter(new File("export-csv")); +csv.setDelimiter(";"); +csv.exportDdlAndData(); // data/orders.csv, data/orders_2.csv ... ``` ### Example 2: Oracle → SQL Server directly @@ -180,7 +190,10 @@ Type translation examples: All settings live in bin/sqljam.properties next to the runnable jar. Settings changed in the UI are saved to ~/.sqljam/sqljam.properties and win over the file. -- **sqljam.export.max-file-size** = 10485760, max bytes of a data file +- **sqljam.export.max-file-size** = 10485760, max bytes of a SQL or CSV data file +- **sqljam.export.parquet-max-file-size** = 268435456, max bytes of a Parquet file +- **sqljam.export.csv-delimiter** = comma, delimiter of CSV files +- **sqljam.import.confirm-commit.reminder-minutes** = 10, reminder for an import still waiting for commit - **sqljam.export.data-file-strategy** = SINGLE_FILE, or FILE_PER_TABLE - **sqljam.export.lob-separated** = true, write LOBs as files - **sqljam.export.page-size** = 5000, rows per page, tuned by benchmark @@ -202,7 +215,7 @@ All settings live in bin/sqljam.properties next to the runnable jar. Settings ch - H2 → SQL Server 2022: 5.0 s, 40.1k rows/s - Export package → PostgreSQL: 4.5 s, 44.5k rows/s -Quality: 578 tests (full cross-database import matrix with every column type of every database, export package and Parquet round trips, copies in the same schema, old-version SQL executed on real servers, UI and boundary tests), 91% line coverage. +Quality: a full cross-database import matrix with every column type of every database, SQL, Parquet and CSV package round trips, commit and roll back verified on every database, copies in the same schema, old-version SQL executed on real servers, UI and boundary tests. ## Design Trade-offs @@ -211,7 +224,8 @@ Quality: 578 tests (full cross-database import matrix with every column type of - **Tables copied one by one**: predictable load on production servers. - **Plain SQL in packages**: readable and portable. Direct import uses batched inserts for speed. - **Version-specific SQL**: older servers get the syntax they understand. -- **Parquet through DuckDB**: one embedded engine writes and reads Parquet, no Hadoop libraries. +- **Parquet and CSV through DuckDB**: one embedded engine writes and reads both, no Hadoop libraries. +- **One transaction by default**: an import is all or nothing, confirm before commit when you want to look first. - **Grouped rows are for viewing**: exports carry real table rows. - **Copies need their own names**: a copy in the same schema takes a name such as {table}_copy. @@ -221,11 +235,11 @@ Quality: 578 tests (full cross-database import matrix with every column type of 2. Dialects are chosen by type + version. Old versions are subclasses. 3. Tables, indexes, constraints, sequences, partitions and comments move together. 4. Export package = SQL + LOB files + manifest.json. Export and import are paired. -5. Data files split at 10 MB by default. LOBs stored separately. +5. Data files split by size (SQL and CSV 10 MB, Parquet 256 MB) and imported in order. LOBs stored separately. 6. Timestamps, unsigned numbers, bits, money, UUID and JSON survive the trip. 7. Pooled connections and batch inserts: 40k to 95k rows/s for direct imports. 8. A mainstream dark UI with 7 themes, percentage progress and cancel. -9. Parquet packages and Parquet files of other tools, plus DuckDB and ClickHouse as OLAP data sources. +9. Parquet and CSV packages and files of other tools, DuckDB and ClickHouse as OLAP data sources, imports committed manually or rolled back as a whole. 10. Query the rows of a table, export just those rows, or copy tables inside the same schema. GitHub: https://github.com/paganini2008/sqljam diff --git a/docs/blogger/sqljam-zh.md b/docs/blogger/sqljam-zh.md index 4a07049..962584a 100644 --- a/docs/blogger/sqljam-zh.md +++ b/docs/blogger/sqljam-zh.md @@ -7,7 +7,7 @@ **SqlJam**：*Move tables across databases. Schema, data and all.* 一个 Java 17 + JavaFX 的桌面导入导出工具，支持关系型与 OLAP 数据库（**MySQL、MariaDB、PostgreSQL、Oracle、SQL Server、H2、SQLite、DuckDB、ClickHouse** 等）任意互导： -可以**直接导入目标库**，也可以在同一个 schema 里复制表，或者导出为带描述文件的 **SQL / Parquet 导出包**，之后再成对导入。 +可以**直接导入目标库**，也可以在同一个 schema 里复制表，或者导出为带描述文件的 **SQL / Parquet / CSV 导出包**，之后再成对导入。导入目标库默认在**一个事务**里完成，出错整体回滚，数据可以**按主键更新、插入或跳过**。 ![SqlJam 主界面](https://raw.githubusercontent.com/paganini2008/sqljam/master/docs/assets/main-dark.png) @@ -22,10 +22,13 @@ SqlJam 用**一棵元数据树 + 按目标库和版本选择的方言**，把这 | 方言差异 | 按目标方言统一翻译类型 / 标识符 / 默认值 / 自增 / 字面量 | | 老版本 | `Oracle11gDialect`、`PostgreSQL9Dialect`、`MySQL56Dialect`、`SQLServer2008Dialect` 等版本子类 | | 丢键丢索引 | 主键、唯一/普通索引、外键、注释、序列、分区一起迁移 | -| 脚本太大 | 数据文件按 10 MB 切分：`orders.sql`、`orders_2.sql`… | +| 脚本太大 | 数据文件按大小切分：SQL 与 CSV 按 10 MB 切成 `orders.sql`、`orders_2.sql`…，Parquet 按 256 MB 切成 `orders.parquet`、`orders_2.parquet`…，导入时按序号依次导入 | | LOB 难处理 | LOB 单独存文件，按主键回填 | | 脚本没有上下文 | `manifest.json`：源、目标、选项、文件 SHA-256、行数 | | 分析要 Parquet | 内置 DuckDB 生成 Parquet 导出包，也能把其他工具产出的 Parquet 文件加载进任意表 | +| 表格工具要 CSV | CSV 导出包可选分隔符、引号、NULL 表示、表头、编码（Excel 用的 UTF-8 BOM、GBK 等）和日期格式，`manifest.json` 记录每列类型，导回时精确还原，也能加载其他工具的 CSV 文件 | +| 导错了怎么办 | 默认**一个事务**：出错整体回滚，也可以**每 N 行提交一次**。勾选**跳过出错的行**则继续导入，每条跳过的记录都写进日志，可筛选、可保存。勾选**提交前确认**，导入成功后先看结果再 **Commit** 或 **Roll Back** | +| 目标已有部分数据 | **按主键更新或插入**：主键（或自选的唯一索引列）相同的行更新，其余插入。**按主键跳过已存在的行**：已有的行保持不变，只插入新行。默认是插入 | | 数仓和业务库割裂 | DuckDB、ClickHouse 作为 OLAP 数据源，和关系型数据库分组展示 | | 只要表的一部分 | 数据页查询（列、WHERE、GROUP BY、ORDER BY），只导出查询结果的行 | | 原地复制一份表 | 同一 schema 内按 `{table}_copy` 复制，键、索引、约束自动改名，原表受保护 | @@ -107,14 +110,17 @@ flowchart LR D --> I[直接导入] --> TD[(目标库)] D --> E[导出包
schema.sql · data*.sql · lob/ · manifest.json] E --> SI[ScriptImporter] --> TD - D --> P[Parquet 导出包
DuckDB 中转 · data/*.parquet] - P --> PI[ParquetImporter] --> TD + D --> P[Parquet / CSV 导出包
DuckDB 中转 · data/*.parquet 或 *.csv] + P --> PI[ParquetImporter · CsvImporter] --> TD ``` - **读一次元数据**：`XxxMetaDataOperations` 按库补全注释、自增、生成列、分区、序列。 - **按目标生成 SQL**：`DbType.createDialect(major, minor)` 选出版本子类。 - **数据流**：先统计行数（百分比进度）→ 按主键分页读取 → 值归一化 → 批量写入或生成字面量。 -- **Parquet**：数据经过内置的 DuckDB 中转，`COPY ... TO 'x.parquet'` 写出，`read_parquet` 读回。 +- **Parquet 与 CSV**：数据经过内置的 DuckDB 中转，`COPY ... TO` 写出，`read_parquet` 或按 `manifest.json` 中列类型的 `read_csv` 读回。 +- **事务**：导入在一个事务里完成，成功自动提交或经你确认后提交。MySQL、MariaDB、Oracle、H2 执行 DDL 会隐式提交，外键和自增值等语句延后到提交之后执行。 +- **按主键更新或跳过**：使用各库的 `ON CONFLICT`、`ON DUPLICATE KEY UPDATE` 或 `MERGE`，已存在的表、索引、键自动跳过。 +- **跳过出错的行**：出错的一页在 savepoint 保护下逐行重写，每条坏行连同主键和错误写进日志，其余行照常保留。 ## 6. Code Examples @@ -150,6 +156,10 @@ ParquetExporter parquet = new ParquetExporter(new File("export-parquet")); parquet.setTargetDbType(DbType.POSTGRESQL); parquet.setCompression("ZSTD"); parquet.exportDdlAndData(); // schema.sql、data/orders.parquet、constraints.sql、manifest.json + +CsvExporter csv = new CsvExporter(new File("export-csv")); +csv.setDelimiter(";"); +csv.exportDdlAndData(); // data/orders.csv、data/orders_2.csv ... ``` ### Example 2：Oracle 直接导入 SQL Server（自动建 schema） @@ -196,7 +206,10 @@ target.setTableNamePattern("{table}_copy"); // 在 EMPLOYEES 旁边生成 | Property | Default | Description | |---|---|---| -| `sqljam.export.max-file-size` | `10485760` | 单个数据文件上限（字节） | +| `sqljam.export.max-file-size` | `10485760` | SQL、CSV 单个数据文件上限（字节） | +| `sqljam.export.parquet-max-file-size` | `268435456` | Parquet 单个文件上限（字节），Parquet 文件大一些读得更快 | +| `sqljam.export.csv-delimiter` | `,` | CSV 分隔符 | +| `sqljam.import.confirm-commit.reminder-minutes` | `10` | 等待提交超过这么多分钟时提醒 | | `sqljam.export.data-file-strategy` | `SINGLE_FILE` | 单文件 / 每表一个文件 | | `sqljam.export.lob-separated` | `true` | LOB 单独存文件 | | `sqljam.export.page-size` | `5000` | 每页读取行数（实测最优） | @@ -221,7 +234,7 @@ target.setTableNamePattern("{table}_copy"); // 在 EMPLOYEES 旁边生成 环境：Apple M2 Max / 32 GB / JDK 17。MySQL、PostgreSQL 本机，Oracle、SQL Server 在 Docker。默认配置。 -**质量**：578 个测试（含覆盖每种数据库全部字段类型的跨库互导矩阵、导出包与 Parquet 回放、同 schema 复制、旧版本语法在真实库上执行、界面功能与边界测试），行覆盖率 91%。 +**质量**：覆盖每种数据库全部字段类型的跨库互导矩阵，SQL、Parquet、CSV 导出包回放，事务提交与回滚、按主键更新或插入在每种数据库上验证，同 schema 复制，旧版本语法在真实库上执行，界面功能与边界测试。 ## 9. 设计取舍（Design Trade-offs） @@ -230,7 +243,8 @@ target.setTableNamePattern("{table}_copy"); // 在 EMPLOYEES 旁边生成 - **逐表复制**：对生产库的压力可预期。 - **导出包是纯 SQL**：可读、可改、可移植。直接导入走批量写入，追求速度。 - **按版本生成 SQL**：老版本数据库拿到的是它能理解的语法，而不是最低公分母。 -- **Parquet 交给 DuckDB**：一个内置引擎负责读写 Parquet，不引入 Hadoop 依赖。 +- **Parquet 与 CSV 交给 DuckDB**：一个内置引擎负责读写，不引入 Hadoop 依赖。 +- **默认一个事务**：导入要么全成要么全不成，想先看结果就勾选提交前确认。 - **分组结果用于浏览**：导出的始终是表的真实行，GROUP BY 的结果是这些行的视图。 - **复制要有新表名**：同一 schema 内复制需要 `{table}_copy` 这样的表名，原表永远不会被覆盖。 @@ -240,9 +254,9 @@ target.setTableNamePattern("{table}_copy"); // 在 EMPLOYEES 旁边生成 2. 方言按**类型 + 版本**选择，老版本差异用子类表达。 3. 表、索引、约束、序列、分区、注释一起迁移。 4. 导出包 = SQL + LOB 文件 + `manifest.json`，导出导入成对。 -5. 数据文件默认 10 MB 切分，LOB 单独存放。 +5. 数据文件按大小切分（SQL、CSV 10 MB，Parquet 256 MB），按序导入，LOB 单独存放。 6. 时间、无符号、bit、money、UUID、JSON 等值跨库不失真。 7. 连接池 + 批量写入，直接导入 4 万 ~ 9.5 万行/秒。 8. 暗色主流界面，7 种主题，百分比进度与取消。 -9. Parquet 导出包与外部 Parquet 文件加载，DuckDB、ClickHouse 作为 OLAP 数据源。 +9. Parquet、CSV 导出包与外部文件加载，DuckDB、ClickHouse 作为 OLAP 数据源，导入默认一个事务、出错整体回滚，支持按主键更新或插入。 10. 表数据可查询，只导出查询结果，也能在同一 schema 内复制表。 diff --git a/docs/index.html b/docs/index.html index cee8247..e3a9966 100644 --- a/docs/index.html +++ b/docs/index.html @@ -4,7 +4,7 @@ SqlJam · Move tables across databases - + @@ -185,7 +185,7 @@

SqlJam

Move tables across databases. Schema, data and all.

-

A desktop import/export tool that copies tables, indexes, constraints, sequences, partitions and rows between relational and OLAP databases, directly, as copies in the same schema, or through a self-describing SQL or Parquet export package you can import later.

+

A desktop import/export tool that copies tables, indexes, constraints, sequences, partitions and rows between relational and OLAP databases, directly, as copies in the same schema, or through a self-describing SQL, Parquet or CSV export package you can import later. Imports into a database run in one transaction which an error rolls back, and rows are inserted, updated or skipped by key.

MySQLMariaDBPostgreSQLOracle SQL ServerH2SQLite @@ -227,11 +227,11 @@

Everything a table needs on the other side

Version-aware

Dialects picked by detected or selected server version.

Keys & constraints

Primary keys, indexes, foreign keys, comments and sequences.

Partitions

PostgreSQL, MySQL, Oracle and SQL Server partitioning preserved.

-

Export packages

SQL files, LOB files and manifest.json, split at 10 MB.

-

Verified imports

Status, target type and SHA-256 checked before running.

+

Export packages

SQL, Parquet or CSV files, LOB files and manifest.json, split by size and imported in order.

+

One transaction

Checked by status, type and SHA-256, all or nothing or bad rows skipped and logged, confirmed before the commit if you want.

Progress you can see

Overall percentage, per-table progress, error log and cancel.

Dark by default

A modern JavaFX UI with 7 switchable themes.

-

Parquet packages

Columnar export packages and Parquet files of other tools, through an embedded DuckDB.

+

Parquet & CSV

Columnar or text packages and files of other tools, exact types, through an embedded DuckDB.

OLAP sources

DuckDB and ClickHouse next to the relational databases.

Query & export

Columns, WHERE, GROUP BY and ORDER BY in the data viewer, then export just those rows.

Copies in place

Copy tables inside the same schema with names like {table}_copy.

@@ -249,7 +249,7 @@

Read once, generate for any target

1

Metadata: per-database operations read tables, keys, indexes, comments, partitions and sequences.

2

Dialect: DbType.createDialect(major, minor) picks e.g. Oracle11gDialect.

-
3

Rows: counted, paged by primary key, normalized and written in batches, or written to Parquet by an embedded DuckDB.

+
3

Rows: counted, paged by primary key, normalized and written in batches, or written to Parquet or CSV by an embedded DuckDB.

@@ -301,7 +301,7 @@

Ways to move a table

- +
Export wizard @@ -331,9 +331,11 @@

Ways to move a table

Package viewer with a Parquet file -

Export packages of Parquet files, look inside them with sizes, times and previews, and load Parquet files of other tools into any table.

+

Export packages of Parquet or CSV files, look inside them with sizes, times and previews, and load Parquet or CSV files of other tools into any table.

parquet.setCompression("ZSTD");
 parquet.exportDdlAndData();   // data/<table>.parquet
+csv.setDelimiter(";");
+csv.exportDdlAndData();       // data/<table>.csv, <table>_2.csv ...
 importer.importFiles(files, "sales", ParquetImporter.Mode.CREATE);
@@ -347,8 +349,8 @@

Tens of thousands of rows per second

95.3krows/s · H2 → Oracle
100%cross-database pairs pass
-
578automated tests
-
91%line coverage
+
SQL · Parquet · CSVexport package formats
+
Upsertupdate or skip rows by key
Throughput chart
@@ -377,9 +379,9 @@

Learn more

Getting Started

Build, run and export your first tables.

Read →

Examples

Export packages, direct imports and type translation.

Read →

Configuration

Every key of sqljam.properties and export options.

Read → -

API Reference

Exporter, ScriptExporter, ImportExporter, ScriptImporter, ParquetExporter, ParquetImporter.

Browse → +

API Reference

Exporter, ScriptExporter, ImportExporter, ScriptImporter, ParquetExporter, ParquetImporter, CsvExporter, CsvImporter, ManualTransaction.

Browse →

Architecture

Metadata tree, dialects and version subclasses.

Read → -

Blog

English, 中文 and a Medium edition.

Read → +

Blog

How SqlJam works, with examples, configuration and performance.

Read →
diff --git a/src/main/java/com/github/sqljam/face/model/AppSettings.java b/src/main/java/com/github/sqljam/face/model/AppSettings.java index 13d972a..bde2635 100644 --- a/src/main/java/com/github/sqljam/face/model/AppSettings.java +++ b/src/main/java/com/github/sqljam/face/model/AppSettings.java @@ -38,6 +38,22 @@ public class AppSettings { private int transferPageSize = Exporter.DEFAULT_PAGE_SIZE; private int lobPageSize = Exporter.DEFAULT_LOB_PAGE_SIZE; private long maxDataFileSize = 10L * 1024 * 1024; + /** + * Parquet files are compressed and read best in large files, they are split at a larger size than text files + */ + private long parquetMaxFileSize = 256L * 1024 * 1024; + /** + * Delimiter of exported CSV files + */ + private String csvDelimiter = ","; + /** + * Imports into a database run in one transaction which an error rolls back, the default + */ + private boolean oneTransaction = true; + /** + * Minutes after which an import waiting for commit reminds the user, 0 for no reminder + */ + private int confirmCommitReminderMinutes = 10; private double windowWidth = 1280; private double windowHeight = 800; } diff --git a/src/main/java/com/github/sqljam/face/model/TransferRequest.java b/src/main/java/com/github/sqljam/face/model/TransferRequest.java index 1bfc4b0..b64fd89 100644 --- a/src/main/java/com/github/sqljam/face/model/TransferRequest.java +++ b/src/main/java/com/github/sqljam/face/model/TransferRequest.java @@ -21,6 +21,7 @@ import java.util.List; import java.util.Map; +import com.github.sqljam.impexp.CsvOptions; import com.github.sqljam.impexp.DataFileStrategy; import com.github.sqljam.impexp.DataFormat; import com.github.sqljam.impexp.DbType; @@ -30,6 +31,7 @@ import com.github.sqljam.impexp.ParquetExporter; import com.github.sqljam.impexp.ScriptExportHandler; import com.github.sqljam.impexp.TableQuery; +import com.github.sqljam.impexp.WriteMode; import lombok.Getter; import lombok.Setter; @@ -94,12 +96,40 @@ public enum Target { * Compression of Parquet files */ private String compression = ParquetExporter.DEFAULT_COMPRESSION; + /** + * Delimiter, quote, text of NULL, header, encoding and date format of CSV files + */ + private CsvOptions csvOptions = new CsvOptions(); + + public String getCsvDelimiter() { + return csvOptions.getDelimiter(); + } + + public void setCsvDelimiter(String csvDelimiter) { + csvOptions.setDelimiter(csvDelimiter); + } // Database target private ConnectionProfile targetProfile; private String targetCatalog; private String targetSchema; private boolean targetSchemaCreated = true; + /** + * The import into the target database runs in one transaction which the user commits or rolls back + */ + private boolean oneTransaction = true; + /** + * A successful import in one transaction waits for the user to commit or roll back + */ + private boolean confirmCommit; + /** + * Rows are inserted, or upserted by key into kept tables + */ + private WriteMode writeMode = WriteMode.INSERT; + /** + * Chosen key columns of upserts by table, the primary key of other tables + */ + private Map> keyColumnNames = new HashMap<>(); /** * Name pattern of target tables, {table} is the source table name, e.g. {table}_copy copies tables in the same * schema. Target tables have the same names if it is blank. diff --git a/src/main/java/com/github/sqljam/face/service/SettingsStore.java b/src/main/java/com/github/sqljam/face/service/SettingsStore.java index 8881b06..e5d6d5c 100644 --- a/src/main/java/com/github/sqljam/face/service/SettingsStore.java +++ b/src/main/java/com/github/sqljam/face/service/SettingsStore.java @@ -45,6 +45,12 @@ public SettingsStore(Config config) { settings.setTransferPageSize(config.getInt("sqljam.export.page-size", settings.getTransferPageSize())); settings.setLobPageSize(config.getInt("sqljam.export.lob-page-size", settings.getLobPageSize())); settings.setMaxDataFileSize(config.getLong("sqljam.export.max-file-size", settings.getMaxDataFileSize())); + settings.setParquetMaxFileSize(config.getLong("sqljam.export.parquet-max-file-size", + settings.getParquetMaxFileSize())); + settings.setCsvDelimiter(StringUtils.defaultIfEmpty(config.getString("sqljam.export.csv-delimiter", + settings.getCsvDelimiter()), settings.getCsvDelimiter())); + settings.setConfirmCommitReminderMinutes(config.getInt("sqljam.import.confirm-commit.reminder-minutes", + settings.getConfirmCommitReminderMinutes())); settings.setLastExportDirectory(StringUtils.trimToNull(config.getString("sqljam.export.last-directory", null))); settings.setLastImportDirectory(StringUtils.trimToNull(config.getString("sqljam.import.last-directory", @@ -66,6 +72,8 @@ public synchronized void save() throws IOException { config.set("sqljam.ui.data.page-size", settings.getDataPageSize()); config.set("sqljam.export.page-size", settings.getTransferPageSize()); config.set("sqljam.export.max-file-size", settings.getMaxDataFileSize()); + config.set("sqljam.export.parquet-max-file-size", settings.getParquetMaxFileSize()); + config.set("sqljam.export.csv-delimiter", settings.getCsvDelimiter()); config.set("sqljam.export.last-directory", settings.getLastExportDirectory()); config.set("sqljam.import.last-directory", settings.getLastImportDirectory()); config.save(); diff --git a/src/main/java/com/github/sqljam/face/service/TransferService.java b/src/main/java/com/github/sqljam/face/service/TransferService.java index cf76c59..d3bb3b3 100644 --- a/src/main/java/com/github/sqljam/face/service/TransferService.java +++ b/src/main/java/com/github/sqljam/face/service/TransferService.java @@ -28,6 +28,10 @@ import com.github.sqljam.face.model.ConnectionProfile; import com.github.sqljam.face.model.TransferRequest; import com.github.sqljam.impexp.AbstractFileImporter; +import com.github.sqljam.impexp.CsvExporter; +import com.github.sqljam.impexp.CsvImporter; +import com.github.sqljam.impexp.CsvOptions; +import com.github.sqljam.impexp.DataFileImporter; import com.github.sqljam.impexp.DataFormat; import com.github.sqljam.impexp.DbType; import com.github.sqljam.impexp.DuckDBWorkspace; @@ -39,6 +43,8 @@ import com.github.sqljam.impexp.ParquetImporter; import com.github.sqljam.impexp.ScriptExporter; import com.github.sqljam.impexp.SqlScriptRunner; +import com.github.sqljam.jdbc.ManualTransaction; +import com.github.sqljam.jdbc.SharedConnectionFactory; /** * @Description: TransferService runs exports to sql scripts, imports into other databases and imports of script @@ -50,10 +56,60 @@ public class TransferService { public void transfer(TransferRequest request, ExportListener listener) throws Exception { + transfer(request, null, listener); + } + + /** + * Exports into files, or imports into the target database, in the transaction if it is given + * + * @param transaction transaction of the target database opened by {@link #beginTransaction}, null to commit + * page by page + */ + public void transfer(TransferRequest request, ManualTransaction transaction, ExportListener listener) + throws Exception { if (request.getTarget() == TransferRequest.Target.SCRIPT) { exportScripts(request, listener); } else { - importDatabase(request, listener); + importDatabase(request, transaction, listener); + } + } + + /** + * Opens a transaction of the target database for an import which the user commits or rolls back + */ + public ManualTransaction beginTransaction(ConnectionProfile target, String catalog, String schema) + throws SQLException { + if (!target.getDbType().isTransactionSupported()) { + throw new IllegalArgumentException(target.getDbType().getDisplayName() + " has no transactions."); + } + Connection connection = connect(target, catalog, schema); + try { + return new ManualTransaction(connection, target.getDbType().isTransactionalDdl()); + } catch (SQLException | RuntimeException e) { + closeQuietly(connection); + throw e; + } + } + + /** + * Connection of the target database in the catalog and schema + */ + private static Connection connect(ConnectionProfile target, String catalog, String schema) throws SQLException { + String url = target.getJdbcUrl(StringUtils.defaultIfBlank(catalog, null)); + Connection connection = StringUtils.isBlank(target.getUsername()) ? DriverManager.getConnection(url) + : DriverManager.getConnection(url, target.getUsername(), target.getPassword()); + try { + if (StringUtils.isNotBlank(schema) && target.getDbType().isCanSetSchema()) { + connection.setSchema(schema); + } + if (StringUtils.isNotBlank(catalog) && (target.getDbType() == DbType.MYSQL + || target.getDbType() == DbType.MARIADB)) { + connection.setCatalog(catalog); + } + return connection; + } catch (SQLException | RuntimeException e) { + closeQuietly(connection); + throw e; } } @@ -74,6 +130,8 @@ private void configureSource(Exporter.ExportConfiguration configuration, Transfe configuration.setLobPageSize(request.getLobPageSize()); configuration.setConnectionPoolEnabled(true); configuration.setTableQueries(new HashMap<>(request.getTableQueries())); + configuration.setWriteMode(request.getWriteMode()); + configuration.setKeyColumnNames(new HashMap<>(request.getKeyColumnNames())); } private void exportScripts(TransferRequest request, ExportListener listener) throws Exception { @@ -84,6 +142,10 @@ private void exportScripts(TransferRequest request, ExportListener listener) thr exportParquet(request, listener); return; } + if (request.getDataFormat() == DataFormat.CSV) { + exportCsv(request, listener); + return; + } ScriptExporter scriptExporter = new ScriptExporter(request.getOutputDirectory(), request.getDataFileStrategy(), request.getMaxFileSize()); configureSource(scriptExporter.getConfiguration(), request); @@ -109,6 +171,7 @@ private void exportParquet(TransferRequest request, ExportListener listener) thr parquetExporter.setTargetSchemaName(request.getScriptSchema()); parquetExporter.setIdentifierCase(request.getIdentifierCase()); parquetExporter.setCompression(request.getCompression()); + parquetExporter.setMaxFileSize(request.getMaxFileSize()); int[] version = parseVersion(request.getScriptDbVersion()); if (version != null) { parquetExporter.setTargetVersion(version[0], version[1]); @@ -117,31 +180,85 @@ private void exportParquet(TransferRequest request, ExportListener listener) thr parquetExporter.export(request.getExportMode()); } + /** + * Export package with CSV files: schema.sql and constraints.sql for the target database type, the types of + * the columns are kept in manifest.json + */ + private void exportCsv(TransferRequest request, ExportListener listener) throws Exception { + CsvExporter csvExporter = new CsvExporter(request.getOutputDirectory()); + configureSource(csvExporter.getConfiguration(), request); + csvExporter.setTargetDbType(request.getScriptDbType()); + csvExporter.setTargetSchemaName(request.getScriptSchema()); + csvExporter.setIdentifierCase(request.getIdentifierCase()); + csvExporter.setOptions(request.getCsvOptions()); + csvExporter.setMaxFileSize(request.getMaxFileSize()); + int[] version = parseVersion(request.getScriptDbVersion()); + if (version != null) { + csvExporter.setTargetVersion(version[0], version[1]); + } + csvExporter.setExportListener(listener); + csvExporter.export(request.getExportMode()); + } + /** * Imports Parquet files of other tools into a table: a new table created from the Parquet schema, or rows * appended to an existing table */ - public ParquetImporter importParquet(ConnectionProfile target, String catalog, String schema, List files, - String tableName, ParquetImporter.Mode mode, ExportListener listener) + public DataFileImporter importParquet(ConnectionProfile target, String catalog, String schema, List files, + String tableName, DataFileImporter.Mode mode, ExportListener listener) throws Exception { - String url = target.getJdbcUrl(StringUtils.defaultIfBlank(catalog, null)); - try (Connection connection = StringUtils.isBlank(target.getUsername()) ? DriverManager.getConnection(url) - : DriverManager.getConnection(url, target.getUsername(), target.getPassword())) { - if (StringUtils.isNotBlank(schema) && target.getDbType().isCanSetSchema()) { - connection.setSchema(schema); - } - if (StringUtils.isNotBlank(catalog) && (target.getDbType() == DbType.MYSQL - || target.getDbType() == DbType.MARIADB)) { - connection.setCatalog(catalog); + return importFiles(target, catalog, schema, files, tableName, mode, null, listener); + } + + /** + * Imports Parquet or CSV files of other tools into a table (CSV when all files are .csv, .tsv or .txt): a new + * table created from the columns of the files, or rows appended to an existing table + * + * @param transaction transaction opened by {@link #beginTransaction}, null to commit page by page + */ + public DataFileImporter importFiles(ConnectionProfile target, String catalog, String schema, List files, + String tableName, DataFileImporter.Mode mode, ManualTransaction transaction, + ExportListener listener) throws Exception { + return importFiles(target, catalog, schema, files, tableName, mode, Collections.emptyList(), null, false, 0, + transaction, listener); + } + + /** + * Imports Parquet or CSV files of other tools into a table, rows are matched by the key columns + * + * @param keyColumns key columns of upserts and skips, empty for the primary key of the table + * @param csvOptions options of CSV files, null to detect them + * @param skipErrors whether rows with errors are skipped, otherwise the first error stops the import + * @param commitRows rows per commit without transaction, 0 for the default page size + */ + public DataFileImporter importFiles(ConnectionProfile target, String catalog, String schema, List files, + String tableName, DataFileImporter.Mode mode, List keyColumns, + CsvOptions csvOptions, boolean skipErrors, int commitRows, + ManualTransaction transaction, ExportListener listener) throws Exception { + Connection connection = transaction != null ? transaction.getConnection() : connect(target, catalog, schema); + try { + DataFileImporter importer; + if (files.stream().allMatch(DuckDBWorkspace::isCsv)) { + CsvImporter csvImporter = new CsvImporter(connection, target.getDbType()); + csvImporter.setOptions(csvOptions); + importer = csvImporter; + } else { + importer = new ParquetImporter(connection, target.getDbType()); } - ParquetImporter importer = new ParquetImporter(connection, target.getDbType()); if (StringUtils.isNotBlank(schema) && target.getDbType().isSchemaSupported()) { importer.setTargetSchemaName(schema); } - importer.setPageSize(Config.getInstance().getInt("sqljam.export.page-size", Exporter.DEFAULT_PAGE_SIZE)); + importer.setPageSize(commitRows > 0 ? commitRows : Config.getInstance().getInt("sqljam.export.page-size", + Exporter.DEFAULT_PAGE_SIZE)); + importer.setKeyColumns(keyColumns); + importer.setStopOnError(!skipErrors); importer.setExportListener(listener); importer.importFiles(files, tableName, mode); return importer; + } finally { + if (transaction == null) { + closeQuietly(connection); + } } } @@ -156,10 +273,18 @@ public ParquetPreview previewParquet(List files) throws Exception { * Columns, rows and the first rows (values as text) of Parquet files */ public ParquetPreview previewParquet(List files, int rowLimit) throws Exception { + return previewParquet(files, rowLimit, null); + } + + /** + * Columns, rows and first rows of Parquet or CSV files, CSV files are read with the options (null to detect) + */ + public ParquetPreview previewParquet(List files, int rowLimit, CsvOptions csvOptions) throws Exception { try (DuckDBWorkspace workspace = new DuckDBWorkspace(null)) { - List> firstRows = rowLimit > 0 ? workspace.readParquetRows(files, rowLimit) + List> firstRows = rowLimit > 0 ? workspace.readRows(files, rowLimit, csvOptions) : Collections.emptyList(); - return new ParquetPreview(workspace.describeParquet(files), workspace.countParquetRows(files), firstRows); + return new ParquetPreview(workspace.describeFiles(files, csvOptions), workspace.countFileRows(files, + csvOptions), firstRows); } } @@ -197,7 +322,8 @@ public long getRows() { } } - private void importDatabase(TransferRequest request, ExportListener listener) throws Exception { + private void importDatabase(TransferRequest request, ManualTransaction transaction, ExportListener listener) + throws Exception { ConnectionProfile target = request.getTargetProfile(); if (target == null) { throw new IllegalArgumentException("Target database must be required."); @@ -213,35 +339,52 @@ private void importDatabase(TransferRequest request, ExportListener listener) th importConfiguration.setTargetSchemaName(request.getTargetSchema()); importConfiguration.setTargetSchemaCreated(request.isTargetSchemaCreated()); importConfiguration.setTableNamePattern(request.getTableNamePattern()); - importConfiguration.setConnectionPoolEnabled(true); + importConfiguration.setConnectionPoolEnabled(transaction == null); + if (transaction != null) { + // All pages and tables in the one transaction + importExporter.setTargetConnectionFactory(new SharedConnectionFactory(transaction.getConnection())); + } importExporter.setIdentifierCase(request.getIdentifierCase()); importExporter.setExportListener(listener); importExporter.export(request.getExportMode()); } /** - * Imports an export package (sql scripts or Parquet files) into the target database + * Imports an export package (sql scripts, Parquet or CSV files) into the target database */ public AbstractFileImporter importScripts(ConnectionProfile target, String catalog, String schema, File directory, - boolean stopOnError, ExportListener listener) throws Exception { - String url = target.getJdbcUrl(StringUtils.defaultIfBlank(catalog, null)); - try (Connection connection = StringUtils.isBlank(target.getUsername()) ? DriverManager.getConnection(url) - : DriverManager.getConnection(url, target.getUsername(), target.getPassword())) { - if (StringUtils.isNotBlank(schema) && target.getDbType().isCanSetSchema()) { - connection.setSchema(schema); - } - if (StringUtils.isNotBlank(catalog) && (target.getDbType() == DbType.MYSQL - || target.getDbType() == DbType.MARIADB)) { - connection.setCatalog(catalog); - } + boolean stopOnError, ExportListener listener) throws Exception { + return importScripts(target, catalog, schema, directory, stopOnError, 0, null, listener); + } + + /** + * Imports an export package into the target database, in the transaction if it is given + * + * @param commitRows rows (statements of scripts) per commit without transaction, 0 for the default batch + * and page sizes + * @param transaction transaction opened by {@link #beginTransaction}, null to commit batch by batch + */ + public AbstractFileImporter importScripts(ConnectionProfile target, String catalog, String schema, File directory, + boolean stopOnError, int commitRows, ManualTransaction transaction, + ExportListener listener) throws Exception { + Connection connection = transaction != null ? transaction.getConnection() : connect(target, catalog, schema); + try { AbstractFileImporter importer = AbstractFileImporter.forDirectory(directory, connection, target.getDbType()); importer.setStopOnError(stopOnError); - importer.setBatchSize(Config.getInstance().getInt("sqljam.import.batch-size", - SqlScriptRunner.DEFAULT_BATCH_SIZE)); + importer.setBatchSize(commitRows > 0 ? commitRows : Config.getInstance().getInt( + "sqljam.import.batch-size", SqlScriptRunner.DEFAULT_BATCH_SIZE)); + // Data files are committed page by page + if (commitRows > 0 && importer instanceof DataFileImporter) { + ((DataFileImporter) importer).setPageSize(commitRows); + } importer.setExportListener(listener); importer.importDirectory(directory); return importer; + } finally { + if (transaction == null) { + closeQuietly(connection); + } } } diff --git a/src/main/java/com/github/sqljam/face/view/CsvOptionsPane.java b/src/main/java/com/github/sqljam/face/view/CsvOptionsPane.java new file mode 100644 index 0000000..e655768 --- /dev/null +++ b/src/main/java/com/github/sqljam/face/view/CsvOptionsPane.java @@ -0,0 +1,165 @@ +/* + * Copyright 2023-2026 Fred Feng + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.github.sqljam.face.view; + +import java.util.List; +import java.util.function.Function; + +import org.apache.commons.lang3.StringUtils; +import com.github.sqljam.impexp.CsvExporter; +import com.github.sqljam.impexp.CsvOptions; +import javafx.scene.Node; +import javafx.scene.control.CheckBox; +import javafx.scene.control.ComboBox; +import javafx.scene.control.Label; +import javafx.scene.control.TextField; +import javafx.scene.layout.GridPane; +import javafx.util.StringConverter; + +/** + * @Description: CsvOptionsPane edits the options of CSV files: delimiter, quote character, text of NULL, header row, + * encoding and date format. Options of files of other tools may be detected: an empty delimiter and an + * empty date format are detected from the files. + * @Author: Fred Feng + * @Date: 26/03/2023 + * @Version 1.0.0 + */ +final class CsvOptionsPane { + + private final ComboBox delimiterCombo = new ComboBox<>(); + private final ComboBox quoteCombo = new ComboBox<>(); + private final TextField nullField = new TextField(); + private final CheckBox headerCheck = new CheckBox(Messages.get("csv.header")); + private final ComboBox encodingCombo = new ComboBox<>(); + private final ComboBox dateFormatCombo = new ComboBox<>(); + private final GridPane node = new GridPane(); + private final boolean detected; + + /** + * @param detected whether the delimiter and the date format of files of other tools may be detected + */ + CsvOptionsPane(CsvOptions options, boolean detected) { + this.detected = detected; + delimiterCombo.setId("csvDelimiterCombo"); + quoteCombo.setId("csvQuoteCombo"); + nullField.setId("csvNullField"); + headerCheck.setId("csvHeaderCheck"); + encodingCombo.setId("csvEncodingCombo"); + dateFormatCombo.setId("csvDateFormatCombo"); + if (detected) { + delimiterCombo.getItems().add(""); + } + delimiterCombo.getItems().addAll(CsvExporter.DELIMITERS); + delimiterCombo.setConverter(converter(CsvOptionsPane::delimiterName)); + quoteCombo.getItems().addAll(CsvOptions.QUOTES); + quoteCombo.setConverter(converter(quote -> "\"".equals(quote) ? Messages.get("csv.quote.double") + : Messages.get("csv.quote.single"))); + nullField.setPromptText(Messages.get("csv.null.prompt")); + encodingCombo.getItems().addAll(CsvOptions.ENCODINGS); + dateFormatCombo.getItems().addAll(CsvOptions.DATE_FORMATS); + dateFormatCombo.setConverter(converter(format -> StringUtils.isEmpty(format) ? Messages.get(detected + ? "csv.dateFormat.detect" : "csv.dateFormat.iso") : format.replace("%Y", "yyyy").replace("%m", "MM") + .replace("%d", "dd"))); + for (ComboBox combo : List.of(delimiterCombo, quoteCombo, encodingCombo, dateFormatCombo)) { + combo.setMaxWidth(Double.MAX_VALUE); + } + node.setHgap(10); + node.setVgap(8); + node.addRow(0, new Label(Messages.get("csv.delimiter")), delimiterCombo, new Label(Messages.get("csv.quote")), + quoteCombo); + node.addRow(1, new Label(Messages.get("csv.null")), nullField, new Label(Messages.get("csv.encoding")), + encodingCombo); + node.addRow(2, new Label(Messages.get("csv.dateFormat")), dateFormatCombo, headerCheck); + setOptions(options); + } + + private static StringConverter converter(Function name) { + return new StringConverter<>() { + @Override + public String toString(String value) { + return value == null ? "" : name.apply(value); + } + + @Override + public String fromString(String text) { + return text; + } + }; + } + + /** + * Name of a delimiter: a tab has no visible character, an empty delimiter is detected + */ + static String delimiterName(String delimiter) { + if (delimiter == null) { + return ""; + } + switch (delimiter) { + case "": + return Messages.get("csv.delimiter.detect"); + case ",": + return Messages.get("export.format.delimiter.comma"); + case ";": + return Messages.get("export.format.delimiter.semicolon"); + case "\t": + return Messages.get("export.format.delimiter.tab"); + case "|": + return Messages.get("export.format.delimiter.pipe"); + default: + return delimiter; + } + } + + void setOptions(CsvOptions options) { + CsvOptions value = options != null ? options : new CsvOptions(); + delimiterCombo.setValue(delimiterCombo.getItems().contains(value.getDelimiter()) ? value.getDelimiter() + : detected ? "" : ","); + quoteCombo.setValue(quoteCombo.getItems().contains(value.getQuote()) ? value.getQuote() : "\""); + nullField.setText(value.getNullString()); + headerCheck.setSelected(value.isHeader()); + encodingCombo.setValue(encodingCombo.getItems().contains(value.getEncoding()) ? value.getEncoding() + : "UTF-8"); + dateFormatCombo.setValue(dateFormatCombo.getItems().contains(value.getDateFormat()) ? value.getDateFormat() + : ""); + } + + CsvOptions getOptions() { + CsvOptions options = new CsvOptions(); + options.setDelimiter(StringUtils.defaultString(delimiterCombo.getValue())); + options.setQuote(StringUtils.defaultIfEmpty(quoteCombo.getValue(), "\"")); + options.setNullString(StringUtils.defaultString(nullField.getText())); + options.setHeader(headerCheck.isSelected()); + options.setEncoding(StringUtils.defaultIfEmpty(encodingCombo.getValue(), "UTF-8")); + options.setDateFormat(StringUtils.defaultString(dateFormatCombo.getValue())); + return options; + } + + /** + * Runs the action when an option is changed, e.g. to preview the files again + */ + void setOnChange(Runnable action) { + delimiterCombo.valueProperty().addListener((obs, oldValue, value) -> action.run()); + quoteCombo.valueProperty().addListener((obs, oldValue, value) -> action.run()); + nullField.textProperty().addListener((obs, oldValue, value) -> action.run()); + headerCheck.selectedProperty().addListener((obs, oldValue, value) -> action.run()); + encodingCombo.valueProperty().addListener((obs, oldValue, value) -> action.run()); + dateFormatCombo.valueProperty().addListener((obs, oldValue, value) -> action.run()); + } + + Node getNode() { + return node; + } +} diff --git a/src/main/java/com/github/sqljam/face/view/ExportDialog.java b/src/main/java/com/github/sqljam/face/view/ExportDialog.java index c5ac061..e288dfb 100644 --- a/src/main/java/com/github/sqljam/face/view/ExportDialog.java +++ b/src/main/java/com/github/sqljam/face/view/ExportDialog.java @@ -18,6 +18,7 @@ import java.io.File; import java.util.ArrayList; import java.util.EnumMap; +import java.util.HashMap; import java.util.List; import java.util.Map; import java.util.Objects; @@ -29,6 +30,9 @@ import com.github.sqljam.face.model.TransferRequest; import com.github.sqljam.face.service.DatabaseSession; import com.github.sqljam.impexp.DataFileStrategy; +import com.github.sqljam.impexp.CsvExporter; +import com.github.sqljam.impexp.CsvOptions; +import com.github.sqljam.impexp.DuckDBWorkspace; import com.github.sqljam.impexp.DataFormat; import com.github.sqljam.impexp.DbType; import com.github.sqljam.impexp.Dialect; @@ -37,6 +41,7 @@ import com.github.sqljam.impexp.IdentifierCase; import com.github.sqljam.impexp.ParquetExporter; import com.github.sqljam.impexp.TableQuery; +import com.github.sqljam.impexp.WriteMode; import javafx.beans.InvalidationListener; import javafx.beans.property.BooleanProperty; import javafx.beans.property.SimpleBooleanProperty; @@ -60,8 +65,10 @@ import javafx.scene.control.SpinnerValueFactory; import javafx.scene.control.TextField; import javafx.scene.control.ToggleGroup; +import javafx.scene.control.Tooltip; import javafx.scene.control.cell.CheckBoxListCell; import javafx.scene.layout.ColumnConstraints; +import javafx.scene.layout.FlowPane; import javafx.scene.layout.GridPane; import javafx.scene.layout.HBox; import javafx.scene.layout.Priority; @@ -130,7 +137,23 @@ public class ExportDialog extends Dialog { private final ToggleGroup formatGroup = new ToggleGroup(); private final RadioButton sqlFormatRadio = new RadioButton(Messages.get("export.format.sql")); private final RadioButton parquetFormatRadio = new RadioButton(Messages.get("export.format.parquet")); + private final RadioButton csvFormatRadio = new RadioButton(Messages.get("export.format.csv")); private final ComboBox compressionCombo = new ComboBox<>(); + private CsvOptionsPane csvOptionsPane; + private TransactionOption transactionOption; + private final RadioButton insertRadio = new RadioButton(Messages.get("export.writeMode.insert")); + private final RadioButton upsertRadio = new RadioButton(Messages.get("export.writeMode.upsert")); + private final RadioButton skipExistingRadio = new RadioButton(Messages.get("export.writeMode.skip")); + private final Button keyColumnsButton = new Button(Messages.get("export.writeMode.keys")); + /** + * Chosen key columns of upserts by table, the primary key of the other tables + */ + private Map> keyColumnNames = new HashMap<>(); + /** + * Max file size of the shown format in MB: sql scripts and CSV files share one, Parquet has its own + */ + private long textMaxFileSize; + private long parquetMaxFileSize; private final Label layoutHint = new Label(); private final ComboBox targetCombo = new ComboBox<>(); private final ComboBox targetCatalogCombo = new ComboBox<>(); @@ -194,6 +217,7 @@ public ExportDialog(Window owner, AppContext context, DbNode node, TableQuery qu sqlFormatRadio.setId("sqlFormatRadio"); parquetFormatRadio.setId("parquetFormatRadio"); compressionCombo.setId("compressionCombo"); + csvFormatRadio.setId("csvFormatRadio"); layoutHint.setId("layoutHint"); targetCombo.setId("targetCombo"); targetCatalogCombo.setId("targetCatalogCombo"); @@ -235,6 +259,9 @@ public ExportDialog(Window owner, AppContext context, DbNode node, TableQuery qu }); setResultConverter(buttonType -> buttonType == ButtonType.OK ? buildRequest() : null); registerValidation(); + targetGroup.selectedToggleProperty().addListener((obs, oldToggle, toggle) -> updateWriteModes()); + targetCombo.valueProperty().addListener((obs, oldProfile, profile) -> updateWriteModes()); + scriptDbTypeCombo.valueProperty().addListener((obs, oldType, type) -> updateWriteModes()); preselect(node); if (tableQuery != null) { queryLabel.setText(Messages.format("export.query", queryTableName, tableQuery)); @@ -403,7 +430,29 @@ public IdentifierCase fromString(String text) { checks.addRow(2, failFastCheck); GridPane grid = grid(); grid.addRow(0, new Label(Messages.get("export.option.pageSize")), pageSizeSpinner); + insertRadio.setId("insertRadio"); + upsertRadio.setId("upsertRadio"); + skipExistingRadio.setId("skipExistingRadio"); + keyColumnsButton.setId("keyColumnsButton"); + ToggleGroup writeModeGroup = new ToggleGroup(); + insertRadio.setToggleGroup(writeModeGroup); + upsertRadio.setToggleGroup(writeModeGroup); + skipExistingRadio.setToggleGroup(writeModeGroup); + insertRadio.setSelected(true); + keyColumnsButton.disableProperty().bind(insertRadio.selectedProperty()); + keyColumnsButton.setOnAction(event -> chooseKeyColumns()); + insertRadio.selectedProperty().addListener((obs, oldValue, insert) -> { + // Rows of kept tables are matched by key, tables are not dropped + if (!insert) { + recreateCheck.setSelected(false); + } + recreateCheck.setDisable(!insert); + }); + // Wraps in narrow windows + FlowPane writeModeBox = new FlowPane(16, 6, insertRadio, upsertRadio, skipExistingRadio, keyColumnsButton); + writeModeBox.setAlignment(Pos.CENTER_LEFT); grid.addRow(1, new Label(Messages.get("export.option.identifierCase")), identifierCaseCombo); + grid.addRow(2, new Label(Messages.get("export.writeMode")), writeModeBox); return new VBox(12, checks, grid); } @@ -434,20 +483,44 @@ private Node createTargetPane() { singleFileRadio.setSelected(true); maxFileSizeSpinner.setEditable(true); long maxFileSize = context.getSettings().getMaxDataFileSize(); - maxFileSizeSpinner.getValueFactory().setValue(maxFileSize > 0 ? (int) Math.max(1, maxFileSize / MB) : 10); + textMaxFileSize = maxFileSize > 0 ? Math.max(1, maxFileSize / MB) : 0; + long parquetSize = context.getSettings().getParquetMaxFileSize(); + parquetMaxFileSize = parquetSize > 0 ? Math.max(1, parquetSize / MB) : 0; + maxFileSizeSpinner.getValueFactory().setValue((int) textMaxFileSize); lobSeparatedCheck.setSelected(true); sqlFormatRadio.setToggleGroup(formatGroup); parquetFormatRadio.setToggleGroup(formatGroup); + csvFormatRadio.setToggleGroup(formatGroup); sqlFormatRadio.setSelected(true); compressionCombo.getItems().setAll(ParquetExporter.COMPRESSIONS); compressionCombo.setValue(ParquetExporter.DEFAULT_COMPRESSION); - // Options of sql scripts do not apply to Parquet files + // Options of CSV files, the delimiter of the last export + CsvOptions csvOptions = new CsvOptions(); + csvOptions.setDelimiter(context.getSettings().getCsvDelimiter()); + csvOptionsPane = new CsvOptionsPane(csvOptions, false); + // Options of sql scripts do not apply to data files, each data file format has its own options compressionCombo.disableProperty().bind(parquetFormatRadio.selectedProperty().not()); - singleFileRadio.disableProperty().bind(parquetFormatRadio.selectedProperty()); - perTableRadio.disableProperty().bind(parquetFormatRadio.selectedProperty()); - maxFileSizeSpinner.disableProperty().bind(parquetFormatRadio.selectedProperty()); - lobSeparatedCheck.disableProperty().bind(parquetFormatRadio.selectedProperty()); - formatGroup.selectedToggleProperty().addListener((obs, oldToggle, toggle) -> updateLayoutHint()); + csvOptionsPane.getNode().disableProperty().bind(csvFormatRadio.selectedProperty().not()); + singleFileRadio.disableProperty().bind(sqlFormatRadio.selectedProperty().not()); + perTableRadio.disableProperty().bind(sqlFormatRadio.selectedProperty().not()); + lobSeparatedCheck.disableProperty().bind(sqlFormatRadio.selectedProperty().not()); + formatGroup.selectedToggleProperty().addListener((obs, oldToggle, toggle) -> { + // Parquet files are split at their own size, sql scripts and CSV files share one + boolean wasParquet = oldToggle == parquetFormatRadio; + boolean parquet = toggle == parquetFormatRadio; + if (wasParquet != parquet) { + commitSpinner(maxFileSizeSpinner); + long current = maxFileSizeSpinner.getValue(); + if (wasParquet) { + parquetMaxFileSize = current; + } else { + textMaxFileSize = current; + } + maxFileSizeSpinner.getValueFactory().setValue((int) (parquet ? parquetMaxFileSize + : textMaxFileSize)); + } + updateLayoutHint(); + }); layoutHint.getStyleClass().addAll("muted", "mono"); layoutHint.setWrapText(true); strategyGroup.selectedToggleProperty().addListener((obs, oldToggle, toggle) -> updateLayoutHint()); @@ -461,8 +534,9 @@ private Node createTargetPane() { scriptGrid.addRow(row++, new Label(Messages.get("export.script.version")), scriptVersionCombo); scriptGrid.addRow(row++, new Label(Messages.get("export.script.schema")), scriptSchemaField); scriptGrid.addRow(row++, new Label(Messages.get("export.format")), new HBox(16, sqlFormatRadio, - parquetFormatRadio)); + parquetFormatRadio, csvFormatRadio)); scriptGrid.addRow(row++, new Label(Messages.get("export.format.compression")), compressionCombo); + scriptGrid.addRow(row++, new Label(Messages.get("csv.options")), csvOptionsPane.getNode()); scriptGrid.addRow(row++, new Label(Messages.get("export.script.strategy")), new HBox(16, singleFileRadio, perTableRadio)); scriptGrid.addRow(row++, new Label(Messages.get("export.script.maxFileSize")), new HBox(8, @@ -502,6 +576,20 @@ private Node createTargetPane() { tableNameHint.setWrapText(true); databaseGrid.addRow(4, new Label(Messages.get("export.database.tableName")), tableNamePatternField); databaseGrid.add(tableNameHint, 1, 5); + transactionOption = new TransactionOption(context, targetCombo.valueProperty(), pageSizeSpinner.getValue()); + // Errors of an import into a database are skipped or not by the transaction option + failFastCheck.visibleProperty().bind(scriptRadio.selectedProperty()); + failFastCheck.managedProperty().bind(scriptRadio.selectedProperty()); + // Rows are committed page by page: the rows per commit and the page size are the same value + pageSizeSpinner.getValueFactory().valueProperty().bindBidirectional(transactionOption.commitRowsProperty()); + // The rows per commit follow the page size while it is typed, the values follow when it is committed + pageSizeSpinner.getEditor().textProperty().addListener((obs, oldText, text) -> { + String rows = StringUtils.trim(text); + if (StringUtils.isNumeric(rows) && !rows.isEmpty() && rows.length() < 9) { + transactionOption.showCommitRows(Integer.parseInt(rows)); + } + }); + databaseGrid.addRow(6, new Label(Messages.get("import.transaction")), transactionOption.getNode()); VBox databasePane = new VBox(databaseGrid); databasePane.getStyleClass().add("target-pane"); @@ -512,13 +600,23 @@ private Node createTargetPane() { return new VBox(10, new HBox(16, scriptRadio, databaseRadio), scriptPane, databasePane); } + /** + * Name of a CSV delimiter, a tab has no visible character + */ + static String delimiterName(String delimiter) { + return CsvOptionsPane.delimiterName(delimiter); + } + private void updateLayoutHint() { - if (parquetFormatRadio.isSelected()) { - layoutHint.setText("schema.sql\ndata/
.parquet\nconstraints.sql\nmanifest.json"); + boolean split = maxFileSizeSpinner.getValue() != null && maxFileSizeSpinner.getValue() > 0; + if (!sqlFormatRadio.isSelected()) { + String extension = parquetFormatRadio.isSelected() ? ParquetExporter.PARQUET_EXTENSION + : CsvExporter.CSV_EXTENSION; + layoutHint.setText("schema.sql\ndata/
" + extension + (split ? ",
_2" + extension + " ..." + : "") + "\nconstraints.sql\nmanifest.json"); return; } boolean perTable = perTableRadio.isSelected(); - boolean split = maxFileSizeSpinner.getValue() != null && maxFileSizeSpinner.getValue() > 0; StringBuilder hint = new StringBuilder(); hint.append("schema.sql\n"); if (perTable) { @@ -565,6 +663,41 @@ private void preselect(DbNode node) { loadCatalogs(profile, node.getCatalog(), node.getSchema(), table); } + /** + * Chooses key columns of the selected tables (all tables of the schema if none is selected) + */ + private void chooseKeyColumns() { + ConnectionProfile source = sourceCombo.getValue(); + if (source == null) { + return; + } + List tables = tableChoices.stream().filter(choice -> choice.selected.get()) + .map(choice -> choice.table.getName()).collect(Collectors.toList()); + if (tables.isEmpty()) { + tables = tableChoices.stream().map(choice -> choice.table.getName()).collect(Collectors.toList()); + } + new KeyColumnsDialog(getDialogPane().getScene().getWindow(), getSession(source), catalogCombo.isDisabled() + ? null : catalogCombo.getValue(), schemaCombo.isDisabled() ? null : schemaCombo.getValue(), tables, + keyColumnNames).showAndWait().ifPresent(keys -> keyColumnNames = new HashMap<>(keys)); + } + + /** + * Rows are not matched by key for targets which can not do it, e.g. ClickHouse + */ + private void updateWriteModes() { + DbType target = databaseRadio.isSelected() ? (targetCombo.getValue() != null ? targetCombo.getValue() + .getDbType() : null) : scriptDbTypeCombo.getValue(); + boolean supported = target == null || target.createDialect().isUpsertSupported(); + if (!supported) { + insertRadio.setSelected(true); + } + for (RadioButton radio : new RadioButton[]{upsertRadio, skipExistingRadio}) { + radio.setDisable(!supported); + radio.setTooltip(supported ? null : new Tooltip(Messages.format("export.writeMode.unsupported", + target.getDisplayName()))); + } + } + private DatabaseSession getSession(ConnectionProfile profile) { return context.getSessionManager().getSession(profile); } @@ -868,10 +1001,19 @@ TransferRequest buildRequest() { : DataFileStrategy.SINGLE_FILE); request.setMaxFileSize(maxFileSizeSpinner.getValue() * MB); request.setLobSeparated(lobSeparatedCheck.isSelected()); - request.setDataFormat(parquetFormatRadio.isSelected() ? DataFormat.PARQUET : DataFormat.SQL); + request.setDataFormat(parquetFormatRadio.isSelected() ? DataFormat.PARQUET : csvFormatRadio.isSelected() + ? DataFormat.CSV : DataFormat.SQL); request.setCompression(compressionCombo.getValue()); + request.setCsvOptions(csvOptionsPane.getOptions()); context.getSettings().setLastExportDirectory(directory.getAbsolutePath()); - context.getSettings().setMaxDataFileSize(maxFileSizeSpinner.getValue() * MB); + if (parquetFormatRadio.isSelected()) { + context.getSettings().setParquetMaxFileSize(maxFileSizeSpinner.getValue() * MB); + } else { + context.getSettings().setMaxDataFileSize(maxFileSizeSpinner.getValue() * MB); + } + if (csvFormatRadio.isSelected()) { + context.getSettings().setCsvDelimiter(request.getCsvDelimiter()); + } } else { request.setTarget(TransferRequest.Target.DATABASE); request.setTargetProfile(targetCombo.getValue()); @@ -880,7 +1022,18 @@ TransferRequest buildRequest() { targetSchemaCombo.getEditor().getText())); request.setTargetSchemaCreated(createSchemaCheck.isSelected()); request.setTableNamePattern(StringUtils.trimToNull(tableNamePatternField.getText())); + request.setOneTransaction(transactionOption.isOneTransaction()); + request.setConfirmCommit(transactionOption.isConfirmCommit()); + request.setFailFast(!transactionOption.isSkipErrors()); + if (!transactionOption.isOneTransaction()) { + // Rows per commit typed in the option, within the page sizes + request.setPageSize(Math.max(getMin(pageSizeSpinner), Math.min(getMax(pageSizeSpinner), + transactionOption.getCommitRows()))); + } } + request.setWriteMode(upsertRadio.isSelected() ? WriteMode.UPSERT : skipExistingRadio.isSelected() + ? WriteMode.SKIP_EXISTING : WriteMode.INSERT); + request.setKeyColumnNames(insertRadio.isSelected() ? new HashMap<>() : new HashMap<>(keyColumnNames)); // The query of the data viewer narrows its table if (tableQuery != null && selected.contains(queryTableName)) { request.getTableQueries().put(queryTableName, tableQuery); diff --git a/src/main/java/com/github/sqljam/face/view/Icons.java b/src/main/java/com/github/sqljam/face/view/Icons.java index bdc8e67..36a67d1 100644 --- a/src/main/java/com/github/sqljam/face/view/Icons.java +++ b/src/main/java/com/github/sqljam/face/view/Icons.java @@ -54,6 +54,7 @@ public final class Icons { public static final String GITHUB = "fth-github"; public static final String DISCONNECT = "fth-power"; public static final String EXIT = "fth-log-out"; + public static final String SAVE = "fth-save"; private Icons() { } diff --git a/src/main/java/com/github/sqljam/face/view/ImportPackageDialog.java b/src/main/java/com/github/sqljam/face/view/ImportPackageDialog.java index f6ab515..badec75 100644 --- a/src/main/java/com/github/sqljam/face/view/ImportPackageDialog.java +++ b/src/main/java/com/github/sqljam/face/view/ImportPackageDialog.java @@ -22,17 +22,20 @@ import org.apache.commons.io.FileUtils; import org.apache.commons.lang3.StringUtils; +import com.github.sqljam.config.Config; import com.github.sqljam.face.model.ConnectionProfile; +import com.github.sqljam.impexp.CsvExporter; import com.github.sqljam.impexp.DataFormat; import com.github.sqljam.impexp.DbType; import com.github.sqljam.impexp.ExportManifest; import com.github.sqljam.impexp.ParquetExporter; +import com.github.sqljam.impexp.SqlScriptRunner; +import com.github.sqljam.impexp.WriteMode; import javafx.beans.property.ReadOnlyObjectWrapper; import javafx.event.ActionEvent; import javafx.geometry.Insets; import javafx.scene.control.Button; import javafx.scene.control.ButtonType; -import javafx.scene.control.CheckBox; import javafx.scene.control.ComboBox; import javafx.scene.control.Dialog; import javafx.scene.control.Label; @@ -65,13 +68,14 @@ public class ImportPackageDialog extends Dialog tableView = new TableView<>(); private final VBox summaryPane; private final ComboBox targetCombo = new ComboBox<>(); private final ComboBox catalogCombo = new ComboBox<>(); private final ComboBox schemaCombo = new ComboBox<>(); - private final CheckBox stopOnErrorCheck = new CheckBox(Messages.get("import.stopOnError")); + private final TransactionOption transactionOption; private final Label warningLabel = new Label(); private final Button startButton; private ExportManifest manifest; @@ -89,15 +93,31 @@ public static class ImportRequest { private final String schema; private final boolean stopOnError; private final ExportManifest manifest; + /** + * The import runs in one transaction which the user commits or rolls back + */ + private final boolean oneTransaction; + /** + * A successful import in one transaction waits for the user to commit or roll back + */ + private final boolean confirmCommit; + /** + * Rows (statements of scripts) per commit when the import does not run in one transaction, 0 for the + * default batch and page sizes + */ + private final int commitRows; ImportRequest(File directory, ConnectionProfile target, String catalog, String schema, boolean stopOnError, - ExportManifest manifest) { + ExportManifest manifest, boolean oneTransaction, boolean confirmCommit, int commitRows) { this.directory = directory; this.target = target; this.catalog = catalog; this.schema = schema; this.stopOnError = stopOnError; this.manifest = manifest; + this.oneTransaction = oneTransaction; + this.confirmCommit = confirmCommit; + this.commitRows = commitRows; } } @@ -116,12 +136,12 @@ public ImportPackageDialog(Window owner, AppContext context, DbNode node) { targetValue.setId("targetValue"); filesValue.setId("filesValue"); formatValue.setId("formatValue"); + writeModeValue.setId("writeModeValue"); viewButton.setId("viewPackageButton"); tableView.setId("packageTables"); targetCombo.setId("targetCombo"); catalogCombo.setId("catalogCombo"); schemaCombo.setId("schemaCombo"); - stopOnErrorCheck.setId("stopOnErrorCheck"); warningLabel.setId("warningLabel"); directoryField.setOnAction(event -> loadManifest()); directoryField.focusedProperty().addListener((obs, wasFocused, focused) -> { @@ -148,6 +168,7 @@ public ImportPackageDialog(Window owner, AppContext context, DbNode node) { summaryGrid.addRow(3, new Label(Messages.get("import.manifest.target")), targetValue); summaryGrid.addRow(4, new Label(Messages.get("import.manifest.files")), filesValue); summaryGrid.addRow(5, new Label(Messages.get("export.format")), formatValue); + summaryGrid.addRow(6, new Label(Messages.get("export.writeMode")), writeModeValue); setupTableView(); Label summaryTitle = new Label(Messages.get("import.section.package")); summaryTitle.getStyleClass().add("section-title"); @@ -160,7 +181,6 @@ public ImportPackageDialog(Window owner, AppContext context, DbNode node) { schemaCombo.setMaxWidth(Double.MAX_VALUE); schemaCombo.setEditable(true); schemaCombo.setPromptText(Messages.get("import.schema.prompt")); - stopOnErrorCheck.setSelected(true); targetCombo.valueProperty().addListener((obs, oldProfile, profile) -> { loadCatalogs(profile); validateTarget(); @@ -180,7 +200,10 @@ public ImportPackageDialog(Window owner, AppContext context, DbNode node) { })); targetGrid.addRow(1, new Label(Messages.get("export.database.catalog")), catalogCombo); targetGrid.addRow(2, new Label(Messages.get("export.database.schema")), schemaCombo); - targetGrid.add(stopOnErrorCheck, 1, 3); + // Scripts are committed batch by batch, data files page by page + transactionOption = new TransactionOption(context, targetCombo.valueProperty(), Config.getInstance().getInt( + "sqljam.import.batch-size", SqlScriptRunner.DEFAULT_BATCH_SIZE)); + targetGrid.addRow(3, new Label(Messages.get("import.transaction")), transactionOption.getNode()); Label targetTitle = new Label(Messages.get("import.section.target")); targetTitle.getStyleClass().add("section-title"); VBox targetPane = new VBox(8, targetTitle, targetGrid); @@ -316,15 +339,16 @@ static boolean isScriptDirectory(File directory) { } /** - * Directory with Parquet files (directly or in data/) but without manifest.json + * Directory with Parquet or CSV files (directly or in data/) but without manifest.json */ static boolean isParquetDirectory(File directory) { - return hasParquetFiles(directory) || hasParquetFiles(new File(directory, "data")); + return hasDataFiles(directory) || hasDataFiles(new File(directory, "data")); } - private static boolean hasParquetFiles(File directory) { + private static boolean hasDataFiles(File directory) { File[] files = directory.listFiles((parent, name) -> name.toLowerCase(Locale.ENGLISH) - .endsWith(ParquetExporter.PARQUET_EXTENSION)); + .endsWith(ParquetExporter.PARQUET_EXTENSION) || name.toLowerCase(Locale.ENGLISH) + .endsWith(CsvExporter.CSV_EXTENSION)); return files != null && files.length > 0; } @@ -339,6 +363,7 @@ private void showSummary(ExportManifest manifest, boolean valid) { targetValue.setText("-"); filesValue.setText("-"); formatValue.setText(valid ? Messages.get("export.format.sql") : "-"); + writeModeValue.setText(valid ? Messages.get("export.writeMode.insert") : "-"); filterTargets(); return; } @@ -352,8 +377,12 @@ private void showSummary(ExportManifest manifest, boolean valid) { long totalSize = manifest.getFiles().stream().mapToLong(ExportManifest.FileEntry::getSize).sum(); filesValue.setText(Messages.format("import.manifest.filesValue", manifest.getFiles().size(), FileUtils.byteCountToDisplaySize(totalSize))); + // Rows of the same key are updated or skipped by packages of upserts or skips + WriteMode writeMode = WriteMode.of(manifest.getOptions().get("writeMode")); + writeModeValue.setText(Messages.get(writeMode == WriteMode.UPSERT ? "export.writeMode.upsert" + : writeMode == WriteMode.SKIP_EXISTING ? "export.writeMode.skip" : "export.writeMode.insert")); formatValue.setText(Messages.get(manifest.getDataFormat() == DataFormat.PARQUET ? "export.format.parquet" - : "export.format.sql")); + : manifest.getDataFormat() == DataFormat.CSV ? "export.format.csv" : "export.format.sql")); tableView.getItems().setAll(manifest.getTables()); filterTargets(); } @@ -484,6 +513,8 @@ ImportRequest buildRequest() { context.saveSettings(); return new ImportRequest(directory, targetCombo.getValue(), catalogCombo.isDisabled() ? null : catalogCombo.getValue(), schemaCombo.isDisabled() ? null - : StringUtils.trimToNull(schemaCombo.getEditor().getText()), stopOnErrorCheck.isSelected(), manifest); + : StringUtils.trimToNull(schemaCombo.getEditor().getText()), !transactionOption.isSkipErrors(), manifest, + transactionOption.isOneTransaction(), transactionOption.isConfirmCommit(), + transactionOption.isOneTransaction() ? 0 : transactionOption.getCommitRows()); } } diff --git a/src/main/java/com/github/sqljam/face/view/ImportParquetDialog.java b/src/main/java/com/github/sqljam/face/view/ImportParquetDialog.java index ffa5644..504e1f2 100644 --- a/src/main/java/com/github/sqljam/face/view/ImportParquetDialog.java +++ b/src/main/java/com/github/sqljam/face/view/ImportParquetDialog.java @@ -25,6 +25,11 @@ import com.github.sqljam.face.model.ConnectionProfile; import com.github.sqljam.face.service.TransferService; import com.github.sqljam.impexp.ParquetExporter; +import javafx.scene.control.CheckBox; +import com.github.sqljam.impexp.DuckDBWorkspace; +import com.github.sqljam.impexp.CsvSplitter; +import com.github.sqljam.impexp.CsvExporter; +import com.github.sqljam.impexp.CsvOptions; import com.github.sqljam.impexp.ParquetImporter; import javafx.beans.property.ReadOnlyObjectWrapper; import javafx.event.ActionEvent; @@ -71,7 +76,19 @@ public class ImportParquetDialog extends Dialog targetCombo = new ComboBox<>(); + private final TransactionOption transactionOption; private final ComboBox catalogCombo = new ComboBox<>(); private final ComboBox schemaCombo = new ComboBox<>(); private final Label warningLabel = new Label(); @@ -88,21 +105,61 @@ public static class ParquetRequest { private final List files; private final String tableName; private final ParquetImporter.Mode mode; + /** + * Key columns of upserts, empty for the primary key of the table + */ + private final List keyColumns; + /** + * Options of CSV files, null for Parquet files + */ + private final CsvOptions csvOptions; private final ConnectionProfile target; private final String catalog; private final String schema; + /** + * The import runs in one transaction which the user commits or rolls back + */ + private final boolean oneTransaction; + /** + * A successful import in one transaction waits for the user to commit or roll back + */ + private final boolean confirmCommit; + /** + * Rows with errors are skipped, otherwise an error rolls back one transaction or stops the import + */ + private final boolean skipErrors; + /** + * Rows per commit when the import does not run in one transaction, 0 for the default page size + */ + private final int commitRows; - ParquetRequest(List files, String tableName, ParquetImporter.Mode mode, ConnectionProfile target, - String catalog, String schema) { + ParquetRequest(List files, String tableName, ParquetImporter.Mode mode, List keyColumns, + CsvOptions csvOptions, ConnectionProfile target, String catalog, String schema, + boolean oneTransaction, boolean confirmCommit, boolean skipErrors, int commitRows) { this.files = files; this.tableName = tableName; this.mode = mode; + this.keyColumns = keyColumns; + this.csvOptions = csvOptions; this.target = target; this.catalog = catalog; this.schema = schema; + this.oneTransaction = oneTransaction; + this.confirmCommit = confirmCommit; + this.skipErrors = skipErrors; + this.commitRows = commitRows; } } + /** + * Options of CSV files of other tools: the delimiter and the dates are detected + */ + private static CsvOptions detectedCsvOptions() { + CsvOptions options = new CsvOptions(); + options.setDelimiter(""); + return options; + } + public ImportParquetDialog(Window owner, AppContext context, DbNode node) { this.context = context; Dialogs.initOwner(this, owner); @@ -117,6 +174,9 @@ public ImportParquetDialog(Window owner, AppContext context, DbNode node) { tableField.setId("tableField"); createRadio.setId("createRadio"); appendRadio.setId("appendRadio"); + upsertRadio.setId("upsertRadio"); + skipExistingRadio.setId("skipExistingRadio"); + keyColumnsField.setId("keyColumnsField"); targetCombo.setId("targetCombo"); catalogCombo.setId("catalogCombo"); schemaCombo.setId("schemaCombo"); @@ -162,13 +222,26 @@ protected void updateItem(File file, boolean empty) { previewTable.setPlaceholder(new Label(Messages.get("package.noRows"))); Label filesTitle = new Label(Messages.get("parquet.section.files")); filesTitle.getStyleClass().add("section-title"); - VBox filesPane = new VBox(8, filesTitle, fileBox, previewLabel, previewTable); + Label csvTitle = new Label(Messages.get("csv.options")); + csvTitle.getStyleClass().add("muted"); + csvOptionsBox = new VBox(6, csvTitle, csvOptionsPane.getNode()); + csvOptionsBox.setId("csvOptionsBox"); + csvOptionsBox.setVisible(false); + csvOptionsBox.setManaged(false); + csvOptionsPane.setOnChange(this::loadPreview); + VBox filesPane = new VBox(8, filesTitle, fileBox, csvOptionsBox, previewLabel, previewTable); filesPane.getStyleClass().add("form-section"); ToggleGroup modeGroup = new ToggleGroup(); createRadio.setToggleGroup(modeGroup); appendRadio.setToggleGroup(modeGroup); + upsertRadio.setToggleGroup(modeGroup); + skipExistingRadio.setToggleGroup(modeGroup); createRadio.setSelected(true); + keyColumnsField.setPromptText(Messages.get("parquet.keys.prompt")); + keyColumnsField.setTooltip(new Tooltip(Messages.get("parquet.keys.tooltip"))); + keyColumnsField.disableProperty().bind(upsertRadio.selectedProperty().not().and(skipExistingRadio + .selectedProperty().not())); DbTypeCells.setupProfileCombo(targetCombo); targetCombo.setMaxWidth(Double.MAX_VALUE); catalogCombo.setMaxWidth(Double.MAX_VALUE); @@ -196,7 +269,20 @@ protected void updateItem(File file, boolean empty) { targetGrid.addRow(1, new Label(Messages.get("export.database.catalog")), catalogCombo); targetGrid.addRow(2, new Label(Messages.get("export.database.schema")), schemaCombo); targetGrid.addRow(3, new Label(Messages.get("parquet.table")), tableField); - targetGrid.addRow(4, new Label(Messages.get("parquet.mode")), new HBox(16, createRadio, appendRadio)); + targetGrid.addRow(4, new Label(Messages.get("parquet.mode")), new VBox(6, createRadio, appendRadio, + upsertRadio, skipExistingRadio, keyColumnsField)); + // ClickHouse can not match rows by key + targetCombo.valueProperty().addListener((obs, oldProfile, profile) -> { + boolean supported = profile == null || profile.getDbType().createDialect().isUpsertSupported(); + if (!supported && (upsertRadio.isSelected() || skipExistingRadio.isSelected())) { + appendRadio.setSelected(true); + } + upsertRadio.setDisable(!supported); + skipExistingRadio.setDisable(!supported); + }); + transactionOption = new TransactionOption(context, targetCombo.valueProperty(), context.getSettings() + .getTransferPageSize()); + targetGrid.addRow(5, new Label(Messages.get("import.transaction")), transactionOption.getNode()); Label targetTitle = new Label(Messages.get("import.section.target")); targetTitle.getStyleClass().add("section-title"); VBox targetPane = new VBox(8, targetTitle, targetGrid); @@ -244,8 +330,12 @@ private static GridPane grid() { private void chooseFiles() { FileChooser chooser = new FileChooser(); chooser.setTitle(Messages.get("parquet.files.add")); - chooser.getExtensionFilters().add(new FileChooser.ExtensionFilter(Messages.get("parquet.files.filter"), - "*" + ParquetExporter.PARQUET_EXTENSION)); + chooser.getExtensionFilters().addAll(new FileChooser.ExtensionFilter(Messages.get("parquet.files.filter"), + "*" + ParquetExporter.PARQUET_EXTENSION, "*" + CsvExporter.CSV_EXTENSION, "*.tsv", "*.txt"), + new FileChooser.ExtensionFilter(Messages.get("parquet.files.filter.parquet"), + "*" + ParquetExporter.PARQUET_EXTENSION), + new FileChooser.ExtensionFilter(Messages.get("parquet.files.filter.csv"), + "*" + CsvExporter.CSV_EXTENSION, "*.tsv", "*.txt")); String lastDirectory = context.getSettings().getLastImportDirectory(); if (StringUtils.isNotBlank(lastDirectory) && new File(lastDirectory).isDirectory()) { chooser.setInitialDirectory(new File(lastDirectory)); @@ -259,7 +349,8 @@ private void chooseFiles() { } /** - * Adds Parquet files of a table, the table name is the base name of the first file by default + * Adds Parquet or CSV files of a table, the table name is the base name of the first file by default. Files + * are listed in the order of importing: orders.csv, orders_2.csv ... orders_n.csv */ public void addFiles(List files) { for (File file : files) { @@ -267,6 +358,7 @@ public void addFiles(List files) { fileList.getItems().add(file); } } + fileList.getItems().setAll(CsvSplitter.sortParts(fileList.getItems())); if (StringUtils.isBlank(tableField.getText()) && !fileList.getItems().isEmpty()) { tableField.setText(getTableName(fileList.getItems().get(0))); } @@ -300,11 +392,24 @@ private void loadPreview() { validate(); return; } + // Files of one table are all Parquet or all CSV + long csvFiles = files.stream().filter(DuckDBWorkspace::isCsv).count(); + boolean csv = csvFiles > 0 && csvFiles == files.size(); + csvOptionsBox.setVisible(csv); + csvOptionsBox.setManaged(csv); + if (csvFiles > 0 && csvFiles < files.size()) { + showPreview(null); + validate(); + warningLabel.setText(Messages.get("parquet.error.mixed")); + return; + } previewLabel.setText(Messages.get("package.loading")); previewTable.getColumns().clear(); previewTable.getItems().clear(); validate(); - TaskRunner.run(() -> context.getTransferService().previewParquet(files, PREVIEW_ROWS), preview -> { + CsvOptions csvOptions = csv ? csvOptionsPane.getOptions() : null; + TaskRunner.run(() -> context.getTransferService().previewParquet(files, PREVIEW_ROWS, csvOptions), + preview -> { if (version == previewVersion) { filesValid = true; showPreview(preview); @@ -401,8 +506,14 @@ private void loadSchemas(ConnectionProfile profile, String catalog) { ParquetRequest buildRequest() { return new ParquetRequest(new ArrayList<>(fileList.getItems()), tableField.getText().trim(), - appendRadio.isSelected() ? ParquetImporter.Mode.APPEND : ParquetImporter.Mode.CREATE, + upsertRadio.isSelected() ? ParquetImporter.Mode.UPSERT : skipExistingRadio.isSelected() + ? ParquetImporter.Mode.SKIP_EXISTING : appendRadio.isSelected() + ? ParquetImporter.Mode.APPEND : ParquetImporter.Mode.CREATE, KeyColumnsDialog.parse( + keyColumnsField.getText()), csvOptionsBox.isVisible() ? csvOptionsPane.getOptions() : null, targetCombo.getValue(), catalogCombo.isDisabled() ? null : catalogCombo.getValue(), - schemaCombo.isDisabled() ? null : StringUtils.trimToNull(schemaCombo.getEditor().getText())); + schemaCombo.isDisabled() ? null : StringUtils.trimToNull(schemaCombo.getEditor().getText()), + transactionOption.isOneTransaction(), transactionOption.isConfirmCommit(), + transactionOption.isSkipErrors(), transactionOption.isOneTransaction() ? 0 + : transactionOption.getCommitRows()); } } diff --git a/src/main/java/com/github/sqljam/face/view/KeyColumnsDialog.java b/src/main/java/com/github/sqljam/face/view/KeyColumnsDialog.java new file mode 100644 index 0000000..d4134c7 --- /dev/null +++ b/src/main/java/com/github/sqljam/face/view/KeyColumnsDialog.java @@ -0,0 +1,170 @@ +/* + * Copyright 2023-2026 Fred Feng + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.github.sqljam.face.view; + +import java.util.ArrayList; +import java.util.Arrays; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.stream.Collectors; + +import org.apache.commons.lang3.StringUtils; +import com.github.sqljam.face.model.ColumnInfo; +import com.github.sqljam.face.service.DatabaseSession; +import javafx.event.ActionEvent; +import javafx.geometry.Insets; +import javafx.scene.control.Button; +import javafx.scene.control.ButtonType; +import javafx.scene.control.Dialog; +import javafx.scene.control.Label; +import javafx.scene.control.ScrollPane; +import javafx.scene.control.TextField; +import javafx.scene.layout.ColumnConstraints; +import javafx.scene.layout.GridPane; +import javafx.scene.layout.Priority; +import javafx.scene.layout.VBox; +import javafx.stage.Window; + +/** + * @Description: KeyColumnsDialog chooses the columns which identify a row of each table when rows are upserted. An + * empty field is the primary key of the table, a table without primary key and chosen columns is + * inserted. Chosen columns must be columns of the table. + * @Author: Fred Feng + * @Date: 26/03/2023 + * @Version 1.0.0 + */ +public class KeyColumnsDialog extends Dialog>> { + + private final Map fields = new LinkedHashMap<>(); + private final Map> columnsByTable = new LinkedHashMap<>(); + private final Label errorLabel = new Label(); + private final Button okButton; + + /** + * @param chosen key columns chosen before, by table name + */ + public KeyColumnsDialog(Window owner, DatabaseSession session, String catalog, String schema, List tables, + Map> chosen) { + Dialogs.initOwner(this, owner); + Branding.applyIcons(this); + setTitle(Messages.get("keys.title")); + setHeaderText(Messages.get("keys.header")); + setResizable(true); + GridPane grid = new GridPane(); + grid.setId("keyColumnsGrid"); + grid.setHgap(12); + grid.setVgap(8); + ColumnConstraints tableColumn = new ColumnConstraints(); + tableColumn.setMinWidth(160); + ColumnConstraints fieldColumn = new ColumnConstraints(); + fieldColumn.setHgrow(Priority.ALWAYS); + grid.getColumnConstraints().addAll(tableColumn, fieldColumn); + int row = 0; + for (String table : tables) { + TextField field = new TextField(String.join(", ", chosen.getOrDefault(table, List.of()))); + field.setId("keyColumns-" + table); + field.setPromptText(Messages.get("keys.loading")); + field.textProperty().addListener((obs, oldText, text) -> validate()); + fields.put(table, field); + grid.addRow(row++, new Label(table), field); + } + ScrollPane scrollPane = new ScrollPane(grid); + scrollPane.setFitToWidth(true); + scrollPane.setPrefSize(560, Math.min(420, 60 + tables.size() * 40)); + errorLabel.setId("keyColumnsError"); + errorLabel.getStyleClass().add("status-error"); + errorLabel.setWrapText(true); + Label hint = new Label(Messages.get("keys.hint")); + hint.getStyleClass().add("muted"); + hint.setWrapText(true); + // Chosen columns should be unique in the target table + Label uniqueHint = new Label(Messages.get("keys.uniqueHint"), Icons.of(Icons.INFO)); + uniqueHint.setId("keyColumnsUniqueHint"); + uniqueHint.getStyleClass().add("muted"); + uniqueHint.setWrapText(true); + VBox content = new VBox(10, hint, uniqueHint, scrollPane, errorLabel); + content.setPadding(new Insets(4)); + getDialogPane().setContent(content); + getDialogPane().getButtonTypes().addAll(ButtonType.OK, ButtonType.CANCEL); + okButton = (Button) getDialogPane().lookupButton(ButtonType.OK); + okButton.addEventFilter(ActionEvent.ACTION, event -> { + if (validate() != null) { + event.consume(); + } + }); + setResultConverter(type -> type == ButtonType.OK ? getKeyColumns() : null); + // Primary keys and columns of the tables, the primary key is the placeholder of an empty field + TaskRunner.run(() -> { + Map> columns = new LinkedHashMap<>(); + for (String table : tables) { + columns.put(table, session.getColumns(catalog, schema, table)); + } + return columns; + }, columns -> columns.forEach((table, infos) -> { + columnsByTable.put(table, infos.stream().map(ColumnInfo::getName).collect(Collectors.toList())); + String primaryKey = infos.stream().filter(ColumnInfo::isPrimaryKey).map(ColumnInfo::getName) + .collect(Collectors.joining(", ")); + fields.get(table).setPromptText(primaryKey.isEmpty() ? Messages.get("keys.none") + : Messages.format("keys.primaryKey", primaryKey)); + validate(); + }), e -> errorLabel.setText(StringUtils.defaultIfBlank(e.getMessage(), e.getClass().getSimpleName()))); + } + + /** + * The first error of the chosen columns, null if all of them are columns of their tables + */ + String validate() { + String error = null; + for (Map.Entry entry : fields.entrySet()) { + List columns = columnsByTable.get(entry.getKey()); + if (columns == null) { + continue; + } + for (String key : parse(entry.getValue().getText())) { + if (columns.stream().noneMatch(key::equalsIgnoreCase)) { + error = Messages.format("keys.error.column", key, entry.getKey()); + break; + } + } + if (error != null) { + break; + } + } + errorLabel.setText(error); + okButton.setDisable(error != null); + return error; + } + + /** + * Chosen key columns by table, tables of an empty field use their primary keys + */ + Map> getKeyColumns() { + Map> keys = new LinkedHashMap<>(); + fields.forEach((table, field) -> { + List columns = parse(field.getText()); + if (!columns.isEmpty()) { + keys.put(table, columns); + } + }); + return keys; + } + + static List parse(String text) { + return new ArrayList<>(Arrays.stream(StringUtils.defaultString(text).split(",")).map(String::trim) + .filter(StringUtils::isNotEmpty).collect(Collectors.toList())); + } +} diff --git a/src/main/java/com/github/sqljam/face/view/MainView.java b/src/main/java/com/github/sqljam/face/view/MainView.java index 3bfa556..09a486a 100644 --- a/src/main/java/com/github/sqljam/face/view/MainView.java +++ b/src/main/java/com/github/sqljam/face/view/MainView.java @@ -17,14 +17,24 @@ import java.io.File; import java.io.IOException; +import java.util.List; import java.util.Optional; +import java.util.concurrent.atomic.AtomicReference; +import java.util.stream.Collectors; +import com.github.sqljam.config.Config; import com.github.sqljam.face.model.ConnectionProfile; +import com.github.sqljam.face.model.TableInfo; import com.github.sqljam.face.model.TransferRequest; +import com.github.sqljam.face.service.DatabaseSession; import com.github.sqljam.face.service.TransferService; import com.github.sqljam.impexp.AbstractFileImporter; -import com.github.sqljam.impexp.ParquetImporter; +import com.github.sqljam.impexp.DataFileImporter; +import com.github.sqljam.impexp.DataFormat; +import com.github.sqljam.impexp.ExportManifest; +import com.github.sqljam.impexp.SqlScriptRunner; import com.github.sqljam.impexp.TableQuery; +import com.github.sqljam.jdbc.ManualTransaction; import javafx.application.Platform; import javafx.beans.binding.Bindings; import javafx.geometry.Insets; @@ -330,25 +340,35 @@ private void showExportDialog(DbNode node, TableQuery query) { Optional request = dialog.showAndWait(); // Data sources created as the target in the dialog connectionTree.addNewConnections(); - request.ifPresent(transferRequest -> { - int totalTables = transferRequest.getTables().isEmpty() ? dialog.getListedTableCount() - : transferRequest.getTables().size(); - String title = transferRequest.getTarget() == TransferRequest.Target.SCRIPT - ? Messages.get("progress.exportTitle") : Messages.format("progress.importTitle", - transferRequest.getTargetProfile().getName()); - ProgressDialog progress = new ProgressDialog(getWindow(), context, title, totalTables); - if (transferRequest.getTarget() == TransferRequest.Target.SCRIPT) { - progress.setOutputDirectory(transferRequest.getOutputDirectory()); + request.ifPresent(transferRequest -> startTransfer(transferRequest, dialog.getListedTableCount())); + } + + private void startTransfer(TransferRequest transferRequest, int listedTableCount) { + int totalTables = transferRequest.getTables().isEmpty() ? listedTableCount + : transferRequest.getTables().size(); + String title = transferRequest.getTarget() == TransferRequest.Target.SCRIPT + ? Messages.get("progress.exportTitle") : Messages.format("progress.importTitle", + transferRequest.getTargetProfile().getName()); + ProgressDialog progress = new ProgressDialog(getWindow(), context, title, totalTables); + if (transferRequest.getTarget() == TransferRequest.Target.SCRIPT) { + progress.setOutputDirectory(transferRequest.getOutputDirectory()); + } + TransferService transferService = context.getTransferService(); + progress.setSkipErrors(!transferRequest.isFailFast()); + // One transaction is committed when the import is complete, or by the user when it is confirmed + AtomicReference transaction = new AtomicReference<>(); + progress.run(() -> { + if (transferRequest.getTarget() == TransferRequest.Target.DATABASE + && transferRequest.isOneTransaction()) { + transaction.set(transferService.beginTransaction(transferRequest.getTargetProfile(), + transferRequest.getTargetCatalog(), null)); + } + transferService.transfer(transferRequest, transaction.get(), progress.getListener()); + return null; + }, null, transaction::get, transferRequest.isConfirmCommit(), () -> { + if (transferRequest.getTarget() == TransferRequest.Target.DATABASE) { + context.getSessionManager().closeSession(transferRequest.getTargetProfile().getId()); } - TransferService transferService = context.getTransferService(); - progress.run(() -> { - transferService.transfer(transferRequest, progress.getListener()); - return null; - }, null, () -> { - if (transferRequest.getTarget() == TransferRequest.Target.DATABASE) { - context.getSessionManager().closeSession(transferRequest.getTargetProfile().getId()); - } - }); }); } @@ -357,34 +377,54 @@ private void showImportDialog(DbNode node) { Optional request = new ImportPackageDialog(getWindow(), context, node) .showAndWait(); connectionTree.addNewConnections(); - request.ifPresent(importRequest -> { - int totalTables = importRequest.getManifest() != null ? importRequest.getManifest().getTables().size() - : 0; - ProgressDialog progress = new ProgressDialog(getWindow(), context, Messages.format( - "progress.importTitle", importRequest.getTarget().getName()), totalTables); - progress.run(() -> context.getTransferService().importScripts(importRequest.getTarget(), - importRequest.getCatalog(), importRequest.getSchema(), importRequest.getDirectory(), - importRequest.isStopOnError(), progress.getListener()), (AbstractFileImporter importer) -> - Messages.format("import.summary", importer.getExecutedCount(), importer.getLobCount(), - importer.getFailedCount()), () -> context.getSessionManager() - .closeSession(importRequest.getTarget().getId())); - }); + request.ifPresent(this::startImport); + } + + private void startImport(ImportPackageDialog.ImportRequest importRequest) { + int totalTables = importRequest.getManifest() != null ? importRequest.getManifest().getTables().size() + : 0; + ProgressDialog progress = new ProgressDialog(getWindow(), context, Messages.format( + "progress.importTitle", importRequest.getTarget().getName()), totalTables); + progress.setSkipErrors(!importRequest.isStopOnError()); + AtomicReference transaction = new AtomicReference<>(); + progress.run(() -> { + if (importRequest.isOneTransaction()) { + transaction.set(context.getTransferService().beginTransaction(importRequest.getTarget(), + importRequest.getCatalog(), importRequest.getSchema())); + } + return context.getTransferService().importScripts(importRequest.getTarget(), importRequest + .getCatalog(), importRequest.getSchema(), importRequest.getDirectory(), importRequest + .isStopOnError(), importRequest.getCommitRows(), transaction.get(), progress.getListener()); + }, (AbstractFileImporter importer) -> Messages.format("import.summary", importer.getExecutedCount(), + importer.getLobCount(), importer.getFailedCount()), transaction::get, importRequest.isConfirmCommit(), + () -> context.getSessionManager().closeSession(importRequest.getTarget().getId())); } private void showImportParquetDialog(DbNode node) { Optional request = new ImportParquetDialog(getWindow(), context, node) .showAndWait(); connectionTree.addNewConnections(); - request.ifPresent(parquetRequest -> { - ProgressDialog progress = new ProgressDialog(getWindow(), context, Messages.format( - "progress.importTitle", parquetRequest.getTarget().getName()), 1); - progress.run(() -> context.getTransferService().importParquet(parquetRequest.getTarget(), - parquetRequest.getCatalog(), parquetRequest.getSchema(), parquetRequest.getFiles(), - parquetRequest.getTableName(), parquetRequest.getMode(), progress.getListener()), - (ParquetImporter importer) -> Messages.format("parquet.summary", importer.getImportedRows(), - parquetRequest.getTableName()), () -> context.getSessionManager() - .closeSession(parquetRequest.getTarget().getId())); - }); + request.ifPresent(this::startFileImport); + } + + private void startFileImport(ImportParquetDialog.ParquetRequest parquetRequest) { + ProgressDialog progress = new ProgressDialog(getWindow(), context, Messages.format( + "progress.importTitle", parquetRequest.getTarget().getName()), 1); + progress.setSkipErrors(parquetRequest.isSkipErrors()); + AtomicReference transaction = new AtomicReference<>(); + progress.run(() -> { + if (parquetRequest.isOneTransaction()) { + transaction.set(context.getTransferService().beginTransaction(parquetRequest.getTarget(), + parquetRequest.getCatalog(), parquetRequest.getSchema())); + } + return context.getTransferService().importFiles(parquetRequest.getTarget(), parquetRequest + .getCatalog(), parquetRequest.getSchema(), parquetRequest.getFiles(), parquetRequest + .getTableName(), parquetRequest.getMode(), parquetRequest.getKeyColumns(), parquetRequest + .getCsvOptions(), parquetRequest.isSkipErrors(), parquetRequest.getCommitRows(), transaction.get(), + progress.getListener()); + }, (DataFileImporter importer) -> Messages.format("parquet.summary", importer.getImportedRows(), + parquetRequest.getTableName()), transaction::get, parquetRequest.isConfirmCommit(), + () -> context.getSessionManager().closeSession(parquetRequest.getTarget().getId())); } /** diff --git a/src/main/java/com/github/sqljam/face/view/ProgressDialog.java b/src/main/java/com/github/sqljam/face/view/ProgressDialog.java index 495b755..a7ca42d 100644 --- a/src/main/java/com/github/sqljam/face/view/ProgressDialog.java +++ b/src/main/java/com/github/sqljam/face/view/ProgressDialog.java @@ -16,20 +16,38 @@ package com.github.sqljam.face.view; import java.io.File; +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.time.LocalDateTime; +import java.time.LocalTime; +import java.time.format.DateTimeFormatter; +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; import java.util.concurrent.Callable; import java.util.concurrent.atomic.AtomicBoolean; import java.util.function.Function; +import java.util.function.Supplier; import org.apache.commons.lang3.StringUtils; import com.github.sqljam.impexp.ExportCancelledException; import com.github.sqljam.impexp.ExportListener; +import com.github.sqljam.jdbc.ManualTransaction; import javafx.animation.KeyFrame; import javafx.animation.Timeline; import javafx.application.Platform; +import javafx.collections.FXCollections; +import javafx.collections.ObservableList; +import javafx.collections.transformation.FilteredList; import javafx.geometry.Insets; import javafx.geometry.Pos; import javafx.scene.Scene; +import javafx.scene.control.Alert; import javafx.scene.control.Button; +import javafx.scene.control.ButtonBar; +import javafx.scene.control.ButtonType; +import javafx.scene.control.CheckBox; import javafx.scene.control.Label; import javafx.scene.control.ListCell; import javafx.scene.control.ListView; @@ -40,6 +58,7 @@ import javafx.scene.layout.Priority; import javafx.scene.layout.Region; import javafx.scene.layout.VBox; +import javafx.stage.FileChooser; import javafx.stage.Modality; import javafx.stage.Stage; import javafx.stage.Window; @@ -57,6 +76,11 @@ public class ProgressDialog { private static final int MAX_LOG_ENTRIES = 5000; + /** + * Entries kept for Save Log, e.g. every skipped row of a large import + */ + private static final int MAX_SAVED_LOG_ENTRIES = 1_000_000; + private static final DateTimeFormatter LOG_TIME = DateTimeFormatter.ofPattern("HH:mm:ss"); private final AppContext context; private final Stage stage = new Stage(); @@ -68,9 +92,17 @@ public class ProgressDialog { private final ProgressBar tableBar = new ProgressBar(0); private final Label elapsedLabel = new Label(); private final ListView logList = new ListView<>(); + private final ObservableList logEntries = FXCollections.observableArrayList(); + private final FilteredList shownLogEntries = new FilteredList<>(logEntries); + private final List savedLogEntries = new ArrayList<>(); + private long droppedLogEntries; + private final CheckBox errorsOnlyCheck = new CheckBox(Messages.get("progress.errorsOnly")); + private final Button saveLogButton = new Button(Messages.get("progress.saveLog"), Icons.of(Icons.SAVE)); private final Button cancelButton = new Button(Messages.get("button.cancel"), Icons.of(Icons.CANCEL)); private final Button openFolderButton = new Button(Messages.get("progress.openFolder"), Icons.of(Icons.OPEN)); private final Button viewPackageButton = new Button(Messages.get("package.view"), Icons.of(Icons.TABLE)); + private final Button commitButton = new Button(Messages.get("progress.commit"), Icons.of(Icons.START)); + private final Button rollbackButton = new Button(Messages.get("progress.rollback"), Icons.of(Icons.CANCEL)); private final Label manifestLabel = new Label(); private final AtomicBoolean cancelled = new AtomicBoolean(); private final int totalTables; @@ -80,9 +112,30 @@ public class ProgressDialog { private Timeline timer; private File outputDirectory; private boolean finished; + /** + * Transaction of an import waiting for the decision of the user: commit or roll back + */ + private ManualTransaction pendingTransaction; + private Runnable pendingCompleted; + /** + * Reminds the user of an import waiting for commit, its transaction holds locks, undo and log space + */ + private Timeline reminder; + private Duration reminderInterval; + private long pendingSince; + private boolean reminded; + /** + * The transaction is committed or rolled back without asking the user + */ + private boolean automatic; + /** + * Errors of rows which were skipped do not roll back the transaction + */ + private boolean skipErrors; private static class LogEntry { + private final LocalTime time = LocalTime.now(); private final String message; private final boolean error; @@ -90,6 +143,11 @@ private static class LogEntry { this.message = message; this.error = error; } + + @Override + public String toString() { + return LOG_TIME.format(time) + (error ? " ERROR " : " INFO ") + message; + } } public ProgressDialog(Window owner, AppContext context, String title, int totalTables) { @@ -107,10 +165,14 @@ public ProgressDialog(Window owner, AppContext context, String title, int totalT tableBar.setId("tableBar"); elapsedLabel.setId("elapsedLabel"); logList.setId("logList"); + errorsOnlyCheck.setId("errorsOnlyCheck"); + saveLogButton.setId("saveLogButton"); cancelButton.setId("cancelButton"); openFolderButton.setId("openFolderButton"); viewPackageButton.setId("viewPackageButton"); manifestLabel.setId("manifestLabel"); + commitButton.setId("commitButton"); + rollbackButton.setId("rollbackButton"); statusLabel.getStyleClass().add("progress-status"); statusLabel.setText(Messages.get("progress.running")); @@ -140,6 +202,12 @@ protected void updateItem(LogEntry item, boolean empty) { } }); logList.getStyleClass().addAll("log-list", Styles.DENSE); + logList.setItems(shownLogEntries); + // Errors and skipped rows only + errorsOnlyCheck.selectedProperty().addListener((obs, oldValue, errorsOnly) -> shownLogEntries.setPredicate( + errorsOnly ? entry -> entry.error : null)); + saveLogButton.setMinWidth(Region.USE_PREF_SIZE); + saveLogButton.setOnAction(event -> saveLog()); VBox.setVgrow(logList, Priority.ALWAYS); cancelButton.setOnAction(event -> { @@ -173,9 +241,18 @@ protected void updateItem(LogEntry item, boolean empty) { openFolderButton.setMinWidth(Region.USE_PREF_SIZE); viewPackageButton.setMinWidth(Region.USE_PREF_SIZE); cancelButton.setMinWidth(Region.USE_PREF_SIZE); + commitButton.getStyleClass().add(Styles.ACCENT); + for (Button button : new Button[]{commitButton, rollbackButton}) { + button.setMinWidth(Region.USE_PREF_SIZE); + button.setVisible(false); + button.setManaged(false); + } + commitButton.setOnAction(event -> decide(true)); + rollbackButton.setOnAction(event -> decide(false)); Region spacer = new Region(); HBox.setHgrow(spacer, Priority.ALWAYS); - HBox buttons = new HBox(8, spacer, viewPackageButton, openFolderButton, cancelButton); + HBox buttons = new HBox(8, errorsOnlyCheck, saveLogButton, spacer, viewPackageButton, openFolderButton, + rollbackButton, commitButton, cancelButton); buttons.setAlignment(Pos.CENTER_RIGHT); Region headerSpacer = new Region(); @@ -196,7 +273,14 @@ protected void updateItem(LogEntry item, boolean empty) { } stage.setScene(scene); stage.setOnCloseRequest(event -> { - if (!finished) { + if (pendingTransaction != null) { + // The window stays until the import is rolled back + event.consume(); + if (Dialogs.confirm(stage, Messages.get("progress.decide.title"), + Messages.get("progress.decide.close"))) { + decide(false, true); + } + } else if (!finished) { cancelled.set(true); } }); @@ -210,7 +294,7 @@ Stage getStage() { * Error entries of the log */ long getErrorLogCount() { - return logList.getItems().stream().filter(entry -> entry.error).count(); + return logEntries.stream().filter(entry -> entry.error).count(); } boolean isFinished() { @@ -312,11 +396,54 @@ private void addLog(String message, boolean error) { if (StringUtils.isBlank(message)) { return; } - logList.getItems().add(new LogEntry(message, error)); - if (logList.getItems().size() > MAX_LOG_ENTRIES) { - logList.getItems().remove(0, logList.getItems().size() - MAX_LOG_ENTRIES); + LogEntry entry = new LogEntry(message, error); + logEntries.add(entry); + if (logEntries.size() > MAX_LOG_ENTRIES) { + logEntries.remove(0, logEntries.size() - MAX_LOG_ENTRIES); + } + if (savedLogEntries.size() < MAX_SAVED_LOG_ENTRIES) { + savedLogEntries.add(entry); + } else { + droppedLogEntries++; + } + if (!shownLogEntries.isEmpty()) { + logList.scrollTo(shownLogEntries.size() - 1); + } + } + + /** + * Saves the whole log, the window shows its last entries only + */ + private void saveLog() { + FileChooser chooser = new FileChooser(); + chooser.setTitle(Messages.get("progress.saveLog")); + chooser.setInitialFileName("sqljam-" + DateTimeFormatter.ofPattern("yyyyMMdd-HHmmss").format( + LocalDateTime.now()) + ".log"); + chooser.getExtensionFilters().add(new FileChooser.ExtensionFilter(Messages.get("progress.logFiles"), + "*.log", "*.txt")); + File file = chooser.showSaveDialog(stage); + if (file != null) { + try { + writeLog(file); + addLog(Messages.format("progress.logSaved", file.getAbsolutePath()), false); + } catch (IOException e) { + Dialogs.showError(stage, Messages.get("progress.saveLog"), e); + } + } + } + + /** + * Writes the log as text, one entry a line with its time and level + */ + void writeLog(File file) throws IOException { + List lines = new ArrayList<>(); + lines.add(stage.getTitle()); + lines.add(Messages.format("progress.statusLine", statusLabel.getText(), errorCount)); + savedLogEntries.forEach(entry -> lines.add(entry.toString())); + if (droppedLogEntries > 0) { + lines.add(Messages.format("progress.logDropped", droppedLogEntries)); } - logList.scrollTo(logList.getItems().size() - 1); + Files.write(file.toPath(), lines, StandardCharsets.UTF_8); } /** @@ -325,6 +452,26 @@ private void addLog(String message, boolean error) { * @param summary summary text of the job result when completed, may be null */ public void run(Callable job, Function summary, Runnable onCompleted) { + run(job, summary, null, onCompleted); + } + + /** + * Shows the dialog and runs an import in one transaction which waits for the user to commit or roll back + */ + public void run(Callable job, Function summary, Supplier transaction, + Runnable onCompleted) { + run(job, summary, transaction, true, onCompleted); + } + + /** + * Shows the dialog and runs an import in background. An import in one transaction is committed when it + * succeeds and rolled back when it fails or has errors, with confirm the user commits or rolls back. + * + * @param transaction the transaction opened by the job, null or a supplier of null for commits every N rows + * @param confirm whether a successful import waits for the user to commit or roll back + */ + public void run(Callable job, Function summary, Supplier transaction, + boolean confirm, Runnable onCompleted) { startTime = System.currentTimeMillis(); timer = new Timeline(new KeyFrame(Duration.seconds(1), event -> updateElapsed())); timer.setCycleCount(Timeline.INDEFINITE); @@ -335,14 +482,30 @@ public void run(Callable job, Function summary, Runnable onCom TaskRunner.execute(() -> { try { T result = job.call(); - Platform.runLater(() -> finish(errorCount == 0 ? Messages.get("progress.completed") - : Messages.format("progress.completedWithErrors", errorCount), false, false, - summary != null ? summary.apply(result) : null, onCompleted)); + ManualTransaction pending = transaction != null ? transaction.get() : null; + Platform.runLater(() -> { + String text = summary != null ? summary.apply(result) : null; + if (pending != null && confirm) { + awaitDecision(pending, text, onCompleted); + } else if (pending != null) { + completeTransaction(pending, text, onCompleted); + } else { + finish(errorCount == 0 ? Messages.get("progress.completed") + : Messages.format("progress.completedWithErrors", errorCount), false, false, text, + onCompleted); + } + }); } catch (Throwable e) { boolean cancel = isCancellation(e); if (!cancel && log.isErrorEnabled()) { log.error(e.getMessage(), e); } + // Nothing of a failed or cancelled import is kept + ManualTransaction pending = transaction != null ? transaction.get() : null; + if (pending != null) { + pending.close(); + Platform.runLater(() -> addLog(Messages.get("progress.rolledBack"), false)); + } Platform.runLater(() -> { if (!cancel) { errorCount++; @@ -355,6 +518,215 @@ public void run(Callable job, Function summary, Runnable onCom }); } + /** + * The import is written in a transaction: the user is asked to commit or roll back. An import with errors is + * rolled back, nothing of it is kept, unless the rows with errors were skipped. + */ + private void awaitDecision(ManualTransaction transaction, String summary, Runnable onCompleted) { + if (StringUtils.isNotBlank(summary)) { + addLog(summary, false); + } + pendingTransaction = transaction; + pendingCompleted = onCompleted; + if (transaction.isRolledBack() || (errorCount > 0 && !skipErrors)) { + decide(false, false); + return; + } + timer.stop(); + updateElapsed(); + completedTables = Math.max(completedTables, totalTables); + updateOverall(); + updatePercent(1, 1); + if (tableBar.getProgress() < 0) { + tableBar.setProgress(1); + } + statusLabel.setText(Messages.get("progress.pending")); + statusLabel.getStyleClass().add("status-pending"); + addLog(Messages.get("progress.pending.message"), false); + if (!transaction.isTransactionalDdl()) { + addLog(Messages.get("progress.pending.ddl"), false); + } + if (!transaction.getDeferredStatements().isEmpty()) { + addLog(Messages.format("progress.pending.deferred", transaction.getDeferredStatements().size()), false); + } + cancelButton.setVisible(false); + cancelButton.setManaged(false); + for (Button button : new Button[]{commitButton, rollbackButton}) { + button.setVisible(true); + button.setManaged(true); + } + startReminder(); + Platform.runLater(this::askDecision); + } + + /** + * Whether the import skips rows with errors: its transaction is committed (or confirmed) with the errors + */ + public void setSkipErrors(boolean skipErrors) { + this.skipErrors = skipErrors; + } + + /** + * Interval of the reminders, sqljam.import.confirm-commit.reminder-minutes by default + */ + void setReminderInterval(Duration reminderInterval) { + this.reminderInterval = reminderInterval; + } + + private void startReminder() { + pendingSince = System.currentTimeMillis(); + Duration interval = reminderInterval != null ? reminderInterval + : Duration.minutes(context.getSettings().getConfirmCommitReminderMinutes()); + if (interval.lessThanOrEqualTo(Duration.ZERO)) { + return; + } + reminder = new Timeline(new KeyFrame(interval, event -> remind())); + reminder.setCycleCount(Timeline.INDEFINITE); + reminder.play(); + } + + /** + * Each reminder is logged, the first one is also shown without blocking the buttons of the window + */ + private void remind() { + if (pendingTransaction == null) { + return; + } + long minutes = Math.round((System.currentTimeMillis() - pendingSince) / 60000.0); + addLog(Messages.format("progress.reminder.log", minutes), false); + statusLabel.setText(Messages.get("progress.pending") + " (" + minutes + " min)"); + if (!reminded) { + reminded = true; + Alert alert = new Alert(Alert.AlertType.WARNING, Messages.format("progress.reminder.message", minutes)); + Dialogs.initOwner(alert, stage); + Branding.applyIcons(alert); + alert.setTitle(Messages.get("progress.reminder.title")); + alert.setHeaderText(Messages.get("progress.reminder.title")); + alert.getDialogPane().setId("reminderPane"); + alert.show(); + } + } + + /** + * One transaction without confirm: committed when the import succeeds, rolled back when it has errors which + * were not skipped + */ + private void completeTransaction(ManualTransaction transaction, String summary, Runnable onCompleted) { + if (StringUtils.isNotBlank(summary)) { + addLog(summary, false); + } + pendingTransaction = transaction; + pendingCompleted = onCompleted; + automatic = true; + decide(!transaction.isRolledBack() && (errorCount == 0 || skipErrors), false); + } + + /** + * Asks the user to commit or roll back, closing the question keeps both buttons in the window + */ + private void askDecision() { + if (pendingTransaction == null || !stage.isShowing()) { + return; + } + ButtonType commit = new ButtonType(Messages.get("progress.commit"), ButtonBar.ButtonData.OK_DONE); + ButtonType rollback = new ButtonType(Messages.get("progress.rollback"), ButtonBar.ButtonData.OTHER); + Alert alert = new Alert(Alert.AlertType.CONFIRMATION, Messages.format("progress.decide.message", + stage.getTitle()), commit, rollback, ButtonType.CANCEL); + Dialogs.initOwner(alert, stage); + Branding.applyIcons(alert); + alert.setTitle(Messages.get("progress.decide.title")); + alert.setHeaderText(Messages.get("progress.decide.title")); + alert.getDialogPane().setId("decisionPane"); + ((Button) alert.getDialogPane().lookupButton(ButtonType.CANCEL)).setText(Messages.get("progress.decide.later")); + alert.showAndWait().ifPresent(type -> { + if (type == commit) { + decide(true); + } else if (type == rollback) { + decide(false); + } + }); + } + + private void decide(boolean commit) { + decide(commit, false); + } + + /** + * Commits or rolls back the pending transaction in background, the window is closed afterwards if asked + */ + private void decide(boolean commit, boolean close) { + ManualTransaction transaction = pendingTransaction; + if (transaction == null || transaction.isFinished()) { + return; + } + commitButton.setDisable(true); + rollbackButton.setDisable(true); + statusLabel.setText(Messages.get(commit ? "progress.committing" : "progress.rollingBack")); + // Skipped rows do not roll back the import, the user may still do it + boolean errors = transaction.isRolledBack() || (errorCount > 0 && !skipErrors); + int skippedErrors = errorCount; + TaskRunner.run(() -> { + try { + if (commit) { + return transaction.commit(); + } + transaction.rollback(); + return Collections.emptyList(); + } finally { + transaction.close(); + } + }, failures -> { + failures.forEach(failure -> addLog(failure, true)); + errorCount += failures.size(); + String status; + if (!commit) { + status = Messages.get(errors ? "progress.rolledBackErrors" : "progress.rolledBack"); + } else if (!failures.isEmpty()) { + status = Messages.format("progress.committedWithErrors", failures.size()); + } else if (skippedErrors > 0) { + // Rows with errors were skipped, the others are committed + status = Messages.format(automatic ? "progress.completedWithErrors" : "progress.committedSkipped", + skippedErrors); + } else { + status = Messages.get(automatic ? "progress.completed" : "progress.committed"); + } + completeDecision(status, !commit, close); + }, e -> { + errorCount++; + addLog(StringUtils.defaultIfBlank(rootMessage(e), e.getClass().getName()), true); + completeDecision(Messages.get("progress.failed"), true, close); + }); + } + + private void completeDecision(String status, boolean failed, boolean close) { + if (reminder != null) { + reminder.stop(); + reminder = null; + } + Runnable onCompleted = pendingCompleted; + pendingTransaction = null; + pendingCompleted = null; + statusLabel.getStyleClass().remove("status-pending"); + for (Button button : new Button[]{commitButton, rollbackButton}) { + button.setVisible(false); + button.setManaged(false); + } + cancelButton.setVisible(true); + cancelButton.setManaged(true); + // A roll back by the user is not an error of the import + finish(status, failed, failed && errorCount == 0, null, onCompleted); + if (close) { + stage.close(); + } + } + + /** + * Whether the import waits for the user to commit or roll back + */ + boolean isPending() { + return pendingTransaction != null; + } + private boolean isCancellation(Throwable e) { for (Throwable cause = e; cause != null; cause = cause.getCause()) { if (cause instanceof ExportCancelledException) { @@ -379,6 +751,9 @@ private void finish(String status, boolean failed, boolean cancelled, String sum statusLabel.setText(status); statusLabel.getStyleClass().add(failed ? "status-error" : "status-success"); if (!failed) { + // Scripts of a package are imported file by file, every table is complete at the end + completedTables = Math.max(completedTables, totalTables); + updateOverall(); updatePercent(1, 1); } else if (overallBar.getProgress() < 0) { overallBar.setProgress(0); diff --git a/src/main/java/com/github/sqljam/face/view/TransactionOption.java b/src/main/java/com/github/sqljam/face/view/TransactionOption.java new file mode 100644 index 0000000..b0a002d --- /dev/null +++ b/src/main/java/com/github/sqljam/face/view/TransactionOption.java @@ -0,0 +1,175 @@ +/* + * Copyright 2023-2026 Fred Feng + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.github.sqljam.face.view; + +import org.apache.commons.lang3.StringUtils; +import com.github.sqljam.face.model.ConnectionProfile; +import javafx.beans.property.ObjectProperty; +import javafx.beans.value.ObservableValue; +import javafx.geometry.Pos; +import javafx.scene.Node; +import javafx.scene.control.CheckBox; +import javafx.scene.control.Label; +import javafx.scene.control.RadioButton; +import javafx.scene.control.Spinner; +import javafx.scene.control.ToggleGroup; +import javafx.scene.control.Tooltip; +import javafx.scene.layout.HBox; +import javafx.scene.layout.VBox; + +/** + * @Description: TransactionOption is the transaction of an import into a database, like the import wizards of other + * tools: one transaction which is rolled back on an error (the default), or a commit every N rows, + * N is typed in the option. + * Confirm before commit asks the user to commit or roll back a successful import in one transaction, + * commits every N rows are not confirmed. Skip errors logs and skips the rows with errors, otherwise an + * error rolls back one transaction or stops the commits every N rows. Databases without transactions + * (ClickHouse) commit every row as it is written. + * @Author: Fred Feng + * @Date: 26/03/2023 + * @Version 1.0.0 + */ +final class TransactionOption { + + static final int MAX_COMMIT_ROWS = 10_000_000; + + private final RadioButton oneTransactionRadio = new RadioButton(Messages.get("import.transaction.one")); + private final RadioButton commitEveryRadio = new RadioButton(Messages.get("import.transaction.every")); + private final Spinner commitRowsSpinner = new Spinner<>(1, MAX_COMMIT_ROWS, 1, 1000); + private final CheckBox confirmCheck = new CheckBox(Messages.get("import.transaction.confirm")); + private final CheckBox skipErrorsCheck = new CheckBox(Messages.get("import.transaction.skipErrors")); + private final Label unsupportedLabel = new Label(); + private final VBox node; + private boolean oneTransactionChosen; + + /** + * @param target target data source of the dialog + * @param commitRows rows per commit of the import by default, the page or batch size + */ + TransactionOption(AppContext context, ObservableValue target, long commitRows) { + oneTransactionRadio.setId("oneTransactionRadio"); + commitEveryRadio.setId("commitEveryRadio"); + commitRowsSpinner.setId("commitRowsSpinner"); + commitRowsSpinner.setEditable(true); + commitRowsSpinner.setPrefWidth(110); + confirmCheck.setId("confirmCommitCheck"); + skipErrorsCheck.setId("skipErrorsCheck"); + unsupportedLabel.setId("transactionUnsupportedLabel"); + ToggleGroup group = new ToggleGroup(); + oneTransactionRadio.setToggleGroup(group); + commitEveryRadio.setToggleGroup(group); + oneTransactionRadio.setTooltip(new Tooltip(Messages.get("import.transaction.one.tooltip"))); + confirmCheck.setTooltip(new Tooltip(Messages.get("import.transaction.confirm.tooltip"))); + skipErrorsCheck.setTooltip(new Tooltip(Messages.get("import.transaction.skipErrors.tooltip"))); + setCommitRows(commitRows); + oneTransactionChosen = context.getSettings().isOneTransaction(); + (oneTransactionChosen ? oneTransactionRadio : commitEveryRadio).setSelected(true); + unsupportedLabel.getStyleClass().add("muted"); + unsupportedLabel.setWrapText(true); + group.selectedToggleProperty().addListener((obs, oldToggle, toggle) -> { + if (!oneTransactionRadio.isDisabled()) { + oneTransactionChosen = toggle == oneTransactionRadio; + } + update(); + }); + target.addListener((obs, oldProfile, profile) -> applyTarget(profile)); + Label rowsLabel = new Label(Messages.get("import.transaction.rows")); + HBox commitEveryBox = new HBox(6, commitEveryRadio, commitRowsSpinner, rowsLabel); + commitEveryBox.setAlignment(Pos.CENTER_LEFT); + // One choice a line, the labels stay readable in narrow dialogs + node = new VBox(6, oneTransactionRadio, commitEveryBox, confirmCheck, skipErrorsCheck, unsupportedLabel); + applyTarget(target.getValue()); + } + + /** + * Rows per commit of the option, e.g. after the page size is changed + */ + void setCommitRows(long commitRows) { + commitRowsSpinner.getValueFactory().setValue((int) Math.max(1, Math.min(MAX_COMMIT_ROWS, commitRows))); + } + + /** + * Shows rows per commit typed elsewhere, e.g. the page size, without changing the value yet + */ + void showCommitRows(int commitRows) { + commitRowsSpinner.getEditor().setText(String.valueOf(commitRows)); + } + + /** + * Rows per commit, also while they are typed + */ + int getCommitRows() { + String text = StringUtils.trim(commitRowsSpinner.getEditor().getText()); + if (StringUtils.isNumeric(text) && !text.isEmpty() && text.length() < 9) { + return Math.max(1, Math.min(MAX_COMMIT_ROWS, Integer.parseInt(text))); + } + return commitRowsSpinner.getValue(); + } + + /** + * Value of the rows per commit, e.g. bound to the page size of an export + */ + ObjectProperty commitRowsProperty() { + return commitRowsSpinner.getValueFactory().valueProperty(); + } + + private void applyTarget(ConnectionProfile profile) { + boolean supported = profile == null || profile.getDbType().isTransactionSupported(); + oneTransactionRadio.setDisable(!supported); + commitEveryRadio.setDisable(!supported); + (supported && oneTransactionChosen ? oneTransactionRadio : commitEveryRadio).setSelected(true); + unsupportedLabel.setText(supported ? null : Messages.format("import.transaction.unsupported", + profile.getDbType().getDisplayName())); + unsupportedLabel.setVisible(!supported); + unsupportedLabel.setManaged(!supported); + update(); + } + + private void update() { + boolean oneTransaction = isOneTransaction(); + confirmCheck.setDisable(!oneTransaction); + commitRowsSpinner.setDisable(oneTransaction || commitEveryRadio.isDisabled()); + // Commits every N rows are not confirmed + if (!oneTransaction) { + confirmCheck.setSelected(false); + } + } + + Node getNode() { + return node; + } + + /** + * Whether the import runs in one transaction, rolled back on an error + */ + boolean isOneTransaction() { + return oneTransactionRadio.isSelected() && !oneTransactionRadio.isDisabled(); + } + + /** + * Whether the user commits or rolls back a successful import in one transaction + */ + boolean isConfirmCommit() { + return isOneTransaction() && confirmCheck.isSelected(); + } + + /** + * Whether rows with errors are skipped, otherwise an error rolls back one transaction or stops the import + */ + boolean isSkipErrors() { + return skipErrorsCheck.isSelected(); + } +} diff --git a/src/main/java/com/github/sqljam/impexp/AbstractFileImporter.java b/src/main/java/com/github/sqljam/impexp/AbstractFileImporter.java index 69b7434..d5e9474 100644 --- a/src/main/java/com/github/sqljam/impexp/AbstractFileImporter.java +++ b/src/main/java/com/github/sqljam/impexp/AbstractFileImporter.java @@ -36,6 +36,10 @@ public abstract class AbstractFileImporter { protected final Connection connection; protected final DbType dbType; protected boolean stopOnError = true; + /** + * Whether objects of scripts may exist: packages of tables which are not dropped + */ + protected boolean existingSkipped; protected int batchSize = SqlScriptRunner.DEFAULT_BATCH_SIZE; protected ExportListener exportListener = ExportListener.NONE; protected File workspaceDirectory; @@ -54,7 +58,8 @@ protected AbstractFileImporter(Connection connection, DbType dbType) { /** * Importer of an export directory by the data format of its manifest: Parquet packages are imported by - * {@link ParquetImporter}, sql scripts (with or without manifest) by {@link ScriptImporter} + * {@link ParquetImporter}, CSV packages by {@link CsvImporter}, sql scripts (with or without manifest) by + * {@link ScriptImporter} */ public static AbstractFileImporter forDirectory(File dir, Connection connection, DbType dbType) throws IOException { @@ -62,6 +67,9 @@ public static AbstractFileImporter forDirectory(File dir, Connection connection, if (manifest != null && manifest.getDataFormat() == DataFormat.PARQUET) { return new ParquetImporter(connection, dbType); } + if (manifest != null && manifest.getDataFormat() == DataFormat.CSV) { + return new CsvImporter(connection, dbType); + } return new ScriptImporter(connection, dbType); } @@ -118,6 +126,8 @@ public void importDirectory(File dir) throws IOException, SQLException { try { validateManifest(dir, manifest); startProgress(getManifestBytes(dir, manifest)); + // Tables which are not dropped exist, their tables, indexes and constraints are skipped + existingSkipped = Boolean.FALSE.equals(manifest.getOptions().get("tableRecreated")); importManifest(dir, manifest); successful = true; } finally { @@ -210,6 +220,9 @@ protected void runScript(File file) throws IOException, SQLException { exportListener.onMessage("Executing script: " + file.getName()); SqlScriptRunner runner = new SqlScriptRunner(connection); runner.setStopOnError(stopOnError); + if (existingSkipped) { + runner.setSkippedError(dbType.createDialect()::isAlreadyExistsError); + } runner.setBatchSize(batchSize); long fileBytes = file.length(); runner.setExportListener(new ForwardingListener(exportListener) { diff --git a/src/main/java/com/github/sqljam/impexp/CsvExporter.java b/src/main/java/com/github/sqljam/impexp/CsvExporter.java new file mode 100644 index 0000000..646412a --- /dev/null +++ b/src/main/java/com/github/sqljam/impexp/CsvExporter.java @@ -0,0 +1,126 @@ +/* + * Copyright 2023-2026 Fred Feng + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.github.sqljam.impexp; + +import java.io.File; +import java.nio.charset.StandardCharsets; +import java.util.List; +import java.util.Map; + +import org.apache.commons.lang3.StringUtils; + +/** + * @Description: CsvExporter exports tables as an export package with CSV files: schema.sql and constraints.sql for + * the target database (the same as SQL packages), data/<table>.csv with a header row for rows and + * manifest.json. CSV files have no types, manifest.json keeps the type of each column, so that + * {@link CsvImporter} reads the values back exactly. NULL is an empty field, an empty string is "". + * @Author: Fred Feng + * @Date: 26/03/2023 + * @Version 1.0.0 + */ +public final class CsvExporter extends DataFileExporter { + + public static final String CSV_EXTENSION = ".csv"; + /** + * Delimiters offered by the user interface, tab separated files are opened by spreadsheets as well + */ + public static final String[] DELIMITERS = {",", ";", "\t", "|"}; + + private CsvOptions options = new CsvOptions(); + private long maxFileSize; + + public CsvExporter(File root) { + super(root); + } + + /** + * Maximum size of a CSV file in bytes, 0 for no limit. Larger files are split like data files of sql scripts: + * orders.csv, orders_2.csv, orders_3.csv ... each with the header row. + */ + public void setMaxFileSize(long maxFileSize) { + this.maxFileSize = Math.max(0, maxFileSize); + } + + public long getMaxFileSize() { + return maxFileSize; + } + + /** + * Delimiter of the fields, a comma by default + */ + public void setDelimiter(String delimiter) { + options.setDelimiter(StringUtils.defaultIfEmpty(delimiter, DuckDBWorkspace.DEFAULT_DELIMITER)); + } + + public String getDelimiter() { + return options.getDelimiter(); + } + + /** + * Delimiter, quote, text of NULL, header, encoding and date format of the files + */ + public void setOptions(CsvOptions options) { + this.options = options != null ? options : new CsvOptions(); + } + + public CsvOptions getOptions() { + return options; + } + + @Override + protected DataFormat getDataFormat() { + return DataFormat.CSV; + } + + @Override + protected String getExtension() { + return CSV_EXTENSION; + } + + @Override + protected ExportManifest.FileType getFileType() { + return ExportManifest.FileType.CSV; + } + + @Override + protected List writeFile(DuckDBWorkspace workspace, String schema, String table, File file, + Map metadata) throws Exception { + workspace.writeCsv(schema, table, file, options); + // Parts are split as UTF-8 and converted to the encoding afterwards + List parts = CsvSplitter.split(file, maxFileSize, options.getQuote().charAt(0), options.isHeader()); + if (!options.isUtf8()) { + for (File part : parts) { + CsvSplitter.transcode(part, StandardCharsets.UTF_8, part, options.getCharset(), options.isBom()); + } + } + return parts; + } + + @Override + protected void describeTable(ExportManifest.TableEntry tableEntry, DuckDBWorkspace workspace, String schema, + String table) throws Exception { + tableEntry.getColumns().clear(); + for (String[] column : workspace.describeTable(schema, table)) { + tableEntry.getColumns().add(new ExportManifest.ColumnEntry(column[0], column[1])); + } + } + + @Override + protected void putOptions(Map options) { + options.putAll(this.options.toMap()); + options.put("maxFileSize", maxFileSize); + } +} diff --git a/src/main/java/com/github/sqljam/impexp/CsvImporter.java b/src/main/java/com/github/sqljam/impexp/CsvImporter.java new file mode 100644 index 0000000..3351cfe --- /dev/null +++ b/src/main/java/com/github/sqljam/impexp/CsvImporter.java @@ -0,0 +1,91 @@ +/* + * Copyright 2023-2026 Fred Feng + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.github.sqljam.impexp; + +import java.io.File; +import java.io.IOException; +import java.sql.Connection; +import java.sql.SQLException; +import java.util.List; + +/** + * @Description: CsvImporter imports CSV files with a header row into a database by a connection: packages exported + * by {@link CsvExporter}, whose values are read with the column types of manifest.json, or CSV files + * of other tools, whose delimiter and column types are detected, into a new table or appended to an + * existing table + * @Author: Fred Feng + * @Date: 26/03/2023 + * @Version 1.0.0 + */ +public class CsvImporter extends DataFileImporter { + + /** + * Options of the files, null to detect them (files of other tools) + */ + private CsvOptions options; + + public CsvImporter(Connection connection, DbType dbType) { + super(connection, dbType); + } + + /** + * Delimiter of the fields, detected from the files by default + */ + public void setDelimiter(String delimiter) { + if (options == null) { + options = new CsvOptions(); + } + options.setDelimiter(delimiter); + } + + /** + * Options of files of other tools, an empty delimiter is detected + */ + public void setOptions(CsvOptions options) { + this.options = options; + } + + @Override + protected DataFormat getDataFormat() { + return DataFormat.CSV; + } + + @Override + protected ExportManifest.FileType getFileType() { + return ExportManifest.FileType.CSV; + } + + /** + * Rows of the files read with their options, e.g. an encoding or no header + */ + @Override + protected long countRows(DuckDBWorkspace workspace, List files) throws SQLException, IOException { + return workspace.countFileRows(files, options); + } + + @Override + protected void prepare(ExportManifest manifest) { + options = CsvOptions.fromMap(manifest.getOptions()); + } + + @Override + protected void load(DuckDBWorkspace workspace, String schema, String table, List files, + List identityColumns, ExportManifest.TableEntry tableEntry) throws SQLException, + IOException { + workspace.loadCsv(schema, table, files, tableEntry != null ? tableEntry.getColumns() : null, identityColumns, + options); + } +} diff --git a/src/main/java/com/github/sqljam/impexp/CsvOptions.java b/src/main/java/com/github/sqljam/impexp/CsvOptions.java new file mode 100644 index 0000000..79cc4eb --- /dev/null +++ b/src/main/java/com/github/sqljam/impexp/CsvOptions.java @@ -0,0 +1,140 @@ +/* + * Copyright 2023-2026 Fred Feng + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.github.sqljam.impexp; + +import java.nio.charset.Charset; +import java.nio.charset.StandardCharsets; +import java.util.LinkedHashMap; +import java.util.Map; +import java.util.Objects; + +import org.apache.commons.lang3.StringUtils; +import lombok.Getter; +import lombok.Setter; + +/** + * @Description: CsvOptions of CSV files like the CSV settings of spreadsheets: delimiter, quote character (doubled + * inside quoted fields), text of NULL, header row, encoding (UTF-8 with BOM is opened by Excel) and + * formats of dates and timestamps. The defaults (comma, double quote, an empty field for NULL, header, + * UTF-8 and ISO dates) keep every value exactly, other formats of timestamps may drop fractions of + * seconds. The options of an export are kept in manifest.json and used by its import. + * @Author: Fred Feng + * @Date: 26/03/2023 + * @Version 1.0.0 + */ +@Getter +@Setter +public class CsvOptions { + + public static final String UTF_8_BOM = "UTF-8 (BOM)"; + /** + * Encodings offered by the user interface, any encoding of Java can be used + */ + public static final String[] ENCODINGS = {"UTF-8", UTF_8_BOM, "GBK", "GB18030", "UTF-16", "ISO-8859-1"}; + public static final String[] QUOTES = {"\"", "'"}; + /** + * Date formats offered by the user interface (strftime of DuckDB), empty for ISO dates + */ + public static final String[] DATE_FORMATS = {"", "%Y/%m/%d", "%d.%m.%Y", "%m/%d/%Y", "%d/%m/%Y"}; + + private String delimiter = DuckDBWorkspace.DEFAULT_DELIMITER; + private String quote = "\""; + /** + * Text of NULL, an empty field by default (an empty string is quoted) + */ + private String nullString = ""; + private boolean header = true; + private String encoding = "UTF-8"; + /** + * Format of dates, empty for ISO dates (2024-01-31), timestamps use it with the time (%H:%M:%S) + */ + private String dateFormat = ""; + + /** + * Charset of the encoding, UTF-8 with BOM is UTF-8 + */ + public Charset getCharset() { + return isBom() ? StandardCharsets.UTF_8 : Charset.forName(StringUtils.defaultIfBlank(encoding, "UTF-8")); + } + + public boolean isBom() { + return UTF_8_BOM.equals(encoding); + } + + /** + * Whether DuckDB reads and writes the files directly, other encodings are converted + */ + public boolean isUtf8() { + return getCharset().equals(StandardCharsets.UTF_8) && !isBom(); + } + + /** + * Format of timestamps: the date format with the time, empty for ISO timestamps + */ + public String getTimestampFormat() { + return StringUtils.isEmpty(dateFormat) ? "" : dateFormat + " %H:%M:%S"; + } + + /** + * Options of manifest.json + */ + public Map toMap() { + Map map = new LinkedHashMap<>(); + map.put("delimiter", delimiter); + map.put("quote", quote); + map.put("nullString", nullString); + map.put("header", header); + map.put("encoding", encoding); + map.put("dateFormat", dateFormat); + return map; + } + + /** + * Options of an export package, missing options of older packages keep their defaults + */ + public static CsvOptions fromMap(Map map) { + CsvOptions options = new CsvOptions(); + if (map.get("delimiter") != null) { + options.setDelimiter(map.get("delimiter").toString()); + } + if (map.get("quote") != null) { + options.setQuote(map.get("quote").toString()); + } + if (map.get("nullString") != null) { + options.setNullString(map.get("nullString").toString()); + } + if (map.get("header") != null) { + options.setHeader(Boolean.parseBoolean(map.get("header").toString())); + } + if (map.get("encoding") != null) { + options.setEncoding(map.get("encoding").toString()); + } + if (map.get("dateFormat") != null) { + options.setDateFormat(map.get("dateFormat").toString()); + } + return options; + } + + @Override + public boolean equals(Object other) { + return other instanceof CsvOptions && toMap().equals(((CsvOptions) other).toMap()); + } + + @Override + public int hashCode() { + return Objects.hash(toMap()); + } +} diff --git a/src/main/java/com/github/sqljam/impexp/CsvSplitter.java b/src/main/java/com/github/sqljam/impexp/CsvSplitter.java new file mode 100644 index 0000000..2079790 --- /dev/null +++ b/src/main/java/com/github/sqljam/impexp/CsvSplitter.java @@ -0,0 +1,221 @@ +/* + * Copyright 2023-2026 Fred Feng + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.github.sqljam.impexp; + +import java.io.BufferedInputStream; +import java.io.BufferedOutputStream; +import java.io.BufferedReader; +import java.io.BufferedWriter; +import java.io.ByteArrayOutputStream; +import java.io.File; +import java.io.IOException; +import java.io.InputStream; +import java.io.InputStreamReader; +import java.io.OutputStream; +import java.io.OutputStreamWriter; +import java.io.Reader; +import java.io.Writer; +import java.nio.charset.CharacterCodingException; +import java.nio.charset.Charset; +import java.nio.charset.CodingErrorAction; +import java.nio.file.Files; +import java.nio.file.StandardCopyOption; +import java.util.ArrayList; +import java.util.Comparator; +import java.util.List; +import java.util.regex.Matcher; +import java.util.regex.Pattern; + +import org.apache.commons.io.FilenameUtils; + +/** + * @Description: CsvSplitter splits a large CSV file into parts of a maximum size, named like data files of sql + * scripts: orders.csv, orders_2.csv, orders_3.csv ... Every part starts with the header row and ends + * with a complete record, a line break inside a quoted field does not end a record. Quotes and line + * breaks are single bytes in UTF-8, so the file is split as bytes. + * @Author: Fred Feng + * @Date: 26/03/2023 + * @Version 1.0.0 + */ +public final class CsvSplitter { + + /** + * Order of importing: the file itself, then its parts by number (orders.csv, orders_2.csv ... orders_10.csv), + * files of different names by name + */ + public static final Comparator PART_ORDER = Comparator.comparing((File file) -> baseOf(file)) + .thenComparingInt(CsvSplitter::partOf).thenComparing(File::getName); + private static final Pattern PART = Pattern.compile("^(.*)_(\\d+)$"); + + private static final int LINE_FEED = '\n'; + + private CsvSplitter() { + } + + /** + * Splits the file if it is larger than the maximum size, the file keeps the first part + * + * @param maxFileSize maximum size of a part in bytes, 0 or less keeps the file as it is + * @return the file and its further parts + */ + public static List split(File file, long maxFileSize) throws IOException { + return split(file, maxFileSize, '"', true); + } + + /** + * Splits the file if it is larger than the maximum size, the file keeps the first part + * + * @param quote quote character of the fields, a line break inside quotes does not end a record + * @param header whether every part starts with the header row of the file + */ + public static List split(File file, long maxFileSize, char quote, boolean header) throws IOException { + List parts = new ArrayList<>(); + parts.add(file); + if (maxFileSize <= 0 || file.length() <= maxFileSize) { + return parts; + } + File source = new File(file.getParentFile(), file.getName() + ".split"); + Files.move(file.toPath(), source.toPath()); + String baseName = FilenameUtils.getBaseName(file.getName()); + String extension = "." + FilenameUtils.getExtension(file.getName()); + try (InputStream in = new BufferedInputStream(Files.newInputStream(source.toPath()), 1 << 16)) { + byte[] headerRow = header ? readRecord(in, quote) : new byte[0]; + if (headerRow == null) { + headerRow = new byte[0]; + } + OutputStream out = open(file, headerRow); + long size = headerRow.length; + boolean empty = true; + try { + byte[] record; + while ((record = readRecord(in, quote)) != null) { + // A part holds at least one record, also when the record is larger than the maximum size + if (!empty && size + record.length > maxFileSize) { + out.close(); + File part = new File(file.getParentFile(), baseName + "_" + (parts.size() + 1) + extension); + parts.add(part); + out = open(part, headerRow); + size = headerRow.length; + } + out.write(record); + size += record.length; + empty = false; + } + } finally { + out.close(); + } + } finally { + Files.deleteIfExists(source.toPath()); + } + return parts; + } + + /** + * Files in the order of importing + */ + public static List sortParts(List files) { + List sorted = new ArrayList<>(files); + sorted.sort(PART_ORDER); + return sorted; + } + + private static String baseOf(File file) { + String baseName = FilenameUtils.getBaseName(file.getName()); + Matcher matcher = PART.matcher(baseName); + return matcher.matches() ? matcher.group(1) : baseName; + } + + /** + * Number of the part, 1 for the file without a number + */ + private static int partOf(File file) { + Matcher matcher = PART.matcher(FilenameUtils.getBaseName(file.getName())); + if (matcher.matches()) { + try { + return Integer.parseInt(matcher.group(2)); + } catch (NumberFormatException e) { + return Integer.MAX_VALUE; + } + } + return 1; + } + + /** + * Converts a text file between encodings, a byte order mark of UTF-8 is removed from the input and written to + * the output if asked (Excel recognizes UTF-8 by it). Characters which the encoding can not hold, or bytes which + * are not of the source encoding, stop the conversion instead of being replaced. + */ + public static void transcode(File source, Charset from, File target, Charset to, boolean bom) throws IOException { + File output = source.equals(target) ? new File(target.getPath() + ".encoding") : target; + try (Reader reader = new BufferedReader(new InputStreamReader(Files.newInputStream(source.toPath()), + from.newDecoder().onMalformedInput(CodingErrorAction.REPORT).onUnmappableCharacter( + CodingErrorAction.REPORT)), 1 << 16); + Writer writer = new BufferedWriter(new OutputStreamWriter(Files.newOutputStream(output.toPath()), + to.newEncoder().onMalformedInput(CodingErrorAction.REPORT).onUnmappableCharacter( + CodingErrorAction.REPORT)), 1 << 16)) { + if (bom) { + writer.write('\uFEFF'); + } + copy(reader, writer); + } catch (CharacterCodingException e) { + Files.deleteIfExists(output.toPath()); + throw new ImpExpException(String.format("%s can not be converted from %s to %s, some characters are " + + "not of these encodings, please choose UTF-8", source.getName(), from.name(), to.name()), e); + } + if (output != target) { + Files.move(output.toPath(), target.toPath(), StandardCopyOption.REPLACE_EXISTING); + } + } + + /** + * Copies the text, a byte order mark at the start of the input is dropped + */ + private static void copy(Reader reader, Writer writer) throws IOException { + char[] buffer = new char[1 << 15]; + boolean first = true; + int n; + while ((n = reader.read(buffer)) != -1) { + int offset = first && n > 0 && buffer[0] == '\uFEFF' ? 1 : 0; + first = false; + writer.write(buffer, offset, n - offset); + } + } + + private static OutputStream open(File file, byte[] header) throws IOException { + OutputStream out = new BufferedOutputStream(Files.newOutputStream(file.toPath()), 1 << 16); + out.write(header); + return out; + } + + /** + * Reads a record with its line break, null at the end of the input + */ + private static byte[] readRecord(InputStream in, char quote) throws IOException { + ByteArrayOutputStream record = new ByteArrayOutputStream(256); + boolean quoted = false; + int b; + while ((b = in.read()) != -1) { + record.write(b); + if (b == quote) { + // An escaped quote ("") toggles twice + quoted = !quoted; + } else if (b == LINE_FEED && !quoted) { + return record.toByteArray(); + } + } + return record.size() > 0 ? record.toByteArray() : null; + } +} diff --git a/src/main/java/com/github/sqljam/impexp/DataFileExporter.java b/src/main/java/com/github/sqljam/impexp/DataFileExporter.java new file mode 100644 index 0000000..76c5873 --- /dev/null +++ b/src/main/java/com/github/sqljam/impexp/DataFileExporter.java @@ -0,0 +1,438 @@ +/* + * Copyright 2023-2026 Fred Feng + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.github.sqljam.impexp; + +import java.io.File; +import java.lang.reflect.Field; +import java.lang.reflect.Modifier; +import java.sql.DatabaseMetaData; +import java.util.ArrayList; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; + +import org.apache.commons.lang3.StringUtils; +import com.github.sqljam.impexp.db.DuckDBDialect; +import com.github.sqljam.impexp.db.DuckDBMetaDataOperations; +import com.github.sqljam.jdbc.SharedConnectionFactory; +import lombok.extern.slf4j.Slf4j; + +/** + * @Description: DataFileExporter exports tables as an export package with data files: schema.sql and + * constraints.sql for the target database (the same as SQL packages), one data file per table in + * data/ and manifest.json. Rows are copied into a temporary DuckDB workspace first (values which the + * file format can not hold exactly are kept as text), then written by DuckDB in the format of the + * subclass: {@link ParquetExporter} or {@link CsvExporter}. + * @Author: Fred Feng + * @Date: 26/03/2023 + * @Version 1.0.0 + */ +@Slf4j +public abstract class DataFileExporter { + + private final File root; + private final Exporter.ExportConfiguration configuration = new Exporter.ExportConfiguration(); + private ExportListener exportListener = ExportListener.NONE; + private DbType targetDbType; + private IdentifierCase identifierCase; + private String targetCatalogName; + private String targetSchemaName; + private int targetMajorVersion = -1; + private int targetMinorVersion = -1; + private final List writtenFiles = new ArrayList<>(); + + protected DataFileExporter(File root) { + this.root = root; + configuration.setShowCreateUserSql(false); + configuration.setShowCreateCatalogSql(false); + configuration.setShowCreateSchemaSql(false); + } + + /** + * Source database and options, the same as {@link ScriptExporter#getConfiguration()} + */ + public Exporter.ExportConfiguration getConfiguration() { + return configuration; + } + + public void setExportListener(ExportListener exportListener) { + this.exportListener = exportListener != null ? exportListener : ExportListener.NONE; + } + + /** + * Database type of schema.sql and constraints.sql. Default is the same as source database. + */ + public void setTargetDbType(DbType targetDbType) { + this.targetDbType = targetDbType; + } + + public void setIdentifierCase(IdentifierCase identifierCase) { + this.identifierCase = identifierCase; + } + + public void setTargetCatalogName(String targetCatalogName) { + this.targetCatalogName = targetCatalogName; + } + + public void setTargetSchemaName(String targetSchemaName) { + this.targetSchemaName = targetSchemaName; + } + + public void setTargetVersion(int majorVersion, int minorVersion) { + this.targetMajorVersion = majorVersion; + this.targetMinorVersion = minorVersion; + } + + /** + * Format of the data files recorded in manifest.json + */ + protected abstract DataFormat getDataFormat(); + + /** + * Extension of the data files, e.g. .parquet + */ + protected abstract String getExtension(); + + protected abstract ExportManifest.FileType getFileType(); + + /** + * Writes rows of the workspace table into the data file, a format may split large files into parts + * + * @param metadata key value metadata of the file, formats without metadata ignore it + * @return the written files, the file itself and its further parts + */ + protected abstract List writeFile(DuckDBWorkspace workspace, String schema, String table, File file, + Map metadata) throws Exception; + + /** + * Options of the format in manifest.json, e.g. the compression of Parquet files + */ + protected abstract void putOptions(Map options); + + /** + * Describes the written table in its manifest entry, e.g. column types of CSV files + */ + protected void describeTable(ExportManifest.TableEntry tableEntry, DuckDBWorkspace workspace, String schema, + String table) throws Exception { + } + + public File getRoot() { + return root; + } + + public List getWrittenFiles() { + return new ArrayList<>(writtenFiles); + } + + public void exportDdlAndData() throws Exception { + export(ExportMode.DDL_DATA); + } + + public void exportDdl() throws Exception { + export(ExportMode.DDL); + } + + public void exportData() throws Exception { + export(ExportMode.DATA); + } + + public void export(ExportMode exportMode) throws Exception { + writtenFiles.clear(); + if (configuration.getDbType() == null) { + configuration.setDbType(DbType.forUrl(configuration.getUrl())); + } + boolean successful = false; + exportListener.onStart(exportMode); + PhaseListener phaseListener = new PhaseListener(exportListener); + try { + ExportManifest manifest; + if (exportMode != ExportMode.DATA) { + exportSchema(phaseListener); + manifest = ExportManifest.read(root); + } else { + manifest = null; + } + if (manifest == null) { + manifest = new ExportManifest(); + manifest.getSource().setDbType(configuration.getDbType()); + manifest.getTarget().setDbType(targetDbType != null ? targetDbType : configuration.getDbType()); + } + List dataFiles = new ArrayList<>(); + if (exportMode != ExportMode.DDL) { + exportRows(phaseListener, manifest, dataFiles); + } + writeManifest(manifest, exportMode, dataFiles); + successful = !phaseListener.isFailed(); + } catch (ExportCancelledException e) { + throw e; + } catch (Exception e) { + exportListener.onError(e.getMessage(), e); + throw e; + } finally { + exportListener.onEnd(successful); + } + } + + /** + * schema.sql and constraints.sql by the script exporter + */ + private void exportSchema(PhaseListener listener) throws Exception { + exportListener.onMessage("Export schema"); + ScriptExporter scriptExporter = new ScriptExporter(root, DataFileStrategy.SINGLE_FILE, 0); + copyConfiguration(configuration, scriptExporter.getConfiguration()); + scriptExporter.setTargetDbType(targetDbType); + scriptExporter.setIdentifierCase(identifierCase); + scriptExporter.setTargetCatalogName(targetCatalogName); + scriptExporter.setTargetSchemaName(targetSchemaName); + if (targetMajorVersion >= 0) { + scriptExporter.setTargetVersion(targetMajorVersion, targetMinorVersion); + } + scriptExporter.setConstraintsSeparated(true); + scriptExporter.setExportListener(listener.forPhase(false)); + scriptExporter.exportDdl(); + writtenFiles.addAll(scriptExporter.getWrittenFiles()); + } + + /** + * Rows are copied into the workspace and written as data files, one file per table + */ + private void exportRows(PhaseListener listener, ExportManifest manifest, + List dataFiles) throws Exception { + try (DuckDBWorkspace workspace = new DuckDBWorkspace(root)) { + exportListener.onMessage("Copy rows into workspace: " + workspace.getFile()); + ImportExporter importExporter = new ImportExporter(); + copyConfiguration(configuration, importExporter.getExportConfiguration()); + Exporter.ExportConfiguration source = importExporter.getExportConfiguration(); + source.setIdReused(true); + source.setTableRecreated(true); + source.setIndexIncluded(false); + source.setForeignKeyIncluded(false); + source.setCommentIncluded(false); + // Defaults of the same database type may use sequences + source.setSequenceIncluded(true); + ImportExportHandler.ImportConfiguration target = importExporter.getImportConfiguration(); + target.setDbType(DbType.DUCKDB); + target.setUrl(workspace.getUrl()); + target.setConnectionPoolEnabled(false); + DuckDBDialect dialect = new DuckDBDialect(); + dialect.setExactValues(true); + dialect.setExactTargetDbType(targetDbType != null ? targetDbType : configuration.getDbType()); + // Table names of the source are kept + dialect.setIdentifierCase(IdentifierCase.KEEP); + importExporter.setTargetDialect(dialect); + importExporter.setTargetConnectionFactory(new SharedConnectionFactory(workspace.getConnection())); + importExporter.setExportListener(listener.forPhase(true)); + importExporter.exportDdlAndData(); + if (listener.isFailed()) { + return; + } + + List tables = workspace.getTables(); + boolean schemaQualified = tables.stream().map(t -> t[0]).distinct().count() > 1; + DuckDBMetaDataOperations operations = new DuckDBMetaDataOperations(); + DatabaseMetaData metaData = workspace.getConnection().getMetaData(); + for (String[] table : tables) { + if (exportListener.isCancelled()) { + throw new ExportCancelledException(); + } + String schema = table[0]; + String tableName = table[1]; + String fileName = (schemaQualified ? LobManifestWriter.toFileName(schema) + "." : "") + + LobManifestWriter.toFileName(tableName) + getExtension(); + File file = new File(new File(root, ScriptExportHandler.DATA_DIR_NAME), fileName); + Map fileMetadata = new LinkedHashMap<>(); + fileMetadata.put("sqljam_table", tableName); + if (sourceSchema(schema) != null) { + fileMetadata.put("sqljam_schema", sourceSchema(schema)); + } + fileMetadata.put("sqljam_source", String.valueOf(configuration.getDbType())); + List parts = writeFile(workspace, schema, tableName, file, fileMetadata); + writtenFiles.addAll(parts); + long rows = workspace.countRows(schema, tableName); + exportListener.onMessage(String.format("Write %d rows of %s to %s", rows, tableName, + parts.size() > 1 ? parts.size() + " files: " + file : file)); + + ExportManifest.TableEntry tableEntry = findTableEntry(manifest, sourceSchema(schema), tableName) + .orElseGet(() -> { + ExportManifest.TableEntry entry = new ExportManifest.TableEntry(); + entry.setSchema(sourceSchema(schema)); + entry.setName(tableName); + manifest.getTables().add(entry); + return entry; + }); + tableEntry.setRows(rows); + tableEntry.getDataFiles().clear(); + for (File part : parts) { + String path = ScriptExportHandler.DATA_DIR_NAME + "/" + part.getName(); + tableEntry.getDataFiles().add(path); + dataFiles.add(new ExportManifest.FileEntry(path, getFileType())); + } + describeTable(tableEntry, workspace, schema, tableName); + // Chosen key columns of upserts, the primary key of the table otherwise + tableEntry.getKeyColumns().clear(); + configuration.getKeyColumnNames().entrySet().stream().filter(entry -> entry.getKey() + .equalsIgnoreCase(tableName)).findFirst().ifPresent(entry -> tableEntry.getKeyColumns() + .addAll(entry.getValue())); + tableEntry.getIdentityColumns().clear(); + for (Map column : operations.getColumnInfos(metaData, null, schema, tableName)) { + if ("YES".equalsIgnoreCase((String) column.get("IS_AUTOINCREMENT"))) { + tableEntry.getIdentityColumns().add((String) column.get("COLUMN_NAME")); + } + } + } + } + } + + /** + * Schema of the source table, tables of sources without schemas are in the default schema of the workspace + */ + private String sourceSchema(String workspaceSchema) { + return configuration.getDbType() != null && !configuration.getDbType().isSchemaSupported() + && DuckDBWorkspace.DEFAULT_SCHEMA.equals(workspaceSchema) ? null : workspaceSchema; + } + + private static Optional findTableEntry(ExportManifest manifest, String schema, + String table) { + return manifest.getTables().stream() + .filter(entry -> table.equals(entry.getName()) + && (schema == null || entry.getSchema() == null || schema.equals(entry.getSchema()))) + .findFirst(); + } + + /** + * Files in the order of importing: schema.sql, data files, constraints.sql + */ + private void writeManifest(ExportManifest manifest, ExportMode exportMode, + List dataFiles) throws Exception { + manifest.setDataFormat(getDataFormat()); + manifest.setExportMode(exportMode); + manifest.setStatus(ExportManifest.Status.COMPLETED); + putOptions(manifest.getOptions()); + List files = new ArrayList<>(); + for (ExportManifest.FileEntry entry : manifest.getFiles()) { + if (entry.getType() == ExportManifest.FileType.SCHEMA) { + files.add(entry); + } + } + files.addAll(dataFiles); + for (ExportManifest.FileEntry entry : manifest.getFiles()) { + if (entry.getType() == ExportManifest.FileType.CONSTRAINTS) { + files.add(entry); + } + } + manifest.setFiles(files); + manifest.write(root); + File manifestFile = new File(root, ExportManifest.FILE_NAME); + if (!writtenFiles.contains(manifestFile)) { + writtenFiles.add(manifestFile); + } + exportListener.onMessage("Write manifest: " + manifestFile); + } + + /** + * Copies source settings and options, dialects are created by each exporter + */ + static void copyConfiguration(Exporter.ExportConfiguration from, Exporter.ExportConfiguration to) { + for (Field field : Exporter.ExportConfiguration.class.getDeclaredFields()) { + if (Modifier.isStatic(field.getModifiers()) || "dialect".equals(field.getName()) + || "sourceDialect".equals(field.getName())) { + continue; + } + try { + field.setAccessible(true); + field.set(to, field.get(from)); + } catch (IllegalAccessException e) { + throw new IllegalStateException(e); + } + } + } + + /** + * Forwards events of the schema phase and the rows phase as one export: messages and errors of both phases, + * progress of the rows phase only, start and end are reported by the data file exporter + */ + static class PhaseListener { + + private final ExportListener delegate; + private boolean failed; + + PhaseListener(ExportListener delegate) { + this.delegate = delegate; + } + + boolean isFailed() { + return failed; + } + + ExportListener forPhase(boolean progress) { + return new ExportListener() { + + @Override + public void onTableStart(String catalog, String schema, String table, long totalRows) { + if (progress) { + delegate.onTableStart(catalog, schema, table, totalRows); + } + } + + @Override + public void onTableProgress(String catalog, String schema, String table, long rows, + long totalRows) { + if (progress) { + delegate.onTableProgress(catalog, schema, table, rows, totalRows); + } + } + + @Override + public void onTableEnd(String catalog, String schema, String table, long rows) { + if (progress) { + delegate.onTableEnd(catalog, schema, table, rows); + } + } + + @Override + public void onProgress(long processed, long total) { + if (progress) { + delegate.onProgress(processed, total); + } + } + + @Override + public void onMessage(String message) { + delegate.onMessage(message); + } + + @Override + public void onError(String message, Throwable e) { + failed = true; + delegate.onError(message, e); + } + + @Override + public void onEnd(boolean successful) { + if (!successful) { + failed = true; + } + } + + @Override + public boolean isCancelled() { + return delegate.isCancelled(); + } + }; + } + } +} diff --git a/src/main/java/com/github/sqljam/impexp/DataFileImporter.java b/src/main/java/com/github/sqljam/impexp/DataFileImporter.java new file mode 100644 index 0000000..f2d054b --- /dev/null +++ b/src/main/java/com/github/sqljam/impexp/DataFileImporter.java @@ -0,0 +1,515 @@ +/* + * Copyright 2023-2026 Fred Feng + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.github.sqljam.impexp; + +import java.io.File; +import java.io.IOException; +import java.sql.Connection; +import java.sql.DatabaseMetaData; +import java.sql.SQLException; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashSet; +import java.util.LinkedHashMap; +import java.util.LinkedHashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.stream.Collectors; + +import org.apache.commons.lang3.StringUtils; +import org.apache.commons.lang3.Validate; +import com.github.sqljam.jdbc.SharedConnectionFactory; +import lombok.extern.slf4j.Slf4j; + +/** + * @Description: DataFileImporter imports data files into a database by a connection. Packages exported by + * {@link DataFileExporter} are imported in the order of manifest.json: schema.sql, data files into + * the created tables (identity values are kept and identities are reset), constraints.sql. Files of + * other tools are imported into a new table created from the columns of the files, or appended to an + * existing table. Files are read into a temporary DuckDB workspace, then rows are copied by + * {@link ImportExporter} with the dialect of the target database. The format is read by the + * subclass: {@link ParquetImporter} or {@link CsvImporter}. + * @Author: Fred Feng + * @Date: 26/03/2023 + * @Version 1.0.0 + */ +@Slf4j +public abstract class DataFileImporter extends AbstractFileImporter { + + /** + * CREATE drops and creates the table from the columns of the files, APPEND inserts rows into the existing table, + * UPSERT updates the rows of the same key in the existing table and inserts the others, SKIP_EXISTING inserts + * the rows whose key is missing and keeps the existing rows + */ + public enum Mode { + CREATE, + APPEND, + UPSERT, + SKIP_EXISTING + } + + private String targetCatalogName; + private String targetSchemaName; + private IdentifierCase identifierCase; + private int targetMajorVersion = -1; + private int targetMinorVersion = -1; + private int pageSize = Exporter.DEFAULT_PAGE_SIZE; + private long importedRows; + /** + * Key columns of upserts of files, the primary key of the target table by default + */ + private List keyColumns = Collections.emptyList(); + + protected DataFileImporter(Connection connection, DbType dbType) { + super(connection, dbType); + } + + /** + * Format of the packages imported by this importer + */ + protected abstract DataFormat getDataFormat(); + + protected abstract ExportManifest.FileType getFileType(); + + /** + * Loads the files into a table of the workspace + * + * @param tableEntry the table of the package, null for files of other tools + */ + protected abstract void load(DuckDBWorkspace workspace, String schema, String table, List files, + List identityColumns, ExportManifest.TableEntry tableEntry) + throws SQLException, IOException; + + /** + * Rows of files of other tools, e.g. for the progress + */ + protected long countRows(DuckDBWorkspace workspace, List files) throws SQLException, IOException { + return workspace.countFileRows(files); + } + + /** + * Reads options of the package before its files are imported, e.g. the delimiter of CSV files + */ + protected void prepare(ExportManifest manifest) { + } + + /** + * Catalog of the table, the catalog of the connection by default + */ + public void setTargetCatalogName(String targetCatalogName) { + this.targetCatalogName = targetCatalogName; + } + + /** + * Schema of the table, the schema of the connection by default + */ + public void setTargetSchemaName(String targetSchemaName) { + this.targetSchemaName = targetSchemaName; + } + + public void setIdentifierCase(IdentifierCase identifierCase) { + this.identifierCase = identifierCase; + } + + public void setTargetVersion(int majorVersion, int minorVersion) { + this.targetMajorVersion = majorVersion; + this.targetMinorVersion = minorVersion; + } + + public void setPageSize(int pageSize) { + this.pageSize = pageSize; + } + + /** + * Key columns of upserts of files, empty for the primary key of the target table + */ + public void setKeyColumns(List keyColumns) { + this.keyColumns = keyColumns != null ? keyColumns : Collections.emptyList(); + } + + /** + * Rows imported from data files + */ + public long getImportedRows() { + return importedRows; + } + + /** + * Imports data files of other tools into a table + * + * @return number of imported rows + */ + public long importFiles(List files, String tableName, Mode mode) throws Exception { + Validate.notEmpty(files, "Data files must be required"); + files = CsvSplitter.sortParts(files); + Validate.notBlank(tableName, "Table name must be required"); + importedRows = 0; + boolean successful = false; + exportListener.onStart(mode == Mode.CREATE ? ExportMode.DDL_DATA : ExportMode.DATA); + try (DuckDBWorkspace workspace = new DuckDBWorkspace(workspaceDirectory)) { + long total = countRows(workspace, files); + exportListener.onMessage(String.format("Import %d rows of %d %s files into %s", total, files.size(), + getDataFormat(), tableName)); + importTable(workspace, DuckDBWorkspace.DEFAULT_SCHEMA, tableName, files, Collections.emptyList(), null, + mode, null, null, new ProgressListener(exportListener, 0, total)); + executedCount = (int) Math.min(Integer.MAX_VALUE, importedRows); + successful = true; + } finally { + exportListener.onEnd(successful); + } + return importedRows; + } + + /** + * schema.sql, data files (all together, in the order of foreign keys) and constraints.sql + */ + @Override + protected void importManifest(File dir, ExportManifest manifest) throws IOException, SQLException { + if (manifest.getDataFormat() != getDataFormat()) { + throw new ImpExpException(String.format("Packages of %s files are not imported by %s", + manifest.getDataFormat(), getClass().getSimpleName())); + } + prepare(manifest); + importedRows = 0; + boolean dataImported = false; + for (ExportManifest.FileEntry fileEntry : manifest.getFiles()) { + File file = new File(dir, fileEntry.getPath()); + if (fileEntry.getType() == getFileType()) { + if (!dataImported) { + importPackageFiles(dir, manifest); + dataImported = true; + } + } else { + runScript(file); + } + } + } + + private void importPackageFiles(File dir, ExportManifest manifest) throws SQLException { + long dataBytes = manifest.getFiles().stream() + .filter(entry -> entry.getType() == getFileType()) + .mapToLong(entry -> new File(dir, entry.getPath()).length()).sum(); + long startBytes = completedBytes; + ExportListener bytesListener = new ForwardingListener(exportListener) { + + // Tables of the package are completed one by one in the progress of the import + @Override + public void onTableStart(String catalogName, String schemaName, String tableName, long totalRows) { + exportListener.onTableStart(catalogName, schemaName, tableName, totalRows); + } + + @Override + public void onTableProgress(String catalogName, String schemaName, String tableName, long processedRows, + long totalRows) { + exportListener.onTableProgress(catalogName, schemaName, tableName, processedRows, totalRows); + } + + @Override + public void onTableEnd(String catalogName, String schemaName, String tableName, long processedRows) { + exportListener.onTableEnd(catalogName, schemaName, tableName, processedRows); + } + + @Override + public void onProgress(long processed, long total) { + // Rows are converted to bytes of data files + long bytes = total > 0 ? dataBytes * processed / total : 0; + exportListener.onProgress(Math.min(startBytes + bytes, totalBytes), totalBytes); + } + + @Override + public void onError(String message, Throwable e) { + failedCount++; + exportListener.onError(message, e); + } + }; + List tables = manifest.getTables().stream() + .filter(table -> !table.getDataFiles().isEmpty()).collect(Collectors.toList()); + IdentifierCase manifestCase = getIdentifierCase(manifest); + if (identifierCase == null) { + identifierCase = manifestCase; + } + try { + if (createDialect().isForeignKeyOrderRequired()) { + tables = sortByForeignKeys(tables, manifest); + } + long total = tables.stream().mapToLong(ExportManifest.TableEntry::getRows).sum(); + long completed = 0; + try (DuckDBWorkspace workspace = new DuckDBWorkspace(workspaceDirectory)) { + for (ExportManifest.TableEntry table : tables) { + if (exportListener.isCancelled()) { + throw new ExportCancelledException(); + } + // Parts in order: orders.csv, orders_2.csv ... orders_n.csv + List files = CsvSplitter.sortParts(table.getDataFiles().stream() + .map(path -> new File(dir, path)).collect(Collectors.toList())); + String schema = StringUtils.defaultIfBlank(table.getSchema(), DuckDBWorkspace.DEFAULT_SCHEMA); + exportListener.onMessage(String.format("Import %d rows of %s", table.getRows(), table.getName())); + // Packages of upserts update the rows of the same key, others keep them + WriteMode writeMode = WriteMode.of(manifest.getOptions().get("writeMode")); + Mode mode = writeMode.isKeyed() ? Mode.valueOf(writeMode.name()) : Mode.APPEND; + importTable(workspace, schema, table.getName(), files, table.getIdentityColumns(), table, + mode, manifest.getTarget().getCatalog(), manifest.getTarget().getSchema(), + new ProgressListener(bytesListener, completed, total)); + completed += table.getRows(); + } + } + } catch (ImpExpException | SQLException e) { + throw e; + } catch (Exception e) { + throw new ImpExpException(e.getMessage(), e); + } + executedCount += (int) Math.min(Integer.MAX_VALUE, importedRows); + completedBytes = startBytes; + completeBytes(dataBytes); + } + + private void importTable(DuckDBWorkspace workspace, String schema, String tableName, List files, + List identityColumns, ExportManifest.TableEntry tableEntry, Mode mode, + String packageCatalog, String packageSchema, ProgressListener listener) throws Exception { + load(workspace, schema, tableName, files, identityColumns, tableEntry); + String catalog = StringUtils.defaultIfBlank(targetCatalogName, packageCatalog); + String targetSchema = StringUtils.defaultIfBlank(targetSchemaName, packageSchema); + if (mode != Mode.CREATE) { + // Values of generated and row version columns are maintained by the target database + List generated = getGeneratedColumns(tableName, catalog, targetSchema); + List dropped = new ArrayList<>(); + for (String[] column : workspace.describeTable(schema, tableName)) { + if (generated.stream().anyMatch(name -> name.equalsIgnoreCase(column[0]))) { + dropped.add(column[0]); + } + } + workspace.dropColumns(schema, tableName, dropped); + } + ImportExporter importExporter = new ImportExporter(); + Exporter.ExportConfiguration source = importExporter.getExportConfiguration(); + source.setDbType(DbType.DUCKDB); + source.setUrl(workspace.getUrl()); + source.setConnectionPoolEnabled(false); + source.setIncludedSchemaNames(new String[]{schema}); + source.setIncludedTableNames(new String[]{tableName}); + source.setPageSize(pageSize); + source.setIdReused(true); + source.setTableRecreated(mode == Mode.CREATE); + source.setIndexIncluded(false); + source.setForeignKeyIncluded(false); + source.setCommentIncluded(false); + source.setSequenceIncluded(false); + // Rows with errors are skipped when the import goes on after errors + source.setFailFast(stopOnError); + if (mode == Mode.UPSERT || mode == Mode.SKIP_EXISTING) { + source.setWriteMode(WriteMode.valueOf(mode.name())); + List keys = tableEntry != null ? tableEntry.getKeyColumns() : keyColumns; + if (!keys.isEmpty()) { + source.getKeyColumnNames().put(tableName, keys); + } + } + ImportExportHandler.ImportConfiguration target = importExporter.getImportConfiguration(); + target.setDbType(dbType); + target.setTargetCatalogName(catalog); + target.setTargetSchemaName(targetSchema); + importExporter.setTargetConnectionFactory(new SharedConnectionFactory(connection)); + if (identifierCase != null) { + importExporter.setIdentifierCase(identifierCase); + } + if (targetMajorVersion >= 0) { + importExporter.setTargetVersion(targetMajorVersion, targetMinorVersion); + } + // Tables go into the target schema or the default schema, not into a schema named as the workspace one + importExporter.setSourceSchemaPreserved(false); + importExporter.setExportListener(listener); + // The workspace is read by connections of the importer + workspace.closeConnection(); + importExporter.export(mode == Mode.CREATE ? ExportMode.DDL_DATA : ExportMode.DATA); + // Errors which were skipped are reported already, the import goes on + if (listener.getError() != null && stopOnError) { + throw new ImpExpException(listener.getError(), listener.getCause()); + } + importedRows += listener.getRows(); + } + + /** + * Dialect folding names as the importer does: the workspace (DuckDB) is the source + */ + private Dialect createDialect() { + Dialect dialect = dbType.createDialect(); + dialect.setSourceDbType(DbType.DUCKDB); + if (identifierCase != null) { + dialect.setIdentifierCase(identifierCase); + } + return dialect; + } + + private static IdentifierCase getIdentifierCase(ExportManifest manifest) { + Object option = manifest.getOptions().get("identifierCase"); + if (option == null) { + return null; + } + try { + return IdentifierCase.valueOf(option.toString()); + } catch (IllegalArgumentException e) { + return null; + } + } + + /** + * Generated (computed) and row version columns of the existing target table + */ + private List getGeneratedColumns(String tableName, String catalog, String schema) throws SQLException { + List columns = new ArrayList<>(); + MetaDataOperations operations = dbType.createMetaDataOperations(); + String tableSchema = StringUtils.defaultIfBlank(schema, connection.getSchema()); + String tableCatalog = dbType.isCatalogSupported() && dbType != DbType.ORACLE + ? StringUtils.defaultIfBlank(catalog, connection.getCatalog()) : null; + List> columnInfos; + try { + columnInfos = operations.getColumnInfos(connection.getMetaData(), tableCatalog, tableSchema, + createDialect().foldIdentifier(tableName)); + } catch (ImpExpException e) { + // The table does not exist, the importer reports it + columnInfos = Collections.emptyList(); + } + for (Map column : columnInfos) { + if ("YES".equalsIgnoreCase((String) column.get("IS_GENERATEDCOLUMN")) + || Boolean.TRUE.equals(column.get("IS_ROWVERSION"))) { + columns.add((String) column.get("COLUMN_NAME")); + } + } + return columns; + } + + /** + * Referenced tables are imported first, foreign keys are read from the tables created by schema.sql + */ + private List sortByForeignKeys(List tables, + ExportManifest manifest) throws SQLException { + Dialect dialect = createDialect(); + Map byName = new LinkedHashMap<>(); + for (ExportManifest.TableEntry table : tables) { + byName.put(dialect.foldIdentifier(table.getName()).toLowerCase(), table); + } + DatabaseMetaData metaData = connection.getMetaData(); + MetaDataOperations operations = dbType.createMetaDataOperations(); + String schema = StringUtils.defaultIfBlank(targetSchemaName, + StringUtils.defaultIfBlank(manifest.getTarget().getSchema(), connection.getSchema())); + Map> dependencies = new LinkedHashMap<>(); + for (Map.Entry entry : byName.entrySet()) { + Set referenced = new LinkedHashSet<>(); + for (Map keyInfo : operations.getImportedKeyInfos(metaData, null, schema, + dialect.foldIdentifier(entry.getValue().getName()))) { + String refTable = StringUtils.lowerCase((String) keyInfo.get("PKTABLE_NAME")); + if (refTable != null && !refTable.equals(entry.getKey()) && byName.containsKey(refTable)) { + referenced.add(refTable); + } + } + dependencies.put(entry.getKey(), referenced); + } + List sorted = new ArrayList<>(); + Set visited = new HashSet<>(); + for (String name : byName.keySet()) { + addInOrder(name, byName, dependencies, visited, new HashSet<>(), sorted); + } + return sorted; + } + + private static void addInOrder(String name, Map tables, + Map> dependencies, Set visited, Set visiting, + List sorted) { + if (visited.contains(name) || !visiting.add(name)) { + return; + } + for (String referenced : dependencies.getOrDefault(name, Collections.emptySet())) { + addInOrder(referenced, tables, dependencies, visited, visiting, sorted); + } + visited.add(name); + sorted.add(tables.get(name)); + } + + /** + * Forwards events of a table import: progress of the table is added to rows of tables imported before, start + * and end of the inner import are not forwarded, the first error is kept + */ + static class ProgressListener implements ExportListener { + + private final ExportListener delegate; + private final long completed; + private final long total; + private long rows; + private String error; + private Throwable cause; + + ProgressListener(ExportListener delegate, long completed, long total) { + this.delegate = delegate; + this.completed = completed; + this.total = total; + } + + long getRows() { + return rows; + } + + String getError() { + return error; + } + + Throwable getCause() { + return cause; + } + + @Override + public void onTableStart(String catalogName, String schemaName, String tableName, long totalRows) { + delegate.onTableStart(catalogName, schemaName, tableName, totalRows); + } + + @Override + public void onTableProgress(String catalogName, String schemaName, String tableName, long processedRows, + long totalRows) { + delegate.onTableProgress(catalogName, schemaName, tableName, processedRows, totalRows); + } + + @Override + public void onTableEnd(String catalogName, String schemaName, String tableName, long processedRows) { + rows += processedRows; + delegate.onTableEnd(catalogName, schemaName, tableName, processedRows); + } + + @Override + public void onProgress(long processed, long tableTotal) { + if (total > 0) { + delegate.onProgress(Math.min(completed + processed, total), total); + } + } + + @Override + public void onMessage(String message) { + delegate.onMessage(message); + } + + @Override + public void onError(String message, Throwable e) { + if (error == null) { + error = message; + cause = e; + } + delegate.onError(message, e); + } + + @Override + public boolean isCancelled() { + return delegate.isCancelled(); + } + } +} diff --git a/src/main/java/com/github/sqljam/impexp/DataFormat.java b/src/main/java/com/github/sqljam/impexp/DataFormat.java index 22bcf3f..64e1ec4 100644 --- a/src/main/java/com/github/sqljam/impexp/DataFormat.java +++ b/src/main/java/com/github/sqljam/impexp/DataFormat.java @@ -16,8 +16,9 @@ package com.github.sqljam.impexp; /** - * @Description: DataFormat of rows in an export package: SQL insert statements, or Parquet files (one file per - * table) for analytical tools + * @Description: DataFormat of rows in an export package: SQL insert statements, Parquet files (one file per + * table) for analytical tools, or CSV files (one file per table with a header row) for spreadsheets + * and text tools * @Author: Fred Feng * @Date: 26/03/2023 * @Version 1.0.0 @@ -26,5 +27,7 @@ public enum DataFormat { SQL, - PARQUET + PARQUET, + + CSV } diff --git a/src/main/java/com/github/sqljam/impexp/DbType.java b/src/main/java/com/github/sqljam/impexp/DbType.java index ad82908..8e354a6 100644 --- a/src/main/java/com/github/sqljam/impexp/DbType.java +++ b/src/main/java/com/github/sqljam/impexp/DbType.java @@ -323,6 +323,29 @@ public boolean isFileBased() { return false; } + /** + * Whether the database has transactions, an import can be committed manually or rolled back then + */ + public boolean isTransactionSupported() { + return this != CLICKHOUSE; + } + + /** + * Whether DDL statements are part of a transaction and are rolled back with it. MySQL, MariaDB, Oracle and H2 + * commit the transaction before and after a DDL statement. + */ + public boolean isTransactionalDdl() { + switch (this) { + case POSTGRESQL: + case SQLSERVER: + case SQLITE: + case DUCKDB: + return true; + default: + return false; + } + } + public String getValidationQuery() { return "SELECT 1"; } diff --git a/src/main/java/com/github/sqljam/impexp/Dialect.java b/src/main/java/com/github/sqljam/impexp/Dialect.java index 0595f5a..931640a 100644 --- a/src/main/java/com/github/sqljam/impexp/Dialect.java +++ b/src/main/java/com/github/sqljam/impexp/Dialect.java @@ -16,6 +16,7 @@ package com.github.sqljam.impexp; import java.math.BigDecimal; +import java.sql.Connection; import java.sql.ResultSet; import java.sql.SQLException; import java.sql.Time; @@ -31,6 +32,7 @@ import java.time.format.DateTimeFormatter; import java.util.ArrayList; import java.util.Arrays; +import java.util.Collection; import java.util.Collections; import java.util.HashMap; import java.util.HashSet; @@ -38,6 +40,8 @@ import java.util.Locale; import java.util.Map; import java.util.Set; +import java.util.TreeMap; +import java.util.concurrent.ConcurrentHashMap; import java.util.regex.Matcher; import java.util.regex.Pattern; import java.util.stream.Collectors; @@ -98,6 +102,10 @@ public abstract class Dialect { */ private String tableNamePattern; private IdentifierCase identifierCase = IdentifierCase.AUTO; + /** + * Names of existing target tables and columns, they are used as they are + */ + private final Set exactIdentifiers = ConcurrentHashMap.newKeySet(); /** * When set, table names in DDL/DML generated by this dialect are qualified with these names */ @@ -331,6 +339,7 @@ public Dialect forVersion(int majorVersion, int minorVersion) { dialect.setTargetCatalogName(targetCatalogName); dialect.setTargetSchemaName(targetSchemaName); dialect.setSourceSchemaPreserved(sourceSchemaPreserved); + dialect.addExactIdentifiers(exactIdentifiers); return dialect; } @@ -374,10 +383,19 @@ public boolean isReservedWord(String word) { } /** - * Converts identifier case of a source name to this database's convention + * Names of an existing target table and its columns: they are used as they are, e.g. a column "OrderId" of + * PostgreSQL written from a source column orderid + */ + public void addExactIdentifiers(Collection names) { + names.stream().filter(StringUtils::isNotBlank).forEach(exactIdentifiers::add); + } + + /** + * Converts identifier case of a source name to this database's convention, names of existing target objects + * are kept */ public String foldIdentifier(String name) { - if (StringUtils.isBlank(name)) { + if (StringUtils.isBlank(name) || exactIdentifiers.contains(name)) { return name; } switch (identifierCase) { @@ -1128,6 +1146,102 @@ public String getInsertTableStatement(String catalog, String schema, String tabl values); } + /** + * Whether rows can be inserted or updated by their key in one statement + */ + public boolean isUpsertSupported() { + return true; + } + + /** + * Inserts the row, or updates the row of the same key: the columns which are not keys are updated. Values are + * parameters (?) or literals of scripts. + * + * @param keyColumnNames columns which identify a row, e.g. the primary key + * @param updateColumnNames columns updated when the row exists, a row of keys only is inserted if missing + * @param values values of the columns, parameters or literals + */ + public String getUpsertStatement(String catalog, String schema, String tableName, String[] columnNames, + String[] keyColumnNames, String[] updateColumnNames, Object[] values) { + String insert = String.format("INSERT INTO %s(%s) VALUES (%s)", getQualifiedTableName(catalog, schema, + tableName), getIdentifiers(columnNames), StringUtils.join(values, ",")); + if (updateColumnNames.length == 0) { + return String.format("%s ON CONFLICT (%s) DO NOTHING", insert, getIdentifiers(keyColumnNames)); + } + String updates = Arrays.stream(updateColumnNames).map(name -> getIdentifier(name) + " = EXCLUDED." + + getIdentifier(name)).collect(Collectors.joining(", ")); + return String.format("%s ON CONFLICT (%s) DO UPDATE SET %s", insert, getIdentifiers(keyColumnNames), updates); + } + + /** + * Upsert statement with parameters for each column + */ + public String getUpsertStatement(String catalog, String schema, String tableName, String[] columnNames, + String[] keyColumnNames, String[] updateColumnNames) { + return getUpsertStatement(catalog, schema, tableName, columnNames, keyColumnNames, updateColumnNames, + Arrays.stream(columnNames).map(name -> "?").toArray()); + } + + /** + * Whether upserts may update indexed columns, an update of an indexed column of DuckDB is a delete and an + * insert which fails for keys and foreign keys + */ + public boolean isIndexedColumnUpdatable() { + return true; + } + + /** + * Columns of the primary key of an existing table in the order of the key, empty if it has none + */ + public List getPrimaryKeyColumns(Connection connection, String catalog, String schema, String tableName) + throws SQLException { + Map ordered = new TreeMap<>(); + try (ResultSet rs = connection.getMetaData().getPrimaryKeys(catalog, schema, tableName)) { + while (rs.next()) { + ordered.put(rs.getInt("KEY_SEQ"), rs.getString("COLUMN_NAME")); + } + } + return new ArrayList<>(ordered.values()); + } + + /** + * Columns of indexes and keys of an existing table, lower case + */ + public Set getIndexedColumns(Connection connection, String catalog, String schema, String tableName) + throws SQLException { + Set columns = new HashSet<>(); + try (ResultSet rs = connection.getMetaData().getIndexInfo(catalog, schema, tableName, false, true)) { + while (rs.next()) { + if (rs.getString("COLUMN_NAME") != null) { + columns.add(rs.getString("COLUMN_NAME").toLowerCase(Locale.ENGLISH)); + } + } + } + return columns; + } + + /** + * Whether the error means that the created object exists already: a table, an index, a key or a constraint. + * Imports which do not drop tables skip such statements of existing tables. + */ + public boolean isAlreadyExistsError(SQLException e) { + for (Throwable cause = e; cause != null; cause = cause.getCause()) { + if (cause instanceof SQLException) { + String state = ((SQLException) cause).getSQLState(); + // Duplicate table and duplicate object of SQL standard and PostgreSQL, table or index of ODBC + if ("42P07".equals(state) || "42710".equals(state) || "42S01".equals(state) + || "42S11".equals(state)) { + return true; + } + } + String message = StringUtils.lowerCase(cause.getMessage(), Locale.ENGLISH); + if (message != null && message.contains("already exists")) { + return true; + } + } + return false; + } + /** * Statements executed before inserting rows with explicit identity values (e.g. SET IDENTITY_INSERT ON) */ diff --git a/src/main/java/com/github/sqljam/impexp/DuckDBWorkspace.java b/src/main/java/com/github/sqljam/impexp/DuckDBWorkspace.java index 077782c..883e87a 100644 --- a/src/main/java/com/github/sqljam/impexp/DuckDBWorkspace.java +++ b/src/main/java/com/github/sqljam/impexp/DuckDBWorkspace.java @@ -18,6 +18,7 @@ import java.io.Closeable; import java.io.File; import java.io.IOException; +import java.nio.charset.StandardCharsets; import java.nio.file.Files; import java.sql.Connection; import java.sql.DriverManager; @@ -36,9 +37,10 @@ import lombok.extern.slf4j.Slf4j; /** - * @Description: DuckDBWorkspace is a temporary DuckDB database used to write and read Parquet files. Rows of other - * databases are copied into the workspace and written by COPY TO, Parquet files are loaded into the - * workspace by read_parquet and copied into other databases. The database file is deleted on close. + * @Description: DuckDBWorkspace is a temporary DuckDB database used to write and read Parquet and CSV files. Rows + * of other databases are copied into the workspace and written by COPY TO, files are loaded into the + * workspace by read_parquet or read_csv and copied into other databases. The database file is deleted + * on close. * @Author: Fred Feng * @Date: 26/03/2023 * @Version 1.0.0 @@ -47,6 +49,7 @@ public class DuckDBWorkspace implements Closeable { public static final String DEFAULT_SCHEMA = "main"; + public static final String DEFAULT_DELIMITER = ","; private final File directory; private final File file; @@ -104,6 +107,17 @@ public long countRows(String schema, String table) throws SQLException { */ public void writeParquet(String schema, String table, File parquetFile, String compression, Map metadata) throws SQLException, IOException { + writeParquet(schema, table, parquetFile, compression, metadata, 0, 0); + } + + /** + * Writes a range of rows in the order of insertion into a Parquet file, e.g. a part of a large table + * + * @param offset first row of the range + * @param limit rows of the range, 0 or less for all rows + */ + public void writeParquet(String schema, String table, File parquetFile, String compression, + Map metadata, long offset, long limit) throws SQLException, IOException { FileUtils.forceMkdirParent(parquetFile); String columnSql = "SELECT column_name, data_type FROM duckdb_columns() WHERE database_name = current_database()" + " AND schema_name = ? AND table_name = ? ORDER BY column_index"; @@ -124,8 +138,10 @@ public void writeParquet(String schema, String table, File parquetFile, String c .map(e -> e.getKey() + ": " + literal(e.getValue())).collect(Collectors.joining(", "))) .append("}"); } - String sql = String.format("COPY (SELECT %s FROM %s) TO %s (%s)", String.join(", ", columns), - qualify(schema, table), literal(parquetFile.getAbsolutePath()), options); + String range = limit > 0 ? String.format(" ORDER BY rowid LIMIT %d OFFSET %d", limit, Math.max(offset, 0)) + : ""; + String sql = String.format("COPY (SELECT %s FROM %s%s) TO %s (%s)", String.join(", ", columns), + qualify(schema, table), range, literal(parquetFile.getAbsolutePath()), options); execute(sql); } @@ -135,11 +151,16 @@ public void writeParquet(String schema, String table, File parquetFile, String c */ public void loadParquet(String schema, String table, List parquetFiles, List identityColumns) throws SQLException { - String files = parquetFiles.stream().map(f -> literal(f.getAbsolutePath())) - .collect(Collectors.joining(", ", "[", "]")); execute(String.format("CREATE SCHEMA IF NOT EXISTS %s", quote(schema))); execute(String.format("CREATE OR REPLACE TABLE %s AS SELECT * FROM read_parquet(%s)", qualify(schema, table), - files)); + fileList(parquetFiles))); + addIdentities(schema, table, identityColumns); + } + + /** + * Identity columns get a sequence and a nextval default, so that they are recognized as identity columns + */ + private void addIdentities(String schema, String table, List identityColumns) throws SQLException { if (identityColumns != null) { for (String column : identityColumns) { String sequence = qualify(schema, table + "_" + column + "_seq"); @@ -150,6 +171,104 @@ public void loadParquet(String schema, String table, List parquetFiles, Li } } + /** + * Writes rows of the table into a CSV file in UTF-8 with the delimiter, quote, text of NULL, header and date + * format of the options. An empty string is quoted to keep it apart from NULL, binary values are written as + * escaped text (\x00). Other encodings are converted after writing. + */ + public void writeCsv(String schema, String table, File csvFile, CsvOptions options) throws SQLException, + IOException { + FileUtils.forceMkdirParent(csvFile); + StringBuilder sql = new StringBuilder("COPY ").append(qualify(schema, table)).append(" TO ") + .append(literal(csvFile.getAbsolutePath())).append(" (FORMAT csv, HEADER ").append(options.isHeader()) + .append(", DELIMITER ").append(literal(StringUtils.defaultIfEmpty(options.getDelimiter(), + DEFAULT_DELIMITER))).append(", QUOTE ").append(literal(options.getQuote())) + .append(", ESCAPE ").append(literal(options.getQuote())).append(", NULLSTR ") + .append(literal(options.getNullString())); + if (StringUtils.isNotEmpty(options.getDateFormat())) { + sql.append(", DATEFORMAT ").append(literal(options.getDateFormat())).append(", TIMESTAMPFORMAT ") + .append(literal(options.getTimestampFormat())); + } + execute(sql.append(")").toString()); + } + + /** + * Loads CSV files into a table of the workspace. With the columns of the package the values are read with + * their types (exactly as exported), without them the types are detected from the files. + * + * @param columns columns and types of the files, null to detect them + * @param options options of the files, files of other encodings are converted to UTF-8 first + */ + public void loadCsv(String schema, String table, List csvFiles, List columns, + List identityColumns, CsvOptions options) throws SQLException, IOException { + execute(String.format("CREATE SCHEMA IF NOT EXISTS %s", quote(schema))); + execute(String.format("CREATE OR REPLACE TABLE %s AS SELECT * FROM %s", qualify(schema, table), + csvReader(toUtf8(csvFiles, options), columns, options))); + addIdentities(schema, table, identityColumns); + } + + /** + * Files in UTF-8: files of other encodings are converted into the workspace, the BOM of UTF-8 is removed + */ + private List toUtf8(List csvFiles, CsvOptions options) throws IOException { + if (options == null || options.isUtf8()) { + return csvFiles; + } + List converted = new ArrayList<>(); + for (File file : csvFiles) { + File target = new File(directory, "utf8-" + (converted.size() + 1) + "-" + file.getName()); + CsvSplitter.transcode(file, options.getCharset(), target, StandardCharsets.UTF_8, false); + converted.add(target); + } + return converted; + } + + /** + * read_csv of the files: a quoted empty field is an empty string and the text of NULL is NULL. Options of + * files of other tools which are not given are detected. + */ + private static String csvReader(List csvFiles, List columns, + CsvOptions options) { + StringBuilder reader = new StringBuilder("read_csv(").append(fileList(csvFiles)) + .append(", allow_quoted_nulls = false"); + if (options != null) { + reader.append(", header = ").append(options.isHeader()).append(", quote = ").append(literal(options + .getQuote())).append(", escape = ").append(literal(options.getQuote())).append(", nullstr = ") + .append(literal(options.getNullString())); + if (StringUtils.isNotEmpty(options.getDelimiter())) { + reader.append(", delim = ").append(literal(options.getDelimiter())); + } + if (StringUtils.isNotEmpty(options.getDateFormat())) { + reader.append(", dateformat = ").append(literal(options.getDateFormat())).append(", timestampformat = ") + .append(literal(options.getTimestampFormat())); + } + } else { + reader.append(", header = true"); + } + if (columns != null && !columns.isEmpty()) { + reader.append(", auto_detect = false, columns = {").append(columns.stream().map(column -> literal(column + .getName()) + ": " + literal(column.getType())).collect(Collectors.joining(", "))).append("}"); + } + return reader.append(")").toString(); + } + + /** + * Reader of the files by their extension: read_csv for .csv, .tsv and .txt files, read_parquet for others + */ + private String reader(List files, CsvOptions options) throws IOException { + return files.stream().allMatch(DuckDBWorkspace::isCsv) ? csvReader(toUtf8(files, options), null, options) + : String.format("read_parquet(%s)", fileList(files)); + } + + public static boolean isCsv(File file) { + String name = file.getName().toLowerCase(Locale.ENGLISH); + return name.endsWith(".csv") || name.endsWith(".tsv") || name.endsWith(".txt"); + } + + private static String fileList(List files) { + return files.stream().map(f -> literal(f.getAbsolutePath())).collect(Collectors.joining(", ", "[", "]")); + } + /** * Drops columns of a workspace table, e.g. columns which are generated by the target database */ @@ -174,14 +293,20 @@ public List describeTable(String schema, String table) throws SQLExcep } /** - * Columns of Parquet files: name and DuckDB type + * Columns of Parquet or CSV files: name and DuckDB type, types of CSV files are detected + */ + public List describeFiles(List dataFiles) throws SQLException, IOException { + return describeFiles(dataFiles, null); + } + + /** + * Columns of Parquet or CSV files, CSV files are read with the options (null to detect them) */ - public List describeParquet(List parquetFiles) throws SQLException { - String files = parquetFiles.stream().map(f -> literal(f.getAbsolutePath())) - .collect(Collectors.joining(", ", "[", "]")); + public List describeFiles(List dataFiles, CsvOptions options) throws SQLException, IOException { List columns = new ArrayList<>(); try (Statement statement = getConnection().createStatement(); - ResultSet rs = statement.executeQuery(String.format("DESCRIBE SELECT * FROM read_parquet(%s)", files))) { + ResultSet rs = statement.executeQuery(String.format("DESCRIBE SELECT * FROM %s", reader(dataFiles, + options)))) { while (rs.next()) { columns.add(new String[]{rs.getString("column_name"), rs.getString("column_type")}); } @@ -190,19 +315,22 @@ public List describeParquet(List parquetFiles) throws SQLExcepti } /** - * First rows of Parquet files, values as text + * First rows of Parquet or CSV files, values as text */ - public List> readParquetRows(List parquetFiles, int limit) throws SQLException { - String files = parquetFiles.stream().map(f -> literal(f.getAbsolutePath())) - .collect(Collectors.joining(", ", "[", "]")); - List columns = describeParquet(parquetFiles).stream() + public List> readRows(List dataFiles, int limit) throws SQLException, IOException { + return readRows(dataFiles, limit, null); + } + + public List> readRows(List dataFiles, int limit, CsvOptions options) throws SQLException, + IOException { + List columns = describeFiles(dataFiles, options).stream() .map(column -> String.format("CAST(%s AS VARCHAR)", quote(column[0]))) .collect(Collectors.toList()); List> rows = new ArrayList<>(); if (columns.isEmpty()) { return rows; } - String sql = String.format("SELECT %s FROM read_parquet(%s) LIMIT %d", String.join(", ", columns), files, + String sql = String.format("SELECT %s FROM %s LIMIT %d", String.join(", ", columns), reader(dataFiles, options), Math.max(limit, 0)); try (Statement statement = getConnection().createStatement(); ResultSet rs = statement.executeQuery(sql)) { int count = rs.getMetaData().getColumnCount(); @@ -217,11 +345,13 @@ public List> readParquetRows(List parquetFiles, int limit) th return rows; } - public long countParquetRows(List parquetFiles) throws SQLException { - String files = parquetFiles.stream().map(f -> literal(f.getAbsolutePath())) - .collect(Collectors.joining(", ", "[", "]")); + public long countFileRows(List dataFiles) throws SQLException, IOException { + return countFileRows(dataFiles, null); + } + + public long countFileRows(List dataFiles, CsvOptions options) throws SQLException, IOException { Map row = JdbcUtils.fetchOne(getConnection(), - String.format("SELECT count(*) AS n FROM read_parquet(%s)", files), new Object[0]); + String.format("SELECT count(*) AS n FROM %s", reader(dataFiles, options)), new Object[0]); Object count = row != null ? row.get("n") : null; return count instanceof Number ? ((Number) count).longValue() : 0; } diff --git a/src/main/java/com/github/sqljam/impexp/ExportManifest.java b/src/main/java/com/github/sqljam/impexp/ExportManifest.java index 901fca4..5f7870e 100644 --- a/src/main/java/com/github/sqljam/impexp/ExportManifest.java +++ b/src/main/java/com/github/sqljam/impexp/ExportManifest.java @@ -32,7 +32,10 @@ import com.fasterxml.jackson.databind.DeserializationFeature; import com.fasterxml.jackson.databind.ObjectMapper; import com.fasterxml.jackson.databind.SerializationFeature; +import com.fasterxml.jackson.databind.annotation.JsonDeserialize; +import lombok.AllArgsConstructor; import lombok.Getter; +import lombok.NoArgsConstructor; import lombok.Setter; /** @@ -72,7 +75,12 @@ public enum FileType { /** * Rows of a table in a Parquet file */ - PARQUET + PARQUET, + + /** + * Rows of a table in a CSV file with a header row + */ + CSV } private String format = FORMAT; @@ -132,12 +140,38 @@ public static class TableEntry { private String schema; private String name; private long rows; + /** + * Data files in the order of importing, e.g. orders.csv, orders_2.csv + */ + @JsonDeserialize(as = LinkedHashSet.class) private Set dataFiles = new LinkedHashSet<>(); private long lobFiles; /** * Identity columns of the table (Parquet packages), their values are kept and sequences are reset */ private List identityColumns = new ArrayList<>(); + /** + * Columns of the data files and their types (CSV packages), CSV files have no types, the rows are read + * back with these types, so that values are imported exactly + */ + private List columns = new ArrayList<>(); + /** + * Key columns of upserts, empty for the primary key of the table or for inserts + */ + private List keyColumns = new ArrayList<>(); + } + + @Getter + @Setter + @NoArgsConstructor + @AllArgsConstructor + public static class ColumnEntry { + + private String name; + /** + * Type of the column in the DuckDB workspace, e.g. DECIMAL(38,10) + */ + private String type; } private static ObjectMapper createObjectMapper() { diff --git a/src/main/java/com/github/sqljam/impexp/Exporter.java b/src/main/java/com/github/sqljam/impexp/Exporter.java index fad6d91..384544b 100644 --- a/src/main/java/com/github/sqljam/impexp/Exporter.java +++ b/src/main/java/com/github/sqljam/impexp/Exporter.java @@ -19,6 +19,7 @@ import java.sql.DatabaseMetaData; import java.sql.SQLException; import java.util.HashMap; +import java.util.List; import java.util.Map; import org.apache.commons.lang3.ArrayUtils; @@ -114,6 +115,15 @@ public static class ExportConfiguration { * Queries narrowing the rows and columns of tables by table name, e.g. rows filtered in the data viewer */ private Map tableQueries = new HashMap<>(); + /** + * Rows are inserted into the target, or upserted: the rows of the same key are updated + */ + private WriteMode writeMode = WriteMode.INSERT; + /** + * Key columns of upserts by table name, the primary key of the table by default. A table without key is + * inserted. + */ + private Map> keyColumnNames = new HashMap<>(); /** * Query of a table, null if all rows and columns are exported. Grouped rows are not rows of the table. diff --git a/src/main/java/com/github/sqljam/impexp/ImportExportHandler.java b/src/main/java/com/github/sqljam/impexp/ImportExportHandler.java index 8684143..0753326 100644 --- a/src/main/java/com/github/sqljam/impexp/ImportExportHandler.java +++ b/src/main/java/com/github/sqljam/impexp/ImportExportHandler.java @@ -20,18 +20,23 @@ import java.sql.PreparedStatement; import java.sql.ResultSet; import java.sql.SQLException; +import java.sql.Savepoint; import java.sql.Statement; import java.sql.Types; import java.util.ArrayList; +import java.util.Arrays; import java.util.Collections; import java.util.HashMap; +import java.util.HashSet; import java.util.LinkedHashMap; import java.util.LinkedHashSet; import java.util.List; import java.util.Locale; import java.util.Map; import java.util.Set; +import java.util.TreeMap; import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.atomic.AtomicLong; import org.apache.commons.collections4.CollectionUtils; import org.apache.commons.lang3.ArrayUtils; @@ -40,6 +45,7 @@ import com.github.sqljam.impexp.DdlScripter.Schema; import com.github.sqljam.jdbc.ConnectionFactory; import com.github.sqljam.jdbc.JdbcUtils; +import com.github.sqljam.jdbc.ManualTransaction; import com.github.sqljam.jdbc.SimpleConnectionFactory; import com.github.sqljam.page.EachPage; import lombok.Getter; @@ -90,6 +96,24 @@ public static class ImportConfiguration { * tables in the same schema. Target tables have the same names by default. */ private String tableNamePattern; + /** + * Rows are inserted, upserted (the rows of the same key are updated) or inserted unless their key exists + */ + private WriteMode writeMode = WriteMode.INSERT; + /** + * Key columns of upserts by source table name, the primary key of the table by default + */ + private Map> keyColumnNames = new HashMap<>(); + /** + * Whether target tables are dropped and created, existing tables, indexes and constraints are skipped when + * they are kept + */ + private boolean tableRecreated = true; + /** + * Rows with errors are skipped: a failed page is written row by row and the failed rows are reported. + * Otherwise the error ends the page. + */ + private boolean skipErrors; } private ConnectionFactory connectionFactory; @@ -97,6 +121,9 @@ public static class ImportConfiguration { private ExportListener exportListener = ExportListener.NONE; private final IdentityValueTracker identityValueTracker = new IdentityValueTracker(); private final Map> targetColumnTypes = new ConcurrentHashMap<>(); + private final Map keyColumns = new ConcurrentHashMap<>(); + private final Map skippedRows = new ConcurrentHashMap<>(); + private final Map> targetNames = new ConcurrentHashMap<>(); @Override public void setExportListener(ExportListener exportListener) { @@ -260,10 +287,17 @@ private void executeStatements(String catalogName, String schemaName, List targetTypes = Collections.emptyMap(); - String insertSql = dialect.getInsertTableStatement(catalogName, schemaName, targetTableName, columns); + boolean upsert = configuration.getWriteMode().isKeyed(); + String insertSql = upsert ? null : dialect.getInsertTableStatement(catalogName, schemaName, targetTableName, + columns); Connection connection = null; boolean autoCommit = true; boolean transactional = false; @@ -342,42 +425,84 @@ public void exportData(String catalogName, connection = getConnection(catalogName, schemaName); autoCommit = connection.getAutoCommit(); transactional = JdbcUtils.beginTransaction(connection); + // An existing table is written by its own names, names which differ in case only are matched + Map existingNames = getTargetNames(connection, schemaName, targetTableName); + if (!existingNames.isEmpty()) { + dialect.addExactIdentifiers(existingNames.values()); + targetTableName = existingNames.get(""); + for (int i = 0; i < columns.length; i++) { + columns[i] = existingNames.getOrDefault(columns[i].toLowerCase(Locale.ENGLISH), columns[i]); + } + if (!upsert) { + insertSql = dialect.getInsertTableStatement(catalogName, schemaName, targetTableName, columns); + } + } if (dialect.isTargetTypeConversionRequired()) { targetTypes = getTargetColumnTypes(connection, targetTableName); } + String[] keys = upsert ? getKeyColumnNames(connection, schemaName, tableName, targetTableName, + tableMetaData, columns) : null; + if (keys != null && keys.length == 0) { + // A table without key is inserted + upsert = false; + insertSql = dialect.getInsertTableStatement(catalogName, schemaName, targetTableName, columns); + } + if (upsert) { + Set fixed = new HashSet<>(); + Arrays.stream(keys).forEach(name -> fixed.add(name.toLowerCase(Locale.ENGLISH))); + // Identity values of existing rows are kept + identityColumnNames.forEach(name -> fixed.add(name.toLowerCase(Locale.ENGLISH))); + if (!dialect.isIndexedColumnUpdatable()) { + fixed.addAll(dialect.getIndexedColumns(connection, getTargetCatalog(connection), + getTargetSchema(connection, schemaName), dialect.foldIdentifier(targetTableName))); + tableMetaData.getIndexMetaDatas().forEach(index -> fixed.add(index.getColumnName().toLowerCase( + Locale.ENGLISH))); + } + // Existing rows are kept when they are skipped + String[] updates = configuration.getWriteMode() == WriteMode.SKIP_EXISTING ? new String[0] + : Arrays.stream(columns).filter(name -> !fixed.contains(name.toLowerCase(Locale.ENGLISH))) + .toArray(String[]::new); + insertSql = dialect.getUpsertStatement(catalogName, schemaName, targetTableName, columns, keys, + updates); + } executeQuietly(connection, dialect.getStatementBeforeInsert(catalogName, schemaName, targetTableName, identityIncluded)); int rows = 0; + Set arrayColumns = upsert && dialect.isArrayParameterSupported() ? getTargetArrayColumns( + connection, schemaName, targetTableName) : Collections.emptySet(); try (PreparedStatement ps = connection.prepareStatement(insertSql)) { Map arrayParameters = new HashMap<>(); - for (Map row : rowList) { - int index = 0; - for (Object value : row.values()) { - if (value == null) { - ps.setNull(index + 1, nullTypes[index]); - } else { - if (isArrayText(value) && dialect.isArrayParameterSupported() - && isArrayParameter(ps, index + 1, arrayParameters)) { - // Arrays kept as text (Parquet packages) are bound as arrays - value = parseArrayText(value.toString()); - } - if (emptyLobTypes[index] == Types.BLOB && value instanceof byte[] - && ((byte[]) value).length == 0) { - ps.setBlob(index + 1, connection.createBlob()); - } else if (emptyLobTypes[index] == Types.CLOB && "".equals(value)) { - ps.setClob(index + 1, connection.createClob()); - } else { - ps.setObject(index + 1, targetTypes.isEmpty() ? value - : dialect.getTargetJdbcValue(value, targetTypes.get(columns[index] - .toLowerCase(Locale.ENGLISH)))); - } + final Map types = targetTypes; + final Connection target = connection; + RowBinder binder = (statement, row) -> bindRow(target, statement, row, columns, nullTypes, + emptyLobTypes, types, arrayColumns, arrayParameters); + Savepoint savepoint = configuration.isSkipErrors() && transactional ? setSavepointQuietly(connection) + : null; + try { + for (Map row : rowList) { + binder.bind(ps, row); + ps.addBatch(); + } + int[] counts = ps.executeBatch(); + releaseQuietly(connection, savepoint); + if (upsert) { + // Updated rows count twice in MySQL, every row is written once + rows = rowList.size(); + } else { + for (int n : counts) { + rows += n > 0 ? n : (n == Statement.SUCCESS_NO_INFO ? 1 : 0); } - index++; } - ps.addBatch(); - } - for (int n : ps.executeBatch()) { - rows += n > 0 ? n : (n == Statement.SUCCESS_NO_INFO ? 1 : 0); + } catch (SQLException e) { + if (!configuration.isSkipErrors()) { + throw e; + } + // Rows are named by their keys in the log, or by their first columns + List named = keys != null && keys.length > 0 ? Arrays.asList(keys) + : tableMetaData.getPrimaryKeyColumnNames(); + rows = writeRowByRow(connection, ps, insertSql, savepoint, transactional, rowList, binder, tableName, + named.isEmpty() ? Arrays.asList(columns).subList(0, Math.min(3, columns.length)) + : named, e); } } executeQuietly(connection, dialect.getStatementAfterInsert(catalogName, schemaName, targetTableName, @@ -415,6 +540,162 @@ && isArrayParameter(ps, index + 1, arrayParameters)) { } } + /** + * Binds the values of a row to the parameters of the statement + */ + @FunctionalInterface + private interface RowBinder { + void bind(PreparedStatement ps, Map row) throws SQLException; + } + + private void bindRow(Connection connection, PreparedStatement ps, Map row, String[] columns, + int[] nullTypes, int[] emptyLobTypes, Map targetTypes, + Set arrayColumns, Map arrayParameters) throws SQLException { + int index = 0; + for (Object value : row.values()) { + if (value == null) { + ps.setNull(index + 1, nullTypes[index]); + } else { + if (isArrayText(value) && dialect.isArrayParameterSupported() + && (arrayColumns.contains(columns[index].toLowerCase(Locale.ENGLISH)) + || isArrayParameter(ps, index + 1, arrayParameters))) { + // Arrays kept as text (Parquet packages) are bound as arrays + value = parseArrayText(value.toString()); + } + if (emptyLobTypes[index] == Types.BLOB && value instanceof byte[] && ((byte[]) value).length == 0) { + ps.setBlob(index + 1, connection.createBlob()); + } else if (emptyLobTypes[index] == Types.CLOB && "".equals(value)) { + ps.setClob(index + 1, connection.createClob()); + } else { + ps.setObject(index + 1, targetTypes.isEmpty() ? value : dialect.getTargetJdbcValue(value, + targetTypes.get(columns[index].toLowerCase(Locale.ENGLISH)))); + } + } + index++; + } + } + + /** + * Writes the rows of a failed page one by one and skips the rows with errors, each skipped row is reported + * with its key and its error. Each row has its own savepoint, or its own commit when the database has no + * savepoints. A transaction of the whole import without savepoints (DuckDB) can not go on after an error, the + * error is thrown. + * + * @param namedColumns columns which name a row in the log, e.g. its key + * @return rows written + */ + private int writeRowByRow(Connection connection, PreparedStatement ps, String sql, Savepoint pageSavepoint, + boolean transactional, List> rowList, RowBinder binder, + String tableName, List namedColumns, SQLException batchError) + throws SQLException { + try { + ps.clearBatch(); + } catch (SQLException ignored) { + // The batch ended with the error + } + boolean savepoints = pageSavepoint != null; + if (transactional) { + if (savepoints) { + connection.rollback(pageSavepoint); + } else if (ManualTransaction.isManaged(connection)) { + // The transaction can not go on after the error, the whole import is rolled back + connection.rollback(); + throw new SQLException(String.format("Rows of table '%s' with errors can not be skipped in one" + + " transaction without savepoints, the import is rolled back: %s", tableName, + batchError.getMessage()), batchError.getSQLState(), batchError.getErrorCode(), batchError); + } else { + // Rows of this page are written again, rows of former pages are committed already + connection.rollback(); + } + } + int rows = 0; + PreparedStatement statement = ps; + try { + for (Map row : rowList) { + // DuckDB closes a statement when it fails + if (statement.isClosed()) { + statement = connection.prepareStatement(sql); + } + if (writeRow(connection, statement, row, binder, savepoints, transactional, tableName, + namedColumns)) { + rows++; + } + } + } finally { + if (statement != ps) { + JdbcUtils.closeQuietly(statement); + } + } + return rows; + } + + /** + * Writes a row in its own savepoint or commit, a row with an error is rolled back and reported + * + * @return whether the row is written + */ + private boolean writeRow(Connection connection, PreparedStatement statement, Map row, + RowBinder binder, boolean savepoints, boolean transactional, String tableName, + List namedColumns) throws SQLException { + Savepoint savepoint = savepoints ? connection.setSavepoint() : null; + try { + binder.bind(statement, row); + statement.executeUpdate(); + if (savepoints) { + releaseQuietly(connection, savepoint); + } else if (transactional) { + connection.commit(); + } + return true; + } catch (SQLException e) { + if (savepoints) { + connection.rollback(savepoint); + } else if (transactional) { + connection.rollback(); + } + skippedRows.computeIfAbsent(tableName, name -> new AtomicLong()).incrementAndGet(); + String message = String.format("Skipped row of table '%s' (%s): %s", tableName, + describeRow(row, namedColumns), e.getMessage()); + if (log.isWarnEnabled()) { + log.warn(message); + } + exportListener.onError(message, e); + return false; + } + } + + /** + * Rows of the table which were skipped because of errors since the last call + */ + public long takeSkippedRows(String tableName) { + AtomicLong rows = skippedRows.remove(tableName); + return rows != null ? rows.get() : 0; + } + + /** + * Values of the named columns of a row, e.g. ID=5, long values are abbreviated + */ + static String describeRow(Map row, List namedColumns) { + List values = new ArrayList<>(); + for (String name : namedColumns) { + Object value = row.entrySet().stream().filter(entry -> entry.getKey().equalsIgnoreCase(name)) + .map(Map.Entry::getValue).findFirst().orElse(null); + String text = value instanceof byte[] ? "<" + ((byte[]) value).length + " bytes>" + : String.valueOf(value); + values.add(name + "=" + StringUtils.abbreviate(text, 50)); + } + return String.join(", ", values); + } + + private static Savepoint setSavepointQuietly(Connection connection) { + try { + return connection.setSavepoint(); + } catch (SQLException e) { + // Databases without savepoints, e.g. DuckDB + return null; + } + } + private void resetIdentityValues(String catalogName, String schemaName, String tableName, TableMetaData tableMetaData) throws SQLException { Map maxValues = identityValueTracker.remove(tableMetaData); @@ -424,8 +705,12 @@ private void resetIdentityValues(String catalogName, String schemaName, String t List sqls = new ArrayList<>(); for (Map.Entry entry : maxValues.entrySet()) { Dialect targetDialect = tableMetaData.getDialect(); - String sql = targetDialect.getResetIdentityStatement(catalogName, schemaName, - targetDialect.getTargetTableName(tableName), entry.getKey(), entry.getValue() + 1); + // The existing table by its own name + String targetTableName = targetDialect.getTargetTableName(tableName); + Map existingNames = targetNames.getOrDefault(targetTableName, Collections.emptyMap()); + String sql = targetDialect.getResetIdentityStatement(catalogName, schemaName, existingNames.getOrDefault( + "", targetTableName), existingNames.getOrDefault(entry.getKey().toLowerCase(Locale.ENGLISH), + entry.getKey()), entry.getValue() + 1); if (StringUtils.isNotBlank(sql)) { sqls.add(sql); } @@ -458,6 +743,135 @@ private void executeQuietly(Connection connection, String[] sqls) { /** * Type names of the columns of a target table by lower case column name, read once per table */ + /** + * Key columns of upserts: the chosen columns, or the primary key of the source table or of the existing target + * table (e.g. rows of files). Indexes are not guessed, a table without key is inserted. + * + * @return key columns as named in the imported columns, empty if the table has no key + */ + private String[] getKeyColumnNames(Connection connection, String schemaName, String tableName, + String targetTableName, TableMetaData tableMetaData, String[] columns) + throws SQLException { + String[] cached = keyColumns.get(tableName); + if (cached != null) { + return cached; + } + List keys = configuration.getKeyColumnNames().entrySet().stream() + .filter(entry -> entry.getKey().equalsIgnoreCase(tableName)).map(Map.Entry::getValue).findFirst() + .orElse(null); + if (CollectionUtils.isEmpty(keys)) { + keys = tableMetaData.getPrimaryKeyColumnNames(); + } + if (CollectionUtils.isEmpty(keys)) { + keys = dialect.getPrimaryKeyColumns(connection, getTargetCatalog(connection), getTargetSchema(connection, + schemaName), dialect.foldIdentifier(targetTableName)); + } + if (CollectionUtils.isEmpty(keys)) { + exportListener.onMessage(String.format("Table %s has no primary key and no chosen key columns, its rows " + + "are inserted", tableName)); + keyColumns.put(tableName, new String[0]); + return keyColumns.get(tableName); + } + String[] resolved = new String[keys.size()]; + for (int i = 0; i < resolved.length; i++) { + String key = keys.get(i); + resolved[i] = Arrays.stream(columns).filter(column -> column.equalsIgnoreCase(key)).findFirst() + .orElseThrow(() -> new ImpExpException(String.format("Key column %s of table %s is not imported", + key, tableName))); + } + keyColumns.put(tableName, resolved); + return resolved; + } + + /** + * Names of an existing target table and its columns by their lower case names, the table name under the empty + * key. Empty if the table does not exist or its names can not be read, the names of the import are used then. + */ + private Map getTargetNames(Connection connection, String schemaName, String targetTableName) { + Map names = targetNames.get(targetTableName); + if (names != null) { + return names; + } + names = new HashMap<>(); + try { + DatabaseMetaData metaData = connection.getMetaData(); + String catalog = getTargetCatalog(connection); + String schema = getTargetSchema(connection, schemaName); + String table = findTargetTable(metaData, catalog, schema, targetTableName); + if (table != null) { + try (ResultSet rs = metaData.getColumns(catalog, schema, table, null)) { + while (rs.next()) { + String column = rs.getString("COLUMN_NAME"); + names.putIfAbsent(column.toLowerCase(Locale.ENGLISH), column); + } + } + if (!names.isEmpty()) { + names.put("", table); + } + } + } catch (SQLException e) { + if (log.isWarnEnabled()) { + log.warn("Unable to read names of target table {}: {}", targetTableName, e.getMessage()); + } + names.clear(); + } + targetNames.put(targetTableName, names); + return names; + } + + /** + * Name of the existing target table: the name of the import, its folded name, or a name which differs in case + * only (e.g. Orders of MySQL on Linux for orders) + */ + private String findTargetTable(DatabaseMetaData metaData, String catalog, String schema, String tableName) + throws SQLException { + for (String candidate : new String[]{dialect.foldIdentifier(tableName), tableName, null}) { + try (ResultSet rs = metaData.getTables(catalog, schema, candidate, null)) { + while (rs.next()) { + String name = rs.getString("TABLE_NAME"); + if (name != null && name.equalsIgnoreCase(tableName)) { + return name; + } + } + } + } + return null; + } + + /** + * Catalog of the target table: the target catalog or the catalog of the connection + */ + private String getTargetCatalog(Connection connection) throws SQLException { + return StringUtils.defaultIfBlank(configuration.getTargetCatalogName(), connection.getCatalog()); + } + + /** + * Schema of the target table as the import resolves it, the schema of the connection (e.g. dbo of SQL Server) + * is only the default + */ + private String getTargetSchema(Connection connection, String schemaName) throws SQLException { + String schema = dialect != null ? dialect.getTargetSchemaName(schemaName) : configuration.getTargetSchemaName(); + return StringUtils.defaultIfBlank(schema, connection.getSchema()); + } + + /** + * Array columns of the target table, for statements whose parameters have no types (e.g. MERGE of H2) + */ + private Set getTargetArrayColumns(Connection connection, String schemaName, String targetTableName) + throws SQLException { + Set columns = new HashSet<>(); + try (ResultSet rs = connection.getMetaData().getColumns(getTargetCatalog(connection), + getTargetSchema(connection, schemaName), dialect.foldIdentifier(targetTableName), null)) { + while (rs.next()) { + String typeName = StringUtils.defaultString(rs.getString("TYPE_NAME")).toUpperCase(Locale.ENGLISH); + if (rs.getInt("DATA_TYPE") == Types.ARRAY || typeName.endsWith("ARRAY") || typeName.endsWith("[]")) { + columns.add(rs.getString("COLUMN_NAME").toLowerCase(Locale.ENGLISH)); + } + } + } + return columns; + } + private Map getTargetColumnTypes(Connection connection, String targetTableName) throws SQLException { Map types = targetColumnTypes.get(targetTableName); diff --git a/src/main/java/com/github/sqljam/impexp/ImportExporter.java b/src/main/java/com/github/sqljam/impexp/ImportExporter.java index 92d69d9..3013ca1 100644 --- a/src/main/java/com/github/sqljam/impexp/ImportExporter.java +++ b/src/main/java/com/github/sqljam/impexp/ImportExporter.java @@ -54,8 +54,73 @@ public void setTargetConnectionFactory(ConnectionFactory connectionFactory) { importingExportHandler.setConnectionFactory(connectionFactory); } + /** + * Rows of a table are reported without the rows which were skipped because of errors + */ public void setExportListener(ExportListener exportListener) { - exporter.setExportListener(exportListener); + exporter.setExportListener(exportListener == null ? null : new WrittenRowsListener(exportListener, + importingExportHandler)); + } + + /** + * Reports the written rows of a table at its end: the read rows without the skipped rows + */ + private static class WrittenRowsListener implements ExportListener { + + private final ExportListener delegate; + private final ImportExportHandler handler; + + WrittenRowsListener(ExportListener delegate, ImportExportHandler handler) { + this.delegate = delegate; + this.handler = handler; + } + + @Override + public void onStart(ExportMode exportMode) { + delegate.onStart(exportMode); + } + + @Override + public void onTableStart(String catalogName, String schemaName, String tableName, long totalRows) { + delegate.onTableStart(catalogName, schemaName, tableName, totalRows); + } + + @Override + public void onTableProgress(String catalogName, String schemaName, String tableName, long processedRows, + long totalRows) { + delegate.onTableProgress(catalogName, schemaName, tableName, processedRows, totalRows); + } + + @Override + public void onTableEnd(String catalogName, String schemaName, String tableName, long processedRows) { + delegate.onTableEnd(catalogName, schemaName, tableName, Math.max(0, processedRows + - handler.takeSkippedRows(tableName))); + } + + @Override + public void onProgress(long processed, long total) { + delegate.onProgress(processed, total); + } + + @Override + public void onMessage(String message) { + delegate.onMessage(message); + } + + @Override + public void onError(String message, Throwable e) { + delegate.onError(message, e); + } + + @Override + public void onEnd(boolean successful) { + delegate.onEnd(successful); + } + + @Override + public boolean isCancelled() { + return delegate.isCancelled(); + } } /** @@ -130,6 +195,19 @@ private void prepare() { dialect = dialect.forVersion(targetMajorVersion, Math.max(targetMinorVersion, 0)); } exportConfiguration.setDialect(dialect); + importConfiguration.setTableRecreated(exportConfiguration.isTableRecreated()); + if (exportConfiguration.getWriteMode().isKeyed()) { + importConfiguration.setWriteMode(exportConfiguration.getWriteMode()); + } + exportConfiguration.getKeyColumnNames().forEach(importConfiguration.getKeyColumnNames()::putIfAbsent); + // An import which goes on after errors skips the rows with errors + if (!exportConfiguration.isFailFast()) { + importConfiguration.setSkipErrors(true); + } + if (importConfiguration.getWriteMode().isKeyed() && !dialect.isUpsertSupported()) { + throw new ImpExpException(importConfiguration.getDbType().getDisplayName() + + " can not match rows by their key, please insert the rows"); + } // Creating user, database and schema is not the business of importing exportConfiguration.setShowCreateUserSql(false); exportConfiguration.setShowCreateCatalogSql(false); diff --git a/src/main/java/com/github/sqljam/impexp/ParquetExporter.java b/src/main/java/com/github/sqljam/impexp/ParquetExporter.java index fdab76a..7c68f8d 100644 --- a/src/main/java/com/github/sqljam/impexp/ParquetExporter.java +++ b/src/main/java/com/github/sqljam/impexp/ParquetExporter.java @@ -16,89 +16,51 @@ package com.github.sqljam.impexp; import java.io.File; -import java.lang.reflect.Field; -import java.lang.reflect.Modifier; -import java.sql.DatabaseMetaData; import java.util.ArrayList; -import java.util.LinkedHashMap; +import java.util.Collections; import java.util.List; import java.util.Map; -import java.util.Optional; +import org.apache.commons.io.FilenameUtils; import org.apache.commons.lang3.StringUtils; -import com.github.sqljam.impexp.db.DuckDBDialect; -import com.github.sqljam.impexp.db.DuckDBMetaDataOperations; -import com.github.sqljam.jdbc.SharedConnectionFactory; -import lombok.extern.slf4j.Slf4j; /** * @Description: ParquetExporter exports tables as an export package with Parquet files: schema.sql and * constraints.sql for the target database (the same as SQL packages), data/<table>.parquet for - * rows and manifest.json. Rows are copied into a temporary DuckDB workspace first (values which - * Parquet can not hold exactly are kept as text), then written by DuckDB. + * rows and manifest.json. Values which Parquet can not hold exactly are kept as text. * @Author: Fred Feng * @Date: 26/03/2023 * @Version 1.0.0 */ -@Slf4j -public final class ParquetExporter { +public final class ParquetExporter extends DataFileExporter { public static final String PARQUET_EXTENSION = ".parquet"; public static final String DEFAULT_COMPRESSION = "ZSTD"; public static final String[] COMPRESSIONS = {"ZSTD", "SNAPPY", "GZIP", "UNCOMPRESSED"}; - private final File root; - private final Exporter.ExportConfiguration configuration = new Exporter.ExportConfiguration(); - private ExportListener exportListener = ExportListener.NONE; - private DbType targetDbType; - private IdentifierCase identifierCase; - private String targetCatalogName; - private String targetSchemaName; - private int targetMajorVersion = -1; - private int targetMinorVersion = -1; + /** + * Parts are written a little smaller than the maximum size, the size of compressed rows varies + */ + private static final double PART_FILL = 0.9; + private String compression = DEFAULT_COMPRESSION; - private final List writtenFiles = new ArrayList<>(); + private long maxFileSize; public ParquetExporter(File root) { - this.root = root; - configuration.setShowCreateUserSql(false); - configuration.setShowCreateCatalogSql(false); - configuration.setShowCreateSchemaSql(false); + super(root); } /** - * Source database and options, the same as {@link ScriptExporter#getConfiguration()} + * Maximum size of a Parquet file in bytes, 0 for no limit. Larger tables are split into parts of about this + * size like data files of sql scripts: orders.parquet, orders_2.parquet ... each part holds the next rows in + * the order of the table. */ - public Exporter.ExportConfiguration getConfiguration() { - return configuration; + public void setMaxFileSize(long maxFileSize) { + this.maxFileSize = Math.max(0, maxFileSize); } - public void setExportListener(ExportListener exportListener) { - this.exportListener = exportListener != null ? exportListener : ExportListener.NONE; - } - - /** - * Database type of schema.sql and constraints.sql. Default is the same as source database. - */ - public void setTargetDbType(DbType targetDbType) { - this.targetDbType = targetDbType; - } - - public void setIdentifierCase(IdentifierCase identifierCase) { - this.identifierCase = identifierCase; - } - - public void setTargetCatalogName(String targetCatalogName) { - this.targetCatalogName = targetCatalogName; - } - - public void setTargetSchemaName(String targetSchemaName) { - this.targetSchemaName = targetSchemaName; - } - - public void setTargetVersion(int majorVersion, int minorVersion) { - this.targetMajorVersion = majorVersion; - this.targetMinorVersion = minorVersion; + public long getMaxFileSize() { + return maxFileSize; } /** @@ -112,300 +74,45 @@ public String getCompression() { return compression; } - public File getRoot() { - return root; - } - - public List getWrittenFiles() { - return new ArrayList<>(writtenFiles); - } - - public void exportDdlAndData() throws Exception { - export(ExportMode.DDL_DATA); - } - - public void exportDdl() throws Exception { - export(ExportMode.DDL); - } - - public void exportData() throws Exception { - export(ExportMode.DATA); - } - - public void export(ExportMode exportMode) throws Exception { - writtenFiles.clear(); - if (configuration.getDbType() == null) { - configuration.setDbType(DbType.forUrl(configuration.getUrl())); - } - boolean successful = false; - exportListener.onStart(exportMode); - PhaseListener phaseListener = new PhaseListener(exportListener); - try { - ExportManifest manifest; - if (exportMode != ExportMode.DATA) { - exportSchema(phaseListener); - manifest = ExportManifest.read(root); - } else { - manifest = null; - } - if (manifest == null) { - manifest = new ExportManifest(); - manifest.getSource().setDbType(configuration.getDbType()); - manifest.getTarget().setDbType(targetDbType != null ? targetDbType : configuration.getDbType()); - } - List parquetFiles = new ArrayList<>(); - if (exportMode != ExportMode.DDL) { - exportRows(phaseListener, manifest, parquetFiles); - } - writeManifest(manifest, exportMode, parquetFiles); - successful = !phaseListener.isFailed(); - } catch (ExportCancelledException e) { - throw e; - } catch (Exception e) { - exportListener.onError(e.getMessage(), e); - throw e; - } finally { - exportListener.onEnd(successful); - } - } - - /** - * schema.sql and constraints.sql by the script exporter - */ - private void exportSchema(PhaseListener listener) throws Exception { - exportListener.onMessage("Export schema"); - ScriptExporter scriptExporter = new ScriptExporter(root, DataFileStrategy.SINGLE_FILE, 0); - copyConfiguration(configuration, scriptExporter.getConfiguration()); - scriptExporter.setTargetDbType(targetDbType); - scriptExporter.setIdentifierCase(identifierCase); - scriptExporter.setTargetCatalogName(targetCatalogName); - scriptExporter.setTargetSchemaName(targetSchemaName); - if (targetMajorVersion >= 0) { - scriptExporter.setTargetVersion(targetMajorVersion, targetMinorVersion); - } - scriptExporter.setConstraintsSeparated(true); - scriptExporter.setExportListener(listener.forPhase(false)); - scriptExporter.exportDdl(); - writtenFiles.addAll(scriptExporter.getWrittenFiles()); - } - - /** - * Rows are copied into the workspace and written as Parquet files, one file per table - */ - private void exportRows(PhaseListener listener, ExportManifest manifest, - List parquetFiles) throws Exception { - try (DuckDBWorkspace workspace = new DuckDBWorkspace(root)) { - exportListener.onMessage("Copy rows into workspace: " + workspace.getFile()); - ImportExporter importExporter = new ImportExporter(); - copyConfiguration(configuration, importExporter.getExportConfiguration()); - Exporter.ExportConfiguration source = importExporter.getExportConfiguration(); - source.setIdReused(true); - source.setTableRecreated(true); - source.setIndexIncluded(false); - source.setForeignKeyIncluded(false); - source.setCommentIncluded(false); - // Defaults of the same database type may use sequences - source.setSequenceIncluded(true); - ImportExportHandler.ImportConfiguration target = importExporter.getImportConfiguration(); - target.setDbType(DbType.DUCKDB); - target.setUrl(workspace.getUrl()); - target.setConnectionPoolEnabled(false); - DuckDBDialect dialect = new DuckDBDialect(); - dialect.setExactValues(true); - dialect.setExactTargetDbType(targetDbType != null ? targetDbType : configuration.getDbType()); - // Table names of the source are kept - dialect.setIdentifierCase(IdentifierCase.KEEP); - importExporter.setTargetDialect(dialect); - importExporter.setTargetConnectionFactory(new SharedConnectionFactory(workspace.getConnection())); - importExporter.setExportListener(listener.forPhase(true)); - importExporter.exportDdlAndData(); - if (listener.isFailed()) { - return; - } - - List tables = workspace.getTables(); - boolean schemaQualified = tables.stream().map(t -> t[0]).distinct().count() > 1; - DuckDBMetaDataOperations operations = new DuckDBMetaDataOperations(); - DatabaseMetaData metaData = workspace.getConnection().getMetaData(); - for (String[] table : tables) { - if (exportListener.isCancelled()) { - throw new ExportCancelledException(); - } - String schema = table[0]; - String tableName = table[1]; - String fileName = (schemaQualified ? LobManifestWriter.toFileName(schema) + "." : "") - + LobManifestWriter.toFileName(tableName) + PARQUET_EXTENSION; - File file = new File(new File(root, ScriptExportHandler.DATA_DIR_NAME), fileName); - Map fileMetadata = new LinkedHashMap<>(); - fileMetadata.put("sqljam_table", tableName); - if (sourceSchema(schema) != null) { - fileMetadata.put("sqljam_schema", sourceSchema(schema)); - } - fileMetadata.put("sqljam_source", String.valueOf(configuration.getDbType())); - workspace.writeParquet(schema, tableName, file, compression, fileMetadata); - writtenFiles.add(file); - long rows = workspace.countRows(schema, tableName); - exportListener.onMessage(String.format("Write %d rows of %s to %s", rows, tableName, file)); - - ExportManifest.TableEntry tableEntry = findTableEntry(manifest, sourceSchema(schema), tableName) - .orElseGet(() -> { - ExportManifest.TableEntry entry = new ExportManifest.TableEntry(); - entry.setSchema(sourceSchema(schema)); - entry.setName(tableName); - manifest.getTables().add(entry); - return entry; - }); - tableEntry.setRows(rows); - String path = ScriptExportHandler.DATA_DIR_NAME + "/" + fileName; - tableEntry.getDataFiles().clear(); - tableEntry.getDataFiles().add(path); - tableEntry.getIdentityColumns().clear(); - for (Map column : operations.getColumnInfos(metaData, null, schema, tableName)) { - if ("YES".equalsIgnoreCase((String) column.get("IS_AUTOINCREMENT"))) { - tableEntry.getIdentityColumns().add((String) column.get("COLUMN_NAME")); - } - } - parquetFiles.add(new ExportManifest.FileEntry(path, ExportManifest.FileType.PARQUET)); - } - } + @Override + protected DataFormat getDataFormat() { + return DataFormat.PARQUET; } - /** - * Schema of the source table, tables of sources without schemas are in the default schema of the workspace - */ - private String sourceSchema(String workspaceSchema) { - return configuration.getDbType() != null && !configuration.getDbType().isSchemaSupported() - && DuckDBWorkspace.DEFAULT_SCHEMA.equals(workspaceSchema) ? null : workspaceSchema; + @Override + protected String getExtension() { + return PARQUET_EXTENSION; } - private static Optional findTableEntry(ExportManifest manifest, String schema, - String table) { - return manifest.getTables().stream() - .filter(entry -> table.equals(entry.getName()) - && (schema == null || entry.getSchema() == null || schema.equals(entry.getSchema()))) - .findFirst(); + @Override + protected ExportManifest.FileType getFileType() { + return ExportManifest.FileType.PARQUET; } - /** - * Files in the order of importing: schema.sql, Parquet files, constraints.sql - */ - private void writeManifest(ExportManifest manifest, ExportMode exportMode, - List parquetFiles) throws Exception { - manifest.setDataFormat(DataFormat.PARQUET); - manifest.setExportMode(exportMode); - manifest.setStatus(ExportManifest.Status.COMPLETED); - manifest.getOptions().put("compression", compression); - List files = new ArrayList<>(); - for (ExportManifest.FileEntry entry : manifest.getFiles()) { - if (entry.getType() == ExportManifest.FileType.SCHEMA) { - files.add(entry); - } - } - files.addAll(parquetFiles); - for (ExportManifest.FileEntry entry : manifest.getFiles()) { - if (entry.getType() == ExportManifest.FileType.CONSTRAINTS) { - files.add(entry); - } + @Override + protected List writeFile(DuckDBWorkspace workspace, String schema, String table, File file, + Map metadata) throws Exception { + workspace.writeParquet(schema, table, file, compression, metadata); + if (maxFileSize <= 0 || file.length() <= maxFileSize) { + return Collections.singletonList(file); } - manifest.setFiles(files); - manifest.write(root); - File manifestFile = new File(root, ExportManifest.FILE_NAME); - if (!writtenFiles.contains(manifestFile)) { - writtenFiles.add(manifestFile); - } - exportListener.onMessage("Write manifest: " + manifestFile); - } - - /** - * Copies source settings and options, dialects are created by each exporter - */ - static void copyConfiguration(Exporter.ExportConfiguration from, Exporter.ExportConfiguration to) { - for (Field field : Exporter.ExportConfiguration.class.getDeclaredFields()) { - if (Modifier.isStatic(field.getModifiers()) || "dialect".equals(field.getName()) - || "sourceDialect".equals(field.getName())) { - continue; - } - try { - field.setAccessible(true); - field.set(to, field.get(from)); - } catch (IllegalAccessException e) { - throw new IllegalStateException(e); - } + // The size of the whole table decides the number of parts, rows are divided evenly + long rows = workspace.countRows(schema, table); + int parts = (int) Math.min(Math.max(rows, 1), (long) Math.ceil(file.length() / (maxFileSize * PART_FILL))); + long rowsPerPart = (rows + parts - 1) / parts; + String baseName = FilenameUtils.getBaseName(file.getName()); + List files = new ArrayList<>(); + for (int i = 0; i < parts && (long) i * rowsPerPart < rows; i++) { + File part = i == 0 ? file : new File(file.getParentFile(), baseName + "_" + (i + 1) + PARQUET_EXTENSION); + workspace.writeParquet(schema, table, part, compression, metadata, i * rowsPerPart, rowsPerPart); + files.add(part); } + return files; } - /** - * Forwards events of the schema phase and the rows phase as one export: messages and errors of both phases, - * progress of the rows phase only, start and end are reported by the parquet exporter - */ - static class PhaseListener { - - private final ExportListener delegate; - private boolean failed; - - PhaseListener(ExportListener delegate) { - this.delegate = delegate; - } - - boolean isFailed() { - return failed; - } - - ExportListener forPhase(boolean progress) { - return new ExportListener() { - - @Override - public void onTableStart(String catalog, String schema, String table, long totalRows) { - if (progress) { - delegate.onTableStart(catalog, schema, table, totalRows); - } - } - - @Override - public void onTableProgress(String catalog, String schema, String table, long rows, - long totalRows) { - if (progress) { - delegate.onTableProgress(catalog, schema, table, rows, totalRows); - } - } - - @Override - public void onTableEnd(String catalog, String schema, String table, long rows) { - if (progress) { - delegate.onTableEnd(catalog, schema, table, rows); - } - } - - @Override - public void onProgress(long processed, long total) { - if (progress) { - delegate.onProgress(processed, total); - } - } - - @Override - public void onMessage(String message) { - delegate.onMessage(message); - } - - @Override - public void onError(String message, Throwable e) { - failed = true; - delegate.onError(message, e); - } - - @Override - public void onEnd(boolean successful) { - if (!successful) { - failed = true; - } - } - - @Override - public boolean isCancelled() { - return delegate.isCancelled(); - } - }; - } + @Override + protected void putOptions(Map options) { + options.put("compression", compression); + options.put("maxFileSize", maxFileSize); } } diff --git a/src/main/java/com/github/sqljam/impexp/ParquetImporter.java b/src/main/java/com/github/sqljam/impexp/ParquetImporter.java index 220ab71..e5b8b9e 100644 --- a/src/main/java/com/github/sqljam/impexp/ParquetImporter.java +++ b/src/main/java/com/github/sqljam/impexp/ParquetImporter.java @@ -18,419 +18,37 @@ import java.io.File; import java.io.IOException; import java.sql.Connection; -import java.sql.DatabaseMetaData; import java.sql.SQLException; -import java.util.ArrayList; -import java.util.Collections; -import java.util.HashSet; -import java.util.LinkedHashMap; -import java.util.LinkedHashSet; import java.util.List; -import java.util.Map; -import java.util.Set; -import java.util.stream.Collectors; - -import org.apache.commons.lang3.StringUtils; -import org.apache.commons.lang3.Validate; -import com.github.sqljam.jdbc.SharedConnectionFactory; -import lombok.extern.slf4j.Slf4j; /** - * @Description: ParquetImporter imports Parquet files into a database by a connection. Packages exported by - * {@link ParquetExporter} are imported in the order of manifest.json: schema.sql, Parquet files into - * the created tables (identity values are kept and identities are reset), constraints.sql. Parquet - * files of other tools are imported into a new table created from the Parquet schema, or appended - * to an existing table. Files are read into a temporary DuckDB workspace, then rows are copied by - * {@link ImportExporter} with the dialect of the target database. + * @Description: ParquetImporter imports Parquet files into a database by a connection: packages exported by + * {@link ParquetExporter}, or Parquet files of other tools into a new table created from the Parquet + * schema or appended to an existing table * @Author: Fred Feng * @Date: 26/03/2023 * @Version 1.0.0 */ -@Slf4j -public class ParquetImporter extends AbstractFileImporter { - - /** - * CREATE drops and creates the table from the Parquet schema, APPEND inserts rows into the existing table - */ - public enum Mode { - CREATE, - APPEND - } - - private String targetCatalogName; - private String targetSchemaName; - private IdentifierCase identifierCase; - private int targetMajorVersion = -1; - private int targetMinorVersion = -1; - private int pageSize = Exporter.DEFAULT_PAGE_SIZE; - private long importedRows; +public class ParquetImporter extends DataFileImporter { public ParquetImporter(Connection connection, DbType dbType) { super(connection, dbType); } - /** - * Catalog of the table, the catalog of the connection by default - */ - public void setTargetCatalogName(String targetCatalogName) { - this.targetCatalogName = targetCatalogName; - } - - /** - * Schema of the table, the schema of the connection by default - */ - public void setTargetSchemaName(String targetSchemaName) { - this.targetSchemaName = targetSchemaName; - } - - public void setIdentifierCase(IdentifierCase identifierCase) { - this.identifierCase = identifierCase; - } - - public void setTargetVersion(int majorVersion, int minorVersion) { - this.targetMajorVersion = majorVersion; - this.targetMinorVersion = minorVersion; - } - - public void setPageSize(int pageSize) { - this.pageSize = pageSize; - } - - /** - * Rows imported from Parquet files - */ - public long getImportedRows() { - return importedRows; - } - - /** - * Imports Parquet files of other tools into a table - * - * @return number of imported rows - */ - public long importFiles(List files, String tableName, Mode mode) throws Exception { - Validate.notEmpty(files, "Parquet files must be required"); - Validate.notBlank(tableName, "Table name must be required"); - importedRows = 0; - boolean successful = false; - exportListener.onStart(mode == Mode.CREATE ? ExportMode.DDL_DATA : ExportMode.DATA); - try (DuckDBWorkspace workspace = new DuckDBWorkspace(workspaceDirectory)) { - long total = workspace.countParquetRows(files); - exportListener.onMessage(String.format("Import %d rows of %d Parquet files into %s", total, files.size(), - tableName)); - importTable(workspace, DuckDBWorkspace.DEFAULT_SCHEMA, tableName, files, Collections.emptyList(), mode, - null, null, new ProgressListener(exportListener, 0, total)); - executedCount = (int) Math.min(Integer.MAX_VALUE, importedRows); - successful = true; - } finally { - exportListener.onEnd(successful); - } - return importedRows; - } - - /** - * schema.sql, Parquet files (all together, in the order of foreign keys) and constraints.sql - */ @Override - protected void importManifest(File dir, ExportManifest manifest) throws IOException, SQLException { - if (manifest.getDataFormat() != DataFormat.PARQUET) { - throw new ImpExpException("Packages of sql scripts are imported by ScriptImporter"); - } - importedRows = 0; - boolean parquetImported = false; - for (ExportManifest.FileEntry fileEntry : manifest.getFiles()) { - File file = new File(dir, fileEntry.getPath()); - if (fileEntry.getType() == ExportManifest.FileType.PARQUET) { - if (!parquetImported) { - importPackageFiles(dir, manifest); - parquetImported = true; - } - } else { - runScript(file); - } - } - } - - private void importPackageFiles(File dir, ExportManifest manifest) throws SQLException { - long parquetBytes = manifest.getFiles().stream() - .filter(entry -> entry.getType() == ExportManifest.FileType.PARQUET) - .mapToLong(entry -> new File(dir, entry.getPath()).length()).sum(); - long startBytes = completedBytes; - ExportListener bytesListener = new ForwardingListener(exportListener) { - - @Override - public void onProgress(long processed, long total) { - // Rows are converted to bytes of Parquet files - long bytes = total > 0 ? parquetBytes * processed / total : 0; - exportListener.onProgress(Math.min(startBytes + bytes, totalBytes), totalBytes); - } - - @Override - public void onError(String message, Throwable e) { - failedCount++; - exportListener.onError(message, e); - } - }; - List tables = manifest.getTables().stream() - .filter(table -> !table.getDataFiles().isEmpty()).collect(Collectors.toList()); - IdentifierCase manifestCase = getIdentifierCase(manifest); - if (identifierCase == null) { - identifierCase = manifestCase; - } - try { - if (createDialect().isForeignKeyOrderRequired()) { - tables = sortByForeignKeys(tables, manifest); - } - long total = tables.stream().mapToLong(ExportManifest.TableEntry::getRows).sum(); - long completed = 0; - try (DuckDBWorkspace workspace = new DuckDBWorkspace(workspaceDirectory)) { - for (ExportManifest.TableEntry table : tables) { - if (exportListener.isCancelled()) { - throw new ExportCancelledException(); - } - List files = table.getDataFiles().stream().map(path -> new File(dir, path)) - .collect(Collectors.toList()); - String schema = StringUtils.defaultIfBlank(table.getSchema(), DuckDBWorkspace.DEFAULT_SCHEMA); - exportListener.onMessage(String.format("Import %d rows of %s", table.getRows(), table.getName())); - importTable(workspace, schema, table.getName(), files, table.getIdentityColumns(), Mode.APPEND, - manifest.getTarget().getCatalog(), manifest.getTarget().getSchema(), - new ProgressListener(bytesListener, completed, total)); - completed += table.getRows(); - } - } - } catch (ImpExpException | SQLException e) { - throw e; - } catch (Exception e) { - throw new ImpExpException(e.getMessage(), e); - } - executedCount += (int) Math.min(Integer.MAX_VALUE, importedRows); - completedBytes = startBytes; - completeBytes(parquetBytes); - } - - private void importTable(DuckDBWorkspace workspace, String schema, String tableName, List files, - List identityColumns, Mode mode, String packageCatalog, String packageSchema, - ProgressListener listener) throws Exception { - workspace.loadParquet(schema, tableName, files, identityColumns); - String catalog = StringUtils.defaultIfBlank(targetCatalogName, packageCatalog); - String targetSchema = StringUtils.defaultIfBlank(targetSchemaName, packageSchema); - if (mode == Mode.APPEND) { - // Values of generated and row version columns are maintained by the target database - List generated = getGeneratedColumns(tableName, catalog, targetSchema); - List dropped = new ArrayList<>(); - for (String[] column : workspace.describeTable(schema, tableName)) { - if (generated.stream().anyMatch(name -> name.equalsIgnoreCase(column[0]))) { - dropped.add(column[0]); - } - } - workspace.dropColumns(schema, tableName, dropped); - } - ImportExporter importExporter = new ImportExporter(); - Exporter.ExportConfiguration source = importExporter.getExportConfiguration(); - source.setDbType(DbType.DUCKDB); - source.setUrl(workspace.getUrl()); - source.setConnectionPoolEnabled(false); - source.setIncludedSchemaNames(new String[]{schema}); - source.setIncludedTableNames(new String[]{tableName}); - source.setPageSize(pageSize); - source.setIdReused(true); - source.setTableRecreated(mode == Mode.CREATE); - source.setIndexIncluded(false); - source.setForeignKeyIncluded(false); - source.setCommentIncluded(false); - source.setSequenceIncluded(false); - ImportExportHandler.ImportConfiguration target = importExporter.getImportConfiguration(); - target.setDbType(dbType); - target.setTargetCatalogName(catalog); - target.setTargetSchemaName(targetSchema); - importExporter.setTargetConnectionFactory(new SharedConnectionFactory(connection)); - if (identifierCase != null) { - importExporter.setIdentifierCase(identifierCase); - } - if (targetMajorVersion >= 0) { - importExporter.setTargetVersion(targetMajorVersion, targetMinorVersion); - } - // Tables go into the target schema or the default schema, not into a schema named as the workspace one - importExporter.setSourceSchemaPreserved(false); - importExporter.setExportListener(listener); - // The workspace is read by connections of the importer - workspace.closeConnection(); - importExporter.export(mode == Mode.CREATE ? ExportMode.DDL_DATA : ExportMode.DATA); - if (listener.getError() != null) { - throw new ImpExpException(listener.getError(), listener.getCause()); - } - importedRows += listener.getRows(); - } - - /** - * Dialect folding names as the importer does: the workspace (DuckDB) is the source - */ - private Dialect createDialect() { - Dialect dialect = dbType.createDialect(); - dialect.setSourceDbType(DbType.DUCKDB); - if (identifierCase != null) { - dialect.setIdentifierCase(identifierCase); - } - return dialect; - } - - private static IdentifierCase getIdentifierCase(ExportManifest manifest) { - Object option = manifest.getOptions().get("identifierCase"); - if (option == null) { - return null; - } - try { - return IdentifierCase.valueOf(option.toString()); - } catch (IllegalArgumentException e) { - return null; - } - } - - /** - * Generated (computed) and row version columns of the existing target table - */ - private List getGeneratedColumns(String tableName, String catalog, String schema) throws SQLException { - List columns = new ArrayList<>(); - MetaDataOperations operations = dbType.createMetaDataOperations(); - String tableSchema = StringUtils.defaultIfBlank(schema, connection.getSchema()); - String tableCatalog = dbType.isCatalogSupported() && dbType != DbType.ORACLE - ? StringUtils.defaultIfBlank(catalog, connection.getCatalog()) : null; - List> columnInfos; - try { - columnInfos = operations.getColumnInfos(connection.getMetaData(), tableCatalog, tableSchema, - createDialect().foldIdentifier(tableName)); - } catch (ImpExpException e) { - // The table does not exist, the importer reports it - columnInfos = Collections.emptyList(); - } - for (Map column : columnInfos) { - if ("YES".equalsIgnoreCase((String) column.get("IS_GENERATEDCOLUMN")) - || Boolean.TRUE.equals(column.get("IS_ROWVERSION"))) { - columns.add((String) column.get("COLUMN_NAME")); - } - } - return columns; - } - - /** - * Referenced tables are imported first, foreign keys are read from the tables created by schema.sql - */ - private List sortByForeignKeys(List tables, - ExportManifest manifest) throws SQLException { - Dialect dialect = createDialect(); - Map byName = new LinkedHashMap<>(); - for (ExportManifest.TableEntry table : tables) { - byName.put(dialect.foldIdentifier(table.getName()).toLowerCase(), table); - } - DatabaseMetaData metaData = connection.getMetaData(); - MetaDataOperations operations = dbType.createMetaDataOperations(); - String schema = StringUtils.defaultIfBlank(targetSchemaName, - StringUtils.defaultIfBlank(manifest.getTarget().getSchema(), connection.getSchema())); - Map> dependencies = new LinkedHashMap<>(); - for (Map.Entry entry : byName.entrySet()) { - Set referenced = new LinkedHashSet<>(); - for (Map keyInfo : operations.getImportedKeyInfos(metaData, null, schema, - dialect.foldIdentifier(entry.getValue().getName()))) { - String refTable = StringUtils.lowerCase((String) keyInfo.get("PKTABLE_NAME")); - if (refTable != null && !refTable.equals(entry.getKey()) && byName.containsKey(refTable)) { - referenced.add(refTable); - } - } - dependencies.put(entry.getKey(), referenced); - } - List sorted = new ArrayList<>(); - Set visited = new HashSet<>(); - for (String name : byName.keySet()) { - addInOrder(name, byName, dependencies, visited, new HashSet<>(), sorted); - } - return sorted; + protected DataFormat getDataFormat() { + return DataFormat.PARQUET; } - private static void addInOrder(String name, Map tables, - Map> dependencies, Set visited, Set visiting, - List sorted) { - if (visited.contains(name) || !visiting.add(name)) { - return; - } - for (String referenced : dependencies.getOrDefault(name, Collections.emptySet())) { - addInOrder(referenced, tables, dependencies, visited, visiting, sorted); - } - visited.add(name); - sorted.add(tables.get(name)); + @Override + protected ExportManifest.FileType getFileType() { + return ExportManifest.FileType.PARQUET; } - /** - * Forwards events of a table import: progress of the table is added to rows of tables imported before, start - * and end of the inner import are not forwarded, the first error is kept - */ - static class ProgressListener implements ExportListener { - - private final ExportListener delegate; - private final long completed; - private final long total; - private long rows; - private String error; - private Throwable cause; - - ProgressListener(ExportListener delegate, long completed, long total) { - this.delegate = delegate; - this.completed = completed; - this.total = total; - } - - long getRows() { - return rows; - } - - String getError() { - return error; - } - - Throwable getCause() { - return cause; - } - - @Override - public void onTableStart(String catalogName, String schemaName, String tableName, long totalRows) { - delegate.onTableStart(catalogName, schemaName, tableName, totalRows); - } - - @Override - public void onTableProgress(String catalogName, String schemaName, String tableName, long processedRows, - long totalRows) { - delegate.onTableProgress(catalogName, schemaName, tableName, processedRows, totalRows); - } - - @Override - public void onTableEnd(String catalogName, String schemaName, String tableName, long processedRows) { - rows += processedRows; - delegate.onTableEnd(catalogName, schemaName, tableName, processedRows); - } - - @Override - public void onProgress(long processed, long tableTotal) { - if (total > 0) { - delegate.onProgress(Math.min(completed + processed, total), total); - } - } - - @Override - public void onMessage(String message) { - delegate.onMessage(message); - } - - @Override - public void onError(String message, Throwable e) { - if (error == null) { - error = message; - cause = e; - } - delegate.onError(message, e); - } - - @Override - public boolean isCancelled() { - return delegate.isCancelled(); - } + @Override + protected void load(DuckDBWorkspace workspace, String schema, String table, List files, + List identityColumns, ExportManifest.TableEntry tableEntry) throws SQLException, + IOException { + workspace.loadParquet(schema, table, files, identityColumns); } } diff --git a/src/main/java/com/github/sqljam/impexp/ScriptExportHandler.java b/src/main/java/com/github/sqljam/impexp/ScriptExportHandler.java index ef92465..5ac8074 100644 --- a/src/main/java/com/github/sqljam/impexp/ScriptExportHandler.java +++ b/src/main/java/com/github/sqljam/impexp/ScriptExportHandler.java @@ -198,6 +198,7 @@ public void finish(boolean successful) throws Exception { if (configuration != null) { Map options = manifest.getOptions(); options.put("tableRecreated", configuration.isTableRecreated()); + options.put("writeMode", configuration.getWriteMode().name()); options.put("idReused", configuration.isIdReused()); options.put("indexIncluded", configuration.isIndexIncluded()); options.put("foreignKeyIncluded", configuration.isForeignKeyIncluded()); @@ -492,6 +493,24 @@ public void exportData(String catalogName, String[] columnNames; String[] columnStringValues; String insertSql; + // Upserts update the rows of the same key, a table without key is inserted + boolean upsert = configuration != null && configuration.getWriteMode().isKeyed(); + String[] upsertKeys = upsert ? getUpsertKeys(tableName, tableMetaData, template.keySet()) : new String[0]; + // Values of indexed columns are kept by databases which can not update them (DuckDB) + Set uniqueColumns = new HashSet<>(); + if (upsert && !dialect.isIndexedColumnUpdatable()) { + tableMetaData.getIndexMetaDatas().forEach(index -> uniqueColumns.add(index.getColumnName() + .toLowerCase(Locale.ENGLISH))); + // Columns of foreign keys are checked like indexed columns + tableMetaData.getForeignKeyMetaDatas().forEach(key -> key.getDetails().forEach(detail -> uniqueColumns.add( + String.valueOf(detail.get("FKCOLUMN_NAME")).toLowerCase(Locale.ENGLISH)))); + } + // Existing rows are kept when they are skipped + String[] upsertUpdates = !upsert || configuration.getWriteMode() == WriteMode.SKIP_EXISTING ? new String[0] + : Arrays.stream(template.keySet().toArray(new String[0])) + .filter(name -> Arrays.stream(upsertKeys).noneMatch(name::equalsIgnoreCase) + && identityColumnNames.stream().noneMatch(name::equalsIgnoreCase) + && !uniqueColumns.contains(name.toLowerCase(Locale.ENGLISH))).toArray(String[]::new); for (Map row : rowList) { long rowNumber = rowNumbers.merge(tableMetaData, 1L, Long::sum); columnNames = row.keySet().toArray(new String[0]); @@ -520,7 +539,9 @@ public void exportData(String catalogName, getLobManifestWriter(catalogName).writeRow(dialect.getTargetSchemaName(schemaName), dialect.foldIdentifier(tableName), key, lobValues, rowNumber); } - insertSql = dialect.getInsertTableStatement(catalogName, schemaName, tableName, columnNames, + insertSql = upsertKeys.length > 0 ? dialect.getUpsertStatement(catalogName, schemaName, tableName, + columnNames, upsertKeys, upsertUpdates, columnStringValues) + : dialect.getInsertTableStatement(catalogName, schemaName, tableName, columnNames, columnStringValues); sqlLines.add(insertSql); } @@ -549,6 +570,32 @@ public void exportData(String catalogName, this.lastTableName = tableName; } + /** + * Key columns of upserts: the chosen columns or the primary key, recorded in the manifest + * + * @return the keys as named in the columns, empty if the table has no key and is inserted + */ + private String[] getUpsertKeys(String tableName, TableMetaData tableMetaData, Set columns) { + List keys = configuration.getKeyColumnNames().entrySet().stream() + .filter(entry -> entry.getKey().equalsIgnoreCase(tableName)).map(Map.Entry::getValue).findFirst() + .orElse(null); + if (CollectionUtils.isEmpty(keys)) { + keys = tableMetaData.getPrimaryKeyColumnNames(); + } + List resolved = new ArrayList<>(); + for (String key : keys) { + columns.stream().filter(key::equalsIgnoreCase).findFirst().ifPresent(resolved::add); + } + if (resolved.size() != keys.size()) { + resolved.clear(); + } + ExportManifest.TableEntry entry = getTableEntry(tableMetaData); + if (!resolved.equals(entry.getKeyColumns())) { + entry.setKeyColumns(new ArrayList<>(resolved)); + } + return resolved.toArray(new String[0]); + } + private DataFile getDataFile(String catalogName, String tableName) { File catalogDir = getCatalogDir(catalogName); if (dataFileStrategy == DataFileStrategy.FILE_PER_TABLE) { diff --git a/src/main/java/com/github/sqljam/impexp/ScriptImporter.java b/src/main/java/com/github/sqljam/impexp/ScriptImporter.java index 25010e0..690fb68 100644 --- a/src/main/java/com/github/sqljam/impexp/ScriptImporter.java +++ b/src/main/java/com/github/sqljam/impexp/ScriptImporter.java @@ -124,8 +124,10 @@ protected long getManifestBytes(File dir, ExportManifest manifest) { @Override protected void importManifest(File dir, ExportManifest manifest) throws IOException, SQLException { - if (manifest.getDataFormat() == DataFormat.PARQUET) { - throw new ImpExpException("Packages of Parquet files are imported by ParquetImporter"); + if (manifest.getDataFormat() != null && manifest.getDataFormat() != DataFormat.SQL) { + throw new ImpExpException(String.format("Packages of %s files are imported by %s", + manifest.getDataFormat(), manifest.getDataFormat() == DataFormat.CSV ? "CsvImporter" + : "ParquetImporter")); } for (ExportManifest.FileEntry fileEntry : manifest.getFiles()) { File file = new File(dir, fileEntry.getPath()); diff --git a/src/main/java/com/github/sqljam/impexp/SqlScriptRunner.java b/src/main/java/com/github/sqljam/impexp/SqlScriptRunner.java index f86663f..6f3bbb9 100644 --- a/src/main/java/com/github/sqljam/impexp/SqlScriptRunner.java +++ b/src/main/java/com/github/sqljam/impexp/SqlScriptRunner.java @@ -29,6 +29,7 @@ import java.util.ArrayList; import java.util.List; import java.util.Locale; +import java.util.function.Predicate; import java.util.regex.Pattern; import org.apache.commons.lang3.StringUtils; @@ -68,13 +69,19 @@ public class SqlScriptRunner { */ public static final int ORACLE_BLOCK_SIZE = 100; - private static final Pattern INSERT_STATEMENT = Pattern.compile("(?is)^INSERT\\s+INTO\\s.*"); + private static final Pattern INSERT_STATEMENT = Pattern.compile("(?is)^(INSERT|MERGE)\\s+INTO\\s.*"); + private static final Pattern MERGE_STATEMENT = Pattern.compile("(?is)^MERGE\\s+INTO\\s.*"); private final Connection connection; private boolean stopOnError = true; private int batchCommitSize = 500; private int batchSize = DEFAULT_BATCH_SIZE; private Boolean oracle; + private Boolean sqlServer; + /** + * Errors of statements which create existing objects, they are skipped, e.g. tables which are not dropped + */ + private Predicate skippedError; private ExportListener exportListener = ExportListener.NONE; private Statement statement; /** @@ -91,6 +98,13 @@ public SqlScriptRunner(Connection connection) { this.connection = connection; } + /** + * Statements failing with these errors are skipped instead of failed, e.g. objects which exist already + */ + public void setSkippedError(Predicate skippedError) { + this.skippedError = skippedError; + } + public void setStopOnError(boolean stopOnError) { this.stopOnError = stopOnError; } @@ -131,7 +145,8 @@ public void runScript(Reader reader) throws IOException, SQLException { int uncommitted = 0; int index = 0; exportListener.onProgress(0, total); - for (String sql : statements) { + for (String statementText : statements) { + String sql = terminate(statementText); index++; if (exportListener.isCancelled()) { throw new ExportCancelledException(); @@ -219,6 +234,8 @@ static boolean isBatchable(String sql) { * Executes and commits a batch. A failed batch is rolled back and executed statement by statement */ private void executeBatch(List batch) throws SQLException { + // A transaction of the whole import goes on from the savepoint, earlier rows are kept + Savepoint savepoint = !stopOnError && transactional ? setSavepointQuietly() : null; try { if (isOracle()) { for (String sql : getOracleBlockStatements(batch)) { @@ -230,6 +247,7 @@ private void executeBatch(List batch) throws SQLException { } statement().executeBatch(); } + releaseSavepoint(savepoint); commit(); executedCount += batch.size(); } catch (SQLException e) { @@ -237,7 +255,11 @@ private void executeBatch(List batch) throws SQLException { throw e; } statement().clearBatch(); - rollback(); + if (savepoint != null) { + connection.rollback(savepoint); + } else { + rollback(); + } if (log.isDebugEnabled()) { log.debug("Batch failed, statements are executed one by one: {}", e.getMessage()); } @@ -255,16 +277,26 @@ private void executeBatch(List batch) throws SQLException { * keeps the transaction usable after a failure (PostgreSQL aborts the whole transaction otherwise) */ private boolean execute(String sql) throws SQLException { - Savepoint savepoint = stopOnError || !transactional ? null : connection.setSavepoint(); + Savepoint savepoint = (stopOnError && skippedError == null) || !transactional ? null + : setSavepointQuietly(); try { statement().execute(sql); executedCount++; releaseSavepoint(savepoint); return true; } catch (SQLException e) { + if (skippedError != null && skippedError.test(e)) { + // The object exists, the transaction goes on from the savepoint + rollbackQuietly(savepoint); + if (log.isInfoEnabled()) { + log.info("Exists already, skipped: {}", StringUtils.abbreviate(sql, 200)); + } + return false; + } failedCount++; - String message = String.format("Unable to execute sql: %s, cause: %s", StringUtils.abbreviate(sql, 500), - e.getMessage()); + // Statements with errors are skipped when the import goes on + String message = String.format(stopOnError ? "Unable to execute sql: %s, cause: %s" + : "Skipped sql: %s, cause: %s", StringUtils.abbreviate(sql, 500), e.getMessage()); if (log.isErrorEnabled()) { log.error(message); } @@ -291,6 +323,42 @@ private void releaseSavepoint(Savepoint savepoint) { } } + private Savepoint setSavepointQuietly() { + try { + return connection.setSavepoint(); + } catch (SQLException e) { + return null; + } + } + + private void rollbackQuietly(Savepoint savepoint) { + if (savepoint != null) { + try { + connection.rollback(savepoint); + } catch (SQLException ignored) { + // DDL of some databases ends the transaction and its savepoints + } + } + } + + /** + * A MERGE statement of SQL Server must end with a semicolon, scripts are parsed without end marks + */ + private String terminate(String sql) throws SQLException { + if (MERGE_STATEMENT.matcher(sql).matches() && !sql.trim().endsWith(";") && isSqlServer()) { + return sql.trim() + ";"; + } + return sql; + } + + private boolean isSqlServer() throws SQLException { + if (sqlServer == null) { + String productName = connection.getMetaData().getDatabaseProductName(); + sqlServer = productName != null && productName.toLowerCase(Locale.ENGLISH).contains("sql server"); + } + return sqlServer; + } + private boolean isOracle() throws SQLException { if (oracle == null) { String productName = connection.getMetaData().getDatabaseProductName(); diff --git a/src/main/java/com/github/sqljam/impexp/WriteMode.java b/src/main/java/com/github/sqljam/impexp/WriteMode.java new file mode 100644 index 0000000..25053c9 --- /dev/null +++ b/src/main/java/com/github/sqljam/impexp/WriteMode.java @@ -0,0 +1,53 @@ +/* + * Copyright 2023-2026 Fred Feng + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.github.sqljam.impexp; + +/** + * @Description: WriteMode of rows in the target table: INSERT adds the rows, UPSERT updates the rows of the same key + * and inserts the others, SKIP_EXISTING inserts the rows whose key is missing and keeps the others. + * The key is the primary key of the table by default, or the columns chosen by the user. A table + * without both is inserted. + * @Author: Fred Feng + * @Date: 26/03/2023 + * @Version 1.0.0 + */ +public enum WriteMode { + + INSERT, + + UPSERT, + + SKIP_EXISTING; + + /** + * Whether rows are matched by their key + */ + public boolean isKeyed() { + return this != INSERT; + } + + /** + * Write mode of a name, e.g. of a manifest, INSERT if it is unknown + */ + public static WriteMode of(Object name) { + for (WriteMode mode : values()) { + if (mode.name().equals(String.valueOf(name))) { + return mode; + } + } + return INSERT; + } +} diff --git a/src/main/java/com/github/sqljam/impexp/db/ClickHouseDialect.java b/src/main/java/com/github/sqljam/impexp/db/ClickHouseDialect.java index e59906e..02a191e 100644 --- a/src/main/java/com/github/sqljam/impexp/db/ClickHouseDialect.java +++ b/src/main/java/com/github/sqljam/impexp/db/ClickHouseDialect.java @@ -531,4 +531,12 @@ public Object getJdbcValue(Object value, int sourceDataType, String sourceTypeNa } return result; } + + /** + * ClickHouse has no update of a row by its key + */ + @Override + public boolean isUpsertSupported() { + return false; + } } diff --git a/src/main/java/com/github/sqljam/impexp/db/DuckDBDialect.java b/src/main/java/com/github/sqljam/impexp/db/DuckDBDialect.java index 97cd405..505a223 100644 --- a/src/main/java/com/github/sqljam/impexp/db/DuckDBDialect.java +++ b/src/main/java/com/github/sqljam/impexp/db/DuckDBDialect.java @@ -15,11 +15,20 @@ */ package com.github.sqljam.impexp.db; +import java.sql.Connection; +import java.sql.PreparedStatement; +import java.sql.ResultSet; +import java.sql.SQLException; import java.sql.Timestamp; import java.sql.Types; import java.time.LocalDateTime; import java.time.OffsetDateTime; import java.time.OffsetTime; +import java.util.ArrayList; +import java.util.HashSet; +import java.util.List; +import java.util.Locale; +import java.util.Set; import java.util.UUID; import org.apache.commons.lang3.StringUtils; @@ -525,4 +534,66 @@ public Object getJdbcValue(Object value, int sourceDataType, String sourceTypeNa } return result; } + + /** + * An update of an indexed column is a delete and an insert in DuckDB, which fails for keys and for rows + * referenced by foreign keys. Upserts keep the values of such columns. + */ + @Override + public boolean isIndexedColumnUpdatable() { + return false; + } + + /** + * Columns of indexes, keys and unique constraints from the catalog functions of DuckDB, names are matched + * without case like identifiers of DuckDB + */ + @Override + public Set getIndexedColumns(Connection connection, String catalog, String schema, String tableName) + throws SQLException { + Set columns = new HashSet<>(); + String constraints = "SELECT unnest(constraint_column_names) FROM duckdb_constraints() WHERE lower(schema_name) = lower(?)" + + " AND lower(table_name) = lower(?) AND constraint_type IN ('PRIMARY KEY', 'UNIQUE', 'FOREIGN KEY')"; + String indexes = "SELECT expressions FROM duckdb_indexes() WHERE lower(schema_name) = lower(?)" + + " AND lower(table_name) = lower(?)"; + for (String sql : new String[]{constraints, indexes}) { + try (PreparedStatement ps = connection.prepareStatement(sql)) { + ps.setString(1, StringUtils.defaultIfBlank(schema, "main")); + ps.setString(2, tableName); + try (ResultSet rs = ps.executeQuery()) { + while (rs.next()) { + // Expressions of indexes are a list of column names, e.g. [dept_id, name] + for (String name : StringUtils.defaultString(rs.getString(1)).replaceAll("[\\[\\]\"']", "") + .split(",")) { + if (!name.isBlank()) { + columns.add(name.trim().toLowerCase(Locale.ENGLISH)); + } + } + } + } + } + } + return columns; + } + + /** + * Columns of the primary key from the catalog functions of DuckDB + */ + @Override + public List getPrimaryKeyColumns(Connection connection, String catalog, String schema, String tableName) + throws SQLException { + String sql = "SELECT unnest(constraint_column_names) FROM duckdb_constraints() WHERE lower(schema_name) = lower(?)" + + " AND lower(table_name) = lower(?) AND constraint_type = 'PRIMARY KEY'"; + List columns = new ArrayList<>(); + try (PreparedStatement ps = connection.prepareStatement(sql)) { + ps.setString(1, StringUtils.defaultIfBlank(schema, "main")); + ps.setString(2, tableName); + try (ResultSet rs = ps.executeQuery()) { + while (rs.next()) { + columns.add(rs.getString(1)); + } + } + } + return columns; + } } diff --git a/src/main/java/com/github/sqljam/impexp/db/H2Dialect.java b/src/main/java/com/github/sqljam/impexp/db/H2Dialect.java index 41f6dfc..96280f6 100644 --- a/src/main/java/com/github/sqljam/impexp/db/H2Dialect.java +++ b/src/main/java/com/github/sqljam/impexp/db/H2Dialect.java @@ -17,6 +17,8 @@ import java.nio.charset.StandardCharsets; import java.sql.Types; +import java.util.Arrays; +import java.util.stream.Collectors; import org.apache.commons.lang3.StringUtils; @@ -299,4 +301,24 @@ public String getSelectColumnExpression(String columnName, String typeName) { } return super.getSelectColumnExpression(columnName, typeName); } + + /** + * MERGE INTO ... KEY ... VALUES: the row of the same keys is updated, otherwise inserted. Without columns to + * update, MERGE ... USING ... WHEN NOT MATCHED inserts a missing row and keeps the existing one. + */ + @Override + public String getUpsertStatement(String catalog, String schema, String tableName, String[] columnNames, + String[] keyColumnNames, String[] updateColumnNames, Object[] values) { + if (updateColumnNames.length == 0) { + return String.format("MERGE INTO %s AS tgt USING (VALUES (%s)) AS src (%s) ON %s WHEN NOT MATCHED THEN" + + " INSERT (%s) VALUES (%s)", getQualifiedTableName(catalog, schema, tableName), + StringUtils.join(values, ","), getIdentifiers(columnNames), Arrays.stream(keyColumnNames) + .map(name -> "tgt." + getIdentifier(name) + " = src." + getIdentifier(name)) + .collect(Collectors.joining(" AND ")), getIdentifiers(columnNames), + Arrays.stream(columnNames).map(name -> "src." + getIdentifier(name)) + .collect(Collectors.joining(", "))); + } + return String.format("MERGE INTO %s(%s) KEY (%s) VALUES (%s)", getQualifiedTableName(catalog, schema, + tableName), getIdentifiers(columnNames), getIdentifiers(keyColumnNames), StringUtils.join(values, ",")); + } } diff --git a/src/main/java/com/github/sqljam/impexp/db/MariaDBDialect.java b/src/main/java/com/github/sqljam/impexp/db/MariaDBDialect.java index d68c840..f2fb670 100644 --- a/src/main/java/com/github/sqljam/impexp/db/MariaDBDialect.java +++ b/src/main/java/com/github/sqljam/impexp/db/MariaDBDialect.java @@ -87,4 +87,12 @@ public String getCreateSequenceStatement(String catalog, String schema, String s public int getNullSqlType(int sqlType) { return sqlType == Types.OTHER ? Types.VARCHAR : super.getNullSqlType(sqlType); } + + /** + * MariaDB has no row alias of INSERT, VALUES() is the value of the row + */ + @Override + protected boolean isRowAliasSupported() { + return false; + } } diff --git a/src/main/java/com/github/sqljam/impexp/db/MySQLDialect.java b/src/main/java/com/github/sqljam/impexp/db/MySQLDialect.java index 554bd1f..8cb84a3 100644 --- a/src/main/java/com/github/sqljam/impexp/db/MySQLDialect.java +++ b/src/main/java/com/github/sqljam/impexp/db/MySQLDialect.java @@ -17,6 +17,7 @@ import java.nio.ByteBuffer; import java.nio.ByteOrder; +import java.sql.SQLException; import java.sql.Types; import java.util.Arrays; import java.util.Locale; @@ -534,4 +535,37 @@ protected boolean isExpressionDefaultSupported() { protected boolean isDatetimeCurrentTimestampSupported() { return true; } + + /** + * INSERT ... ON DUPLICATE KEY UPDATE, the values of the row by the row alias of MySQL 8.0.19 or by VALUES() + * which is deprecated since then. A row of keys only keeps the existing row. + */ + @Override + public String getUpsertStatement(String catalog, String schema, String tableName, String[] columnNames, + String[] keyColumnNames, String[] updateColumnNames, Object[] values) { + boolean alias = isRowAliasSupported(); + String insert = String.format("INSERT INTO %s(%s) VALUES (%s)%s", getQualifiedTableName(catalog, schema, + tableName), getIdentifiers(columnNames), StringUtils.join(values, ","), alias ? " AS new_row" : ""); + String[] updated = updateColumnNames.length > 0 ? updateColumnNames : new String[]{keyColumnNames[0]}; + // A row of keys only sets a key to its own value, the existing row is kept + String updates = Arrays.stream(updated).map(name -> getIdentifier(name) + " = " + (alias ? "new_row." + + getIdentifier(name) : "VALUES(" + getIdentifier(name) + ")")).collect(Collectors.joining(", ")); + return String.format("%s ON DUPLICATE KEY UPDATE %s", insert, updates); + } + + /** + * Row alias of INSERT (MySQL 8.0.19), assumed for an unknown version and from MySQL 8.1 + */ + protected boolean isRowAliasSupported() { + return getDatabaseMajorVersion() < 0 || isVersionAtLeast(8, 1); + } + + /** + * Table exists (1050), duplicate key name (1061), duplicate foreign key (1826), multiple primary keys (1068) + */ + @Override + public boolean isAlreadyExistsError(SQLException e) { + int code = e.getErrorCode(); + return code == 1050 || code == 1061 || code == 1826 || code == 1068 || super.isAlreadyExistsError(e); + } } diff --git a/src/main/java/com/github/sqljam/impexp/db/OracleDialect.java b/src/main/java/com/github/sqljam/impexp/db/OracleDialect.java index aaddba7..32f05a9 100644 --- a/src/main/java/com/github/sqljam/impexp/db/OracleDialect.java +++ b/src/main/java/com/github/sqljam/impexp/db/OracleDialect.java @@ -16,11 +16,13 @@ package com.github.sqljam.impexp.db; import java.math.BigDecimal; +import java.sql.SQLException; import java.sql.Types; import java.time.LocalTime; import java.time.OffsetDateTime; import java.util.Arrays; import java.util.Map; +import java.util.stream.Collectors; import org.apache.commons.lang3.StringUtils; import com.github.sqljam.impexp.DbType; @@ -478,4 +480,37 @@ public String getSelectColumnExpression(String columnName, String typeName) { } return column; } + + /** + * MERGE with the row selected from dual, the keys join the existing row + */ + @Override + public String getUpsertStatement(String catalog, String schema, String tableName, String[] columnNames, + String[] keyColumnNames, String[] updateColumnNames, Object[] values) { + StringBuilder sql = new StringBuilder("MERGE INTO ").append(getQualifiedTableName(catalog, schema, tableName)) + .append(" tgt USING (SELECT "); + for (int i = 0; i < columnNames.length; i++) { + sql.append(i > 0 ? ", " : "").append(values[i]).append(" ").append(getIdentifier(columnNames[i])); + } + sql.append(" FROM dual) src ON (").append(Arrays.stream(keyColumnNames).map(name -> "tgt." + getIdentifier(name) + + " = src." + getIdentifier(name)).collect(Collectors.joining(" AND "))).append(")"); + if (updateColumnNames.length > 0) { + sql.append(" WHEN MATCHED THEN UPDATE SET ").append(Arrays.stream(updateColumnNames).map(name -> "tgt." + + getIdentifier(name) + " = src." + getIdentifier(name)).collect(Collectors.joining(", "))); + } + return sql.append(" WHEN NOT MATCHED THEN INSERT (").append(getIdentifiers(columnNames)).append(") VALUES (") + .append(Arrays.stream(columnNames).map(name -> "src." + getIdentifier(name)) + .collect(Collectors.joining(", "))).append(")").toString(); + } + + /** + * Name used (ORA-00955), keys and constraints which exist (02260, 02261, 02264, 02275), column list indexed + * (01408) + */ + @Override + public boolean isAlreadyExistsError(SQLException e) { + int code = e.getErrorCode(); + return code == 955 || code == 2260 || code == 2261 || code == 2264 || code == 2275 || code == 1408 + || super.isAlreadyExistsError(e); + } } diff --git a/src/main/java/com/github/sqljam/impexp/db/PostgreSQL9Dialect.java b/src/main/java/com/github/sqljam/impexp/db/PostgreSQL9Dialect.java index ee7a7dd..ff02216 100644 --- a/src/main/java/com/github/sqljam/impexp/db/PostgreSQL9Dialect.java +++ b/src/main/java/com/github/sqljam/impexp/db/PostgreSQL9Dialect.java @@ -57,4 +57,12 @@ public String[] getStatementBeforeSequenceCreated(String catalog, String schema, return new String[]{String.format("DROP SEQUENCE IF EXISTS %s CASCADE", getQualifiedSequenceName(catalog, schema, sequenceName))}; } + + /** + * ON CONFLICT is supported from PostgreSQL 9.5 + */ + @Override + public boolean isUpsertSupported() { + return getDatabaseMajorVersion() < 0 || isVersionAtLeast(9, 5); + } } diff --git a/src/main/java/com/github/sqljam/impexp/db/SQLServerDialect.java b/src/main/java/com/github/sqljam/impexp/db/SQLServerDialect.java index df01d8e..934ccd8 100644 --- a/src/main/java/com/github/sqljam/impexp/db/SQLServerDialect.java +++ b/src/main/java/com/github/sqljam/impexp/db/SQLServerDialect.java @@ -15,6 +15,7 @@ */ package com.github.sqljam.impexp.db; +import java.sql.SQLException; import java.sql.Types; import java.time.LocalDate; import java.time.LocalDateTime; @@ -515,4 +516,34 @@ public String getCreateSchemaIfNotExistsStatement(String schema) { return String.format("IF SCHEMA_ID(N'%s') IS NULL EXEC(N'CREATE SCHEMA %s')", schema.replace("'", "''"), quoteIdentifier(schema).replace("'", "''")); } + + /** + * MERGE with the row of a VALUES table, a MERGE statement must end with a semicolon + */ + @Override + public String getUpsertStatement(String catalog, String schema, String tableName, String[] columnNames, + String[] keyColumnNames, String[] updateColumnNames, Object[] values) { + StringBuilder sql = new StringBuilder("MERGE INTO ").append(getQualifiedTableName(catalog, schema, tableName)) + .append(" AS tgt USING (VALUES (").append(StringUtils.join(values, ",")).append(")) AS src (") + .append(getIdentifiers(columnNames)).append(") ON ").append(Arrays.stream(keyColumnNames) + .map(name -> "tgt." + getIdentifier(name) + " = src." + getIdentifier(name)) + .collect(Collectors.joining(" AND "))); + if (updateColumnNames.length > 0) { + sql.append(" WHEN MATCHED THEN UPDATE SET ").append(Arrays.stream(updateColumnNames).map(name -> "tgt." + + getIdentifier(name) + " = src." + getIdentifier(name)).collect(Collectors.joining(", "))); + } + return sql.append(" WHEN NOT MATCHED THEN INSERT (").append(getIdentifiers(columnNames)).append(") VALUES (") + .append(Arrays.stream(columnNames).map(name -> "src." + getIdentifier(name)) + .collect(Collectors.joining(", "))).append(");").toString(); + } + + /** + * Object exists (2714), index exists (1913), table has a primary key (1779), constraint of a column exists + * (1781) + */ + @Override + public boolean isAlreadyExistsError(SQLException e) { + int code = e.getErrorCode(); + return code == 2714 || code == 1913 || code == 1779 || code == 1781 || super.isAlreadyExistsError(e); + } } diff --git a/src/main/java/com/github/sqljam/jdbc/ManualTransaction.java b/src/main/java/com/github/sqljam/jdbc/ManualTransaction.java new file mode 100644 index 0000000..54cfb53 --- /dev/null +++ b/src/main/java/com/github/sqljam/jdbc/ManualTransaction.java @@ -0,0 +1,306 @@ +/* + * Copyright 2023-2026 Fred Feng + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.github.sqljam.jdbc; + +import java.lang.reflect.InvocationTargetException; +import java.lang.reflect.Method; +import java.lang.reflect.Proxy; +import java.sql.Connection; +import java.sql.SQLException; +import java.sql.Statement; +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; +import java.util.Locale; +import java.util.regex.Pattern; + +import lombok.extern.slf4j.Slf4j; + +/** + * @Description: ManualTransaction runs an import in one transaction which is committed or rolled back by the user + * afterwards. Importers work with {@link #getConnection()}: their commits are ignored, auto commit + * stays off, and a rollback (after an error) rolls back the whole transaction, so an import is all or + * nothing. Databases which commit around DDL statements (MySQL, MariaDB, Oracle, H2) would commit the + * written rows with the first DDL statement after them, such statements (foreign keys, identity and + * sequence values) are kept and executed after {@link #commit()}. + * @Author: Fred Feng + * @Date: 26/03/2023 + * @Version 1.0.0 + */ +@Slf4j +public class ManualTransaction implements AutoCloseable { + + private static final Pattern DDL = Pattern.compile("^(CREATE|ALTER|DROP|TRUNCATE|RENAME|COMMENT|GRANT|REVOKE)\\b.*", + Pattern.CASE_INSENSITIVE | Pattern.DOTALL); + private static final Pattern PLSQL_DDL = Pattern.compile("^(BEGIN|DECLARE)\\b.*EXECUTE\\s+IMMEDIATE.*", + Pattern.CASE_INSENSITIVE | Pattern.DOTALL); + private static final Pattern DML = Pattern.compile("^(INSERT|UPDATE|DELETE|MERGE|UPSERT|REPLACE)\\b.*", + Pattern.CASE_INSENSITIVE | Pattern.DOTALL); + /** + * PL/SQL blocks of INSERT statements, e.g. batches of Oracle scripts + */ + private static final Pattern PLSQL_DML = Pattern.compile("^(BEGIN|DECLARE)\\b.*\\b(INSERT|UPDATE|DELETE|MERGE)\\b.*", + Pattern.CASE_INSENSITIVE | Pattern.DOTALL); + + private final Connection connection; + private final boolean transactionalDdl; + private final Connection proxy; + private final List deferredStatements = Collections.synchronizedList(new ArrayList<>()); + private volatile boolean rowsWritten; + private volatile boolean rolledBack; + private volatile boolean finished; + + /** + * @param connection the connection of the target database, it is closed by {@link #close()} + * @param transactionalDdl whether DDL statements are part of the transaction, e.g. PostgreSQL + */ + public ManualTransaction(Connection connection, boolean transactionalDdl) throws SQLException { + this.connection = connection; + this.transactionalDdl = transactionalDdl; + connection.setAutoCommit(false); + this.proxy = (Connection) Proxy.newProxyInstance(ManualTransaction.class.getClassLoader(), + new Class[]{Connection.class, Managed.class}, (target, method, args) -> invokeConnection(method, + args)); + } + + /** + * Marks the connection of a transaction, its commits and roll backs belong to the whole import + */ + public interface Managed { + } + + /** + * Whether the connection is the connection of a transaction which is committed at the end of the import + */ + public static boolean isManaged(Connection connection) { + if (connection instanceof Managed) { + return true; + } + // Connections shared by a connection factory are unwrapped + try { + Connection unwrapped = connection.unwrap(Connection.class); + return unwrapped != connection && unwrapped instanceof Managed; + } catch (SQLException e) { + return false; + } + } + + /** + * Connection for importers, it is not closed by them + */ + public Connection getConnection() { + return proxy; + } + + /** + * Statements executed after the commit, they would commit the transaction (DDL after written rows) + */ + public List getDeferredStatements() { + synchronized (deferredStatements) { + return new ArrayList<>(deferredStatements); + } + } + + /** + * Whether the transaction was rolled back by an importer, e.g. after an error + */ + public boolean isRolledBack() { + return rolledBack; + } + + public boolean isTransactionalDdl() { + return transactionalDdl; + } + + public boolean isFinished() { + return finished; + } + + /** + * Commits the transaction, then executes the deferred statements one by one + * + * @return errors of deferred statements, empty if all of them are executed + */ + public List commit() throws SQLException { + checkOpen(); + connection.commit(); + finished = true; + List errors = new ArrayList<>(); + for (String sql : getDeferredStatements()) { + try (Statement statement = connection.createStatement()) { + statement.execute(sql); + connection.commit(); + } catch (SQLException e) { + errors.add(sql + ": " + e.getMessage()); + try { + connection.rollback(); + } catch (SQLException ignored) { + // The next statement runs in a new transaction + } + } + } + if (log.isInfoEnabled()) { + log.info("Transaction committed, {} deferred statements, {} failed", deferredStatements.size(), + errors.size()); + } + return errors; + } + + public void rollback() throws SQLException { + checkOpen(); + connection.rollback(); + finished = true; + deferredStatements.clear(); + if (log.isInfoEnabled()) { + log.info("Transaction rolled back"); + } + } + + private void checkOpen() { + if (finished) { + throw new IllegalStateException("The transaction is committed or rolled back already"); + } + } + + /** + * Rolls back an open transaction and closes the connection + */ + @Override + public void close() { + try { + if (!finished && !connection.isClosed()) { + connection.rollback(); + finished = true; + } + } catch (SQLException e) { + if (log.isWarnEnabled()) { + log.warn("Unable to roll back the transaction: {}", e.getMessage()); + } + } + JdbcUtils.closeQuietly(connection); + } + + private Object invokeConnection(Method method, Object[] args) throws Throwable { + switch (method.getName()) { + case "commit": + case "close": + return null; + case "setAutoCommit": + // Auto commit stays off until the user decides + return null; + case "getAutoCommit": + return false; + case "rollback": + if (args == null || args.length == 0) { + rolledBack = true; + } + break; + case "unwrap": + if (args != null && args.length == 1 && args[0] == Connection.class) { + return proxy; + } + break; + case "prepareStatement": + if (args != null && args.length > 0 && args[0] instanceof String && !isDdl((String) args[0]) + && isDml((String) args[0])) { + rowsWritten = true; + } + break; + default: + break; + } + Object result = invoke(connection, method, args); + if (result instanceof Statement && "createStatement".equals(method.getName())) { + Statement statement = (Statement) result; + return Proxy.newProxyInstance(Statement.class.getClassLoader(), new Class[]{Statement.class}, + (target, statementMethod, statementArgs) -> invokeStatement(statement, statementMethod, + statementArgs)); + } + return result; + } + + /** + * DDL after written rows is deferred if it would commit them, a deferred statement reports no update + */ + private Object invokeStatement(Statement statement, Method method, Object[] args) throws Throwable { + if ("getConnection".equals(method.getName())) { + return proxy; + } + if (args != null && args.length > 0 && args[0] instanceof String) { + String sql = ((String) args[0]).trim(); + switch (method.getName()) { + case "execute": + case "executeUpdate": + case "executeLargeUpdate": + case "addBatch": + if (isDdl(sql)) { + if (shouldDefer()) { + deferredStatements.add(sql); + if (log.isInfoEnabled()) { + log.info("Deferred until the commit: {}", sql); + } + return defaultResult(method.getReturnType()); + } + } else if (isDml(sql)) { + rowsWritten = true; + } + break; + default: + break; + } + } + return invoke(statement, method, args); + } + + private boolean shouldDefer() { + return !transactionalDdl && rowsWritten; + } + + static boolean isDdl(String sql) { + String text = sql.trim(); + return DDL.matcher(text).matches() || PLSQL_DDL.matcher(text).matches(); + } + + static boolean isDml(String sql) { + String text = sql.trim(); + return DML.matcher(text).matches() || PLSQL_DML.matcher(text).matches(); + } + + private static Object defaultResult(Class type) { + if (type == boolean.class) { + return false; + } else if (type == int.class) { + return 0; + } else if (type == long.class) { + return 0L; + } + return null; + } + + private static Object invoke(Object target, Method method, Object[] args) throws Throwable { + try { + return method.invoke(target, args); + } catch (InvocationTargetException e) { + throw e.getCause(); + } + } + + @Override + public String toString() { + return String.format(Locale.ENGLISH, "ManualTransaction[rowsWritten=%s, deferred=%d, rolledBack=%s]", + rowsWritten, deferredStatements.size(), rolledBack); + } +} diff --git a/src/main/resources/com/github/sqljam/face/app.css b/src/main/resources/com/github/sqljam/face/app.css index 7665f80..8c03338 100644 --- a/src/main/resources/com/github/sqljam/face/app.css +++ b/src/main/resources/com/github/sqljam/face/app.css @@ -39,6 +39,10 @@ -fx-text-fill: -color-success-fg; } +.status-pending { + -fx-text-fill: -color-warning-fg; +} + .form-status { -fx-font-size: 12px; } diff --git a/src/main/resources/com/github/sqljam/face/messages.properties b/src/main/resources/com/github/sqljam/face/messages.properties index c4788ec..df713a8 100644 --- a/src/main/resources/com/github/sqljam/face/messages.properties +++ b/src/main/resources/com/github/sqljam/face/messages.properties @@ -19,7 +19,7 @@ action.refresh=Refresh action.export=Export... action.import=Import... action.importScripts=Import Export Package... -action.importParquet=Import Parquet Files... +action.importParquet=Import Parquet or CSV Files... action.exit=Exit action.newTargetConnection=New data source as the target action.about=About SqlJam @@ -214,13 +214,22 @@ import.table.rows=Rows import.table.lobs=LOB files import.table.dataFiles=Data files import.table.empty=No tables -import.stopOnError=Stop on first error +import.transaction=Transaction +import.transaction.one=One transaction, roll back on error +import.transaction.one.tooltip=All rows are committed together when the import succeeds, an error rolls back everything +import.transaction.every=Commit every +import.transaction.rows=rows +import.transaction.confirm=Confirm before commit: review, then commit or roll back +import.transaction.confirm.tooltip=A successful import waits for you: commit keeps the rows, roll back undoes the whole import +import.transaction.skipErrors=Skip rows with errors and go on +import.transaction.skipErrors.tooltip=Rows with errors are logged and skipped, the others are written. Off: an error rolls back one transaction, or stops the commits every N rows +import.transaction.unsupported={0} has no transactions, rows are committed as they are written import.schema.prompt=Default schema of the data source import.error.directory=Please choose a directory exported by SqlJam import.error.manifest=Unable to read manifest: {0} import.warning.status=The package is {0}, it can not be imported import.warning.noDataSource=The package is exported for {0}, please create a {0} data source first -import.warning.parquetFiles=The directory contains Parquet files without manifest.json, please use Import Parquet Files +import.warning.parquetFiles=The directory contains Parquet or CSV files without manifest.json, please use Import Parquet or CSV Files import.summary=Executed {0} statements, restored {1} LOB values, {2} failed progress.exportTitle=Exporting SQL scripts @@ -240,6 +249,32 @@ progress.tableDone=Table {0}: {1} rows progress.elapsed=Elapsed {0} progress.openFolder=Open Folder progress.manifest=Manifest: {0} +progress.commit=Commit +progress.rollback=Roll Back +progress.pending=Waiting for commit +progress.pending.message=Rows are written in one transaction, nothing is visible to others yet. Commit keeps them, roll back undoes the import. +progress.pending.ddl=This database commits DDL at once: tables created by the import stay after a roll back, their rows are removed. +progress.pending.deferred={0} statements (foreign keys, identity values) run after the commit. +progress.decide.title=Commit the Import? +progress.decide.message=The import of "{0}" is complete and not committed yet. Commit keeps the rows, roll back undoes the import. +progress.decide.close=The import is not committed. Roll it back and close the window? +progress.decide.later=Decide Later +progress.committing=Committing... +progress.rollingBack=Rolling back... +progress.committed=Committed +progress.committedWithErrors=Committed, {0} statements after the commit failed +progress.committedSkipped=Committed with {0} errors skipped +progress.rolledBack=Rolled back +progress.rolledBackErrors=Rolled back because of errors +progress.errorsOnly=Errors only +progress.saveLog=Save Log... +progress.logFiles=Log files +progress.logSaved=Log saved to {0} +progress.statusLine=Status: {0}, errors and skipped items: {1} +progress.logDropped=... {0} more entries are not kept +progress.reminder.title=Import Still Waiting for Commit +progress.reminder.message=The import has waited {0} minutes. Its transaction holds locks, undo and log space of the target database, commit or roll back soon. +progress.reminder.log=Waiting for commit for {0} minutes, the transaction still holds locks, undo and log space dialog.error.details=Details: @@ -257,6 +292,12 @@ splash.ready=Ready export.format=Data format export.format.sql=SQL scripts export.format.parquet=Parquet +export.format.csv=CSV +export.format.delimiter=CSV delimiter +export.format.delimiter.comma=Comma ( , ) +export.format.delimiter.semicolon=Semicolon ( ; ) +export.format.delimiter.tab=Tab +export.format.delimiter.pipe=Pipe ( | ) export.format.compression=Compression # Export package viewer @@ -273,21 +314,28 @@ package.truncated=... the first {0} of {1} are shown package.column.name=Name package.column.size=Size package.column.modified=Modified -parquet.title=Import Parquet Files -parquet.header=Load Parquet files into a table of a database -parquet.section.files=Parquet files +parquet.title=Import Parquet or CSV Files +parquet.header=Load Parquet or CSV files into a table of a database +parquet.section.files=Data files parquet.files.add=Add Files... parquet.files.remove=Remove -parquet.files.filter=Parquet files -parquet.files.empty=No files, add Parquet files of one table -parquet.files.hint=Files of one table are loaded together, columns and the first rows are previewed +parquet.files.filter=Parquet and CSV files +parquet.files.filter.parquet=Parquet files +parquet.files.filter.csv=CSV files +parquet.files.empty=No files, add Parquet or CSV files of one table +parquet.files.hint=Files of one table are loaded together in this order, columns and the first rows are previewed. Delimiters and column types of CSV files are detected parquet.preview={0} columns, {1} rows, the first {2} are shown parquet.table=Table parquet.table.prompt=Name of the target table parquet.mode=Mode parquet.mode.create=Create or replace the table parquet.mode.append=Append rows to the existing table -parquet.error.read=Unable to read Parquet files: {0} +parquet.mode.upsert=Update or insert rows by key +parquet.mode.skip=Insert rows, skip existing ones by key +parquet.keys.prompt=Key columns, the primary key of the table if empty +parquet.keys.tooltip=Columns of a unique index or the primary key of the target table, separated by commas: MySQL and MariaDB match rows by their unique indexes, PostgreSQL, SQLite and DuckDB require such an index. +parquet.error.read=Unable to read the files: {0} +parquet.error.mixed=Files of one table must be all Parquet or all CSV parquet.error.table=Please enter the name of the target table parquet.summary=Loaded {0} rows into {1} data.query.allColumns=All columns @@ -308,3 +356,30 @@ example.button.tooltip=Create the example database of a small online shop again example.restore.title=Restore Example Database example.restore.message=The example database "{0}" is created again, changes made in it are lost. example.restore.error=Unable to create the example database +keys.title=Key Columns +keys.header=Columns which identify a row of each table +keys.hint=Rows are matched by these columns: existing rows are updated or skipped, other rows are inserted. Leave a field empty for the primary key, separate columns by commas. +keys.uniqueHint=Choose the columns of a unique index or the primary key of the target table: MySQL and MariaDB match rows by their unique indexes, PostgreSQL, SQLite and DuckDB require such an index. +keys.loading=Loading... +keys.primaryKey=Primary key: {0} +keys.none=No primary key, rows are inserted +keys.error.column={0} is not a column of {1} +export.writeMode=Rows +export.writeMode.insert=Insert +export.writeMode.upsert=Update or insert by key +export.writeMode.skip=Skip existing by key +export.writeMode.keys=Key Columns... +export.writeMode.unsupported={0} can not match rows by key +csv.options=CSV options +csv.delimiter=Delimiter +csv.delimiter.detect=Detect +csv.quote=Quote +csv.quote.double=Double quote ( " ) +csv.quote.single=Single quote ( ' ) +csv.null=NULL as +csv.null.prompt=Empty field +csv.header=Header row +csv.encoding=Encoding +csv.dateFormat=Dates +csv.dateFormat.iso=ISO (2024-01-31), exact +csv.dateFormat.detect=Detect diff --git a/src/main/resources/sqljam.properties b/src/main/resources/sqljam.properties index bb2055f..be5c808 100644 --- a/src/main/resources/sqljam.properties +++ b/src/main/resources/sqljam.properties @@ -31,8 +31,14 @@ sqljam.ui.data.page-size=200 sqljam.export.page-size=5000 # Rows read per page of a table with LOB columns, which keeps memory low sqljam.export.lob-page-size=100 -# Max bytes of a data file before it is split (orders.sql, orders_2.sql ...), 10 MB +# Max bytes of a data file of sql scripts and CSV files before it is split (orders.sql, orders_2.sql ..., +# orders.csv, orders_2.csv ...), 10 MB. Parts are imported in order sqljam.export.max-file-size=10485760 +# Max bytes of a Parquet file before the table is split (orders.parquet, orders_2.parquet ...), 256 MB. +# Parquet is compressed and columnar, large files read faster: 100 MB to 1 GB per file is recommended +sqljam.export.parquet-max-file-size=268435456 +# Delimiter of CSV files: , ; | or \t for tab +sqljam.export.csv-delimiter=, # SINGLE_FILE (data.sql) or FILE_PER_TABLE (data/
.sql) sqljam.export.data-file-strategy=SINGLE_FILE # Write LOB values as files in lob/ instead of inline sql literals @@ -40,6 +46,8 @@ sqljam.export.lob-separated=true # INSERT statements per batch when importing an export package (Oracle: 100 statements per PL/SQL block). # Measured: 2 to 10 times faster than statement by statement, 1 disables batches sqljam.import.batch-size=1000 +# An import waiting for commit reminds you after these minutes, its transaction holds locks and log space. 0 off +sqljam.import.confirm-commit.reminder-minutes=10 sqljam.export.last-directory= sqljam.import.last-directory= diff --git a/src/test/java/com/github/sqljam/face/service/TransferServiceTest.java b/src/test/java/com/github/sqljam/face/service/TransferServiceTest.java new file mode 100644 index 0000000..524a239 --- /dev/null +++ b/src/test/java/com/github/sqljam/face/service/TransferServiceTest.java @@ -0,0 +1,176 @@ +/* + * Copyright 2023-2026 Fred Feng + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.github.sqljam.face.service; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.io.File; +import java.sql.Connection; +import java.sql.DriverManager; +import java.sql.ResultSet; +import java.sql.Statement; +import java.util.List; + +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; +import com.github.sqljam.face.model.ConnectionProfile; +import com.github.sqljam.face.model.TransferRequest; +import com.github.sqljam.face.view.UiDatabase; +import com.github.sqljam.impexp.CsvImporter; +import com.github.sqljam.impexp.DataFileImporter; +import com.github.sqljam.impexp.DataFormat; +import com.github.sqljam.impexp.DbType; +import com.github.sqljam.impexp.ExportListener; +import com.github.sqljam.impexp.ExportManifest; +import com.github.sqljam.jdbc.ManualTransaction; + +/** + * @Description: TransferServiceTest verifies CSV export packages and imports in manual transactions of the user + * interface: direct imports, package imports and file imports are rolled back or committed, databases + * without transactions are refused + * @Author: Fred Feng + * @Date: 26/03/2023 + * @Version 1.0.0 + */ +class TransferServiceTest { + + @TempDir + File dir; + + private final TransferService service = new TransferService(); + + private ConnectionProfile target(String name) { + ConnectionProfile profile = new ConnectionProfile(); + profile.setName(name); + profile.setDbType(DbType.H2); + profile.setDatabase(new File(dir, name).getAbsolutePath()); + profile.setUsername("sa"); + profile.setPassword(""); + return profile; + } + + private static long count(ConnectionProfile profile, String table) throws Exception { + try (Connection connection = DriverManager.getConnection(profile.getJdbcUrl(), "sa", ""); + Statement statement = connection.createStatement(); + ResultSet rs = statement.executeQuery("SELECT COUNT(*) FROM " + table)) { + rs.next(); + return rs.getLong(1); + } + } + + private File exportCsv(ConnectionProfile source) throws Exception { + File packageDir = new File(dir, "csv"); + TransferRequest request = new TransferRequest(); + request.setSource(source); + request.setSourceCatalog(UiDatabase.CATALOG); + request.setSourceSchema(UiDatabase.SCHEMA); + request.setTarget(TransferRequest.Target.SCRIPT); + request.setOutputDirectory(packageDir); + request.setScriptDbType(DbType.H2); + request.setDataFormat(DataFormat.CSV); + request.setCsvDelimiter("|"); + request.setMaxFileSize(4 * 1024); + service.transfer(request, ExportListener.NONE); + return packageDir; + } + + @Test + void exportsCsvPackage() throws Exception { + File packageDir = exportCsv(UiDatabase.create(dir, "source")); + ExportManifest manifest = ExportManifest.read(packageDir); + assertEquals(DataFormat.CSV, manifest.getDataFormat()); + assertEquals("|", manifest.getOptions().get("delimiter")); + ExportManifest.TableEntry paged = manifest.getTables().stream().filter(table -> "T_PAGED".equals(table + .getName())).findFirst().orElseThrow(); + assertTrue(paged.getDataFiles().size() > 1, "Parts of T_PAGED: " + paged.getDataFiles()); + } + + @Test + void refusesDatabasesWithoutTransactions() { + ConnectionProfile clickHouse = new ConnectionProfile(); + clickHouse.setDbType(DbType.CLICKHOUSE); + assertThrows(IllegalArgumentException.class, () -> service.beginTransaction(clickHouse, null, null)); + } + + @Test + void importsDirectlyInTransaction() throws Exception { + ConnectionProfile source = UiDatabase.create(dir, "source"); + ConnectionProfile target = target("direct"); + TransferRequest request = new TransferRequest(); + request.setSource(source); + request.setSourceCatalog(UiDatabase.CATALOG); + request.setSourceSchema(UiDatabase.SCHEMA); + request.setTarget(TransferRequest.Target.DATABASE); + request.setTargetProfile(target); + request.setOneTransaction(true); + request.setFailFast(false); + try (ManualTransaction transaction = service.beginTransaction(target, null, null)) { + service.transfer(request, transaction, ExportListener.NONE); + assertFalse(transaction.isRolledBack()); + transaction.rollback(); + } + assertEquals(0, count(target, "T_PAGED")); + try (ManualTransaction transaction = service.beginTransaction(target, null, null)) { + service.transfer(request, transaction, ExportListener.NONE); + assertEquals(List.of(), transaction.commit()); + } + assertEquals(UiDatabase.PAGED_ROWS, count(target, "T_PAGED")); + assertEquals(5, count(target, "T_ONE")); + } + + @Test + void importsPackageAndFilesInTransaction() throws Exception { + File packageDir = exportCsv(UiDatabase.create(dir, "source")); + ConnectionProfile target = target("package"); + try (ManualTransaction transaction = service.beginTransaction(target, null, null)) { + assertTrue(service.importScripts(target, null, null, packageDir, true, 0, transaction, + ExportListener.NONE) instanceof CsvImporter); + transaction.rollback(); + } + assertEquals(0, count(target, "T_PAGED")); + try (ManualTransaction transaction = service.beginTransaction(target, null, null)) { + service.importScripts(target, null, null, packageDir, true, 0, transaction, ExportListener.NONE); + transaction.commit(); + } + assertEquals(UiDatabase.PAGED_ROWS, count(target, "T_PAGED")); + // Commits every 100 rows typed in the dialog: pages of 100 rows + ConnectionProfile paged = target("package100"); + // Rows of T_ONE and T_PAGED + assertEquals(5 + UiDatabase.PAGED_ROWS, ((CsvImporter) service.importScripts(paged, null, null, packageDir, + true, 100, null, ExportListener.NONE)).getImportedRows()); + assertEquals(UiDatabase.PAGED_ROWS, count(paged, "T_PAGED")); + + // CSV files of the package as files of other tools, the parts in order + ExportManifest manifest = ExportManifest.read(packageDir); + List files = manifest.getTables().stream().filter(table -> "T_PAGED".equals(table.getName())) + .findFirst().orElseThrow().getDataFiles().stream().map(path -> new File(packageDir, path)).toList(); + try (ManualTransaction transaction = service.beginTransaction(target, null, null)) { + DataFileImporter importer = service.importFiles(target, null, null, files, "PAGED_COPY", + DataFileImporter.Mode.CREATE, transaction, ExportListener.NONE); + assertTrue(importer instanceof CsvImporter); + assertEquals(UiDatabase.PAGED_ROWS, importer.getImportedRows()); + transaction.commit(); + } + assertEquals(UiDatabase.PAGED_ROWS, count(target, "PAGED_COPY")); + // Without a transaction the rows are committed page by page + assertEquals(UiDatabase.PAGED_ROWS, service.importParquet(target, null, null, files, "PAGED_AUTO", + DataFileImporter.Mode.CREATE, ExportListener.NONE).getImportedRows()); + assertEquals(UiDatabase.PAGED_ROWS, count(target, "PAGED_AUTO")); + } +} diff --git a/src/test/java/com/github/sqljam/face/view/ExportDialogTest.java b/src/test/java/com/github/sqljam/face/view/ExportDialogTest.java index 2b5cf7d..0f018a0 100644 --- a/src/test/java/com/github/sqljam/face/view/ExportDialogTest.java +++ b/src/test/java/com/github/sqljam/face/view/ExportDialogTest.java @@ -27,6 +27,8 @@ import java.sql.ResultSet; import java.sql.Statement; import java.util.List; +import java.util.Map; +import java.util.stream.Collectors; import org.junit.jupiter.api.AfterEach; import org.junit.jupiter.api.Assertions; @@ -39,6 +41,7 @@ import com.github.sqljam.face.model.ConnectionProfile; import com.github.sqljam.face.model.TableInfo; import com.github.sqljam.face.model.TransferRequest; +import com.github.sqljam.face.service.DatabaseSession; import com.github.sqljam.impexp.DataFileStrategy; import com.github.sqljam.impexp.DataFormat; import com.github.sqljam.impexp.DbType; @@ -47,12 +50,16 @@ import com.github.sqljam.impexp.IdentifierCase; import com.github.sqljam.impexp.ImpExpException; import com.github.sqljam.impexp.TableQuery; +import com.github.sqljam.impexp.WriteMode; +import javafx.beans.property.ObjectProperty; +import javafx.beans.property.SimpleObjectProperty; import javafx.scene.Node; import javafx.scene.Scene; import javafx.scene.control.Button; import javafx.scene.control.ButtonType; import javafx.scene.control.CheckBox; import javafx.scene.control.ComboBox; +import javafx.scene.control.DialogPane; import javafx.scene.control.Label; import javafx.scene.control.ListView; import javafx.scene.control.RadioButton; @@ -421,13 +428,13 @@ void mapsParquetFormat() { ((ComboBox) node("compressionCombo")).setValue("SNAPPY"); }); assertFalse(FxTestSupport.call(() -> node("compressionCombo").isDisabled())); - // Options of sql data files are disabled - for (String id : new String[]{"singleFileRadio", "perTableRadio", "maxFileSizeSpinner", - "lobSeparatedCheck"}) { + // Options of sql data files are disabled, large tables are split into Parquet parts + for (String id : new String[]{"singleFileRadio", "perTableRadio", "lobSeparatedCheck", "csvDelimiterCombo"}) { assertTrue(FxTestSupport.call(() -> node(id).isDisabled()), id); } + assertFalse(FxTestSupport.call(() -> node("maxFileSizeSpinner").isDisabled())); String hint = FxTestSupport.call(() -> ((Label) node("layoutHint")).getText()); - assertEquals("schema.sql\ndata/
.parquet\nconstraints.sql\nmanifest.json", hint); + assertEquals("schema.sql\ndata/
.parquet,
_2.parquet ...\nconstraints.sql\nmanifest.json", hint); assertEquals(List.of("ZSTD", "SNAPPY", "GZIP", "UNCOMPRESSED"), FxTestSupport.call(() -> List.copyOf(((ComboBox) node("compressionCombo")).getItems()))); assertFalse(startDisabled(), error()); @@ -440,6 +447,224 @@ void mapsParquetFormat() { assertEquals(DataFormat.SQL, build().getDataFormat()); } + @Test + @SuppressWarnings("unchecked") + void mapsCsvFormat() { + open(schemaNode()); + setText("directoryField", dir.getAbsolutePath()); + assertTrue(FxTestSupport.call(() -> node("csvDelimiterCombo").isDisabled())); + FxTestSupport.run(() -> ((RadioButton) node("csvFormatRadio")).setSelected(true)); + ComboBox delimiter = node("csvDelimiterCombo"); + assertFalse(FxTestSupport.call(delimiter::isDisabled)); + for (String id : new String[]{"singleFileRadio", "perTableRadio", "lobSeparatedCheck", "compressionCombo"}) { + assertTrue(FxTestSupport.call(() -> node(id).isDisabled()), id); + } + assertFalse(FxTestSupport.call(() -> node("maxFileSizeSpinner").isDisabled())); + assertEquals("schema.sql\ndata/
.csv,
_2.csv ...\nconstraints.sql\nmanifest.json", + FxTestSupport.call(() -> ((Label) node("layoutHint")).getText())); + // Delimiters by name, a tab has no visible character + assertEquals(",", FxTestSupport.call(delimiter::getValue)); + assertEquals(List.of("Comma ( , )", "Semicolon ( ; )", "Tab", "Pipe ( | )"), FxTestSupport.call(() -> + delimiter.getItems().stream().map(item -> delimiter.getConverter().toString(item)) + .collect(Collectors.toList()))); + assertEquals("", ExportDialog.delimiterName(null)); + assertEquals("#", ExportDialog.delimiterName("#")); + FxTestSupport.run(() -> delimiter.setValue("\t")); + assertFalse(startDisabled(), error()); + TransferRequest request = build(); + assertEquals(DataFormat.CSV, request.getDataFormat()); + assertEquals("\t", request.getCsvDelimiter()); + assertEquals(10L * 1024 * 1024, request.getMaxFileSize()); + // The delimiter is the default of the next export + assertEquals("\t", context.getSettings().getCsvDelimiter()); + ExportDialog next = open(schemaNode()); + assertEquals("\t", FxTestSupport.call(() -> ((ComboBox) node("csvDelimiterCombo")).getValue())); + FxTestSupport.run(next::close); + } + + /** + * Parquet files are split at their own size, sql scripts and CSV files share one + */ + @Test + void keepsMaxFileSizeOfEachFormat() { + open(schemaNode()); + setText("directoryField", dir.getAbsolutePath()); + Spinner size = node("maxFileSizeSpinner"); + assertEquals(10, FxTestSupport.call(size::getValue).intValue()); + setSpinnerText("maxFileSizeSpinner", "20"); + FxTestSupport.run(() -> ((RadioButton) node("parquetFormatRadio")).setSelected(true)); + assertEquals(256, FxTestSupport.call(size::getValue).intValue()); + setSpinnerText("maxFileSizeSpinner", "512"); + FxTestSupport.run(() -> ((RadioButton) node("csvFormatRadio")).setSelected(true)); + assertEquals(20, FxTestSupport.call(size::getValue).intValue()); + FxTestSupport.run(() -> ((RadioButton) node("sqlFormatRadio")).setSelected(true)); + assertEquals(20, FxTestSupport.call(size::getValue).intValue()); + FxTestSupport.run(() -> ((RadioButton) node("parquetFormatRadio")).setSelected(true)); + assertEquals(512, FxTestSupport.call(size::getValue).intValue()); + assertEquals(512L * 1024 * 1024, build().getMaxFileSize()); + assertEquals(512L * 1024 * 1024, context.getSettings().getParquetMaxFileSize()); + // Text files keep their size + assertEquals(10L * 1024 * 1024, context.getSettings().getMaxDataFileSize()); + } + + @SuppressWarnings("unchecked") + private void chooseTarget() { + FxTestSupport.run(() -> ((RadioButton) node("databaseRadio")).setSelected(true)); + ComboBox targetCombo = (ComboBox) node("targetCombo"); + FxTestSupport.run(() -> targetCombo.setValue(targetCombo.getItems().get(0))); + ComboBox targetSchema = (ComboBox) node("targetSchemaCombo"); + FxTestSupport.waitUntil(() -> !targetSchema.getItems().isEmpty()); + FxTestSupport.run(() -> targetSchema.getEditor().setText("COPY")); + } + + @Test + void importsInOneTransactionByDefault() { + open(schemaNode()); + chooseTarget(); + RadioButton one = node("oneTransactionRadio"); + RadioButton every = node("commitEveryRadio"); + CheckBox confirm = node("confirmCommitCheck"); + CheckBox skipErrors = node("skipErrorsCheck"); + CheckBox failFast = node("failFastCheck"); + assertTrue(FxTestSupport.call(one::isSelected)); + assertFalse(FxTestSupport.call(confirm::isSelected)); + assertFalse(FxTestSupport.call(confirm::isDisabled)); + // The first error rolls back everything, stop on first error of file exports is hidden + assertFalse(FxTestSupport.call(skipErrors::isSelected)); + assertFalse(FxTestSupport.call(failFast::isVisible)); + // Commits every N rows follow the page size, rows are committed page by page + Spinner commitRows = node("commitRowsSpinner"); + assertEquals(5000, FxTestSupport.call(commitRows::getValue)); + setSpinnerText("pageSizeSpinner", "1000"); + assertEquals("1000", FxTestSupport.call(() -> commitRows.getEditor().getText())); + // Text which is not a number keeps the last rows + setSpinnerText("pageSizeSpinner", "abc"); + assertEquals("1000", FxTestSupport.call(() -> commitRows.getEditor().getText())); + setSpinnerText("pageSizeSpinner", "1000"); + TransferRequest request = build(); + assertTrue(request.isOneTransaction()); + assertFalse(request.isConfirmCommit()); + assertTrue(request.isFailFast()); + FxTestSupport.run(() -> confirm.setSelected(true)); + assertTrue(build().isConfirmCommit()); + // Rows with errors may be skipped in one transaction + FxTestSupport.run(() -> skipErrors.setSelected(true)); + assertFalse(build().isFailFast()); + // Commits every N rows have no confirm, rows with errors may still be skipped + FxTestSupport.run(() -> every.setSelected(true)); + assertTrue(FxTestSupport.call(() -> confirm.isDisabled() && !confirm.isSelected())); + assertFalse(FxTestSupport.call(skipErrors::isDisabled)); + // Rows per commit typed in the option are the page size + FxTestSupport.run(() -> commitRows.getValueFactory().setValue(2000)); + assertEquals(2000, FxTestSupport.call(() -> ((Spinner) node("pageSizeSpinner")).getValue())); + FxTestSupport.run(() -> commitRows.getEditor().setText("300")); + request = build(); + assertEquals(300, request.getPageSize()); + assertFalse(request.isOneTransaction()); + assertFalse(request.isConfirmCommit()); + assertFalse(request.isFailFast()); + // Back to a script target: stop on first error is shown again + FxTestSupport.run(() -> ((RadioButton) node("scriptRadio")).setSelected(true)); + assertTrue(FxTestSupport.call(failFast::isVisible)); + assertTrue(build().isFailFast()); + } + + @Test + void offersNoTransactionWithoutTransactions() { + ObjectProperty target = new SimpleObjectProperty<>(); + TransactionOption option = FxTestSupport.call(() -> new TransactionOption(context, target, 5000)); + assertTrue(option.isOneTransaction()); + assertFalse(option.isSkipErrors()); + ConnectionProfile clickHouse = new ConnectionProfile(); + clickHouse.setDbType(DbType.CLICKHOUSE); + FxTestSupport.run(() -> target.set(clickHouse)); + assertFalse(option.isOneTransaction()); + assertFalse(option.isConfirmCommit()); + Label unsupported = (Label) FxTestSupport.call(() -> option.getNode().lookup("#transactionUnsupportedLabel")); + assertTrue(unsupported.isVisible()); + assertEquals("ClickHouse has no transactions, rows are committed as they are written", unsupported.getText()); + // The choice comes back with a database with transactions + ConnectionProfile postgresql = new ConnectionProfile(); + postgresql.setDbType(DbType.POSTGRESQL); + FxTestSupport.run(() -> target.set(postgresql)); + assertTrue(option.isOneTransaction()); + assertFalse(unsupported.isVisible()); + // Commits every N rows by the settings + context.getSettings().setOneTransaction(false); + TransactionOption every = FxTestSupport.call(() -> new TransactionOption(context, target, 100)); + assertFalse(every.isOneTransaction()); + } + + @Test + void upsertsKeptTablesByKey() { + open(schemaNode()); + chooseTarget(); + RadioButton insert = node("insertRadio"); + RadioButton upsert = node("upsertRadio"); + CheckBox recreate = node("recreateCheck"); + Button keys = node("keyColumnsButton"); + // Insert is the default + assertTrue(FxTestSupport.call(insert::isSelected)); + assertTrue(FxTestSupport.call(keys::isDisabled)); + assertEquals(WriteMode.INSERT, build().getWriteMode()); + // Upserts keep the tables + FxTestSupport.run(() -> upsert.setSelected(true)); + assertTrue(FxTestSupport.call(() -> !recreate.isSelected() && recreate.isDisabled())); + assertFalse(FxTestSupport.call(keys::isDisabled)); + // Key columns of the selected tables, an empty field is the primary key + FxTestSupport.fireLater(keys); + FxTestSupport.waitUntil(() -> FxTestSupport.getDialogPanes().size() == 2); + DialogPane pane = FxTestSupport.getDialogPanes().stream().filter(item -> item.lookup("#keyColumnsGrid") + != null).findFirst().orElseThrow(); + // Chosen columns should be a unique index of the target table + assertTrue(FxTestSupport.call(() -> ((Label) pane.lookup("#keyColumnsUniqueHint")).getText()).startsWith( + "Choose the columns of a unique index or the primary key of the target table")); + TextField one = FxTestSupport.call(() -> (TextField) pane.lookup("#keyColumns-T_ONE")); + FxTestSupport.waitUntil(() -> one.getPromptText().startsWith("Primary key")); + assertEquals("Primary key: ID", FxTestSupport.call(one::getPromptText)); + Button ok = FxTestSupport.call(() -> (Button) pane.lookupButton(ButtonType.OK)); + FxTestSupport.run(() -> one.setText("MISSING")); + assertEquals("MISSING is not a column of T_ONE", FxTestSupport.call(() -> ((Label) pane + .lookup("#keyColumnsError")).getText())); + assertTrue(FxTestSupport.call(ok::isDisabled)); + FxTestSupport.run(() -> one.setText(" name , id ")); + assertFalse(FxTestSupport.call(ok::isDisabled)); + FxTestSupport.fireLater(ok); + FxTestSupport.waitUntil(() -> FxTestSupport.getDialogPanes().size() == 1); + TransferRequest request = build(); + assertEquals(WriteMode.UPSERT, request.getWriteMode()); + assertFalse(request.isTableRecreated()); + assertEquals(Map.of("T_ONE", List.of("name", "id")), request.getKeyColumnNames()); + // Back to inserts: tables may be recreated again, keys are not used + FxTestSupport.run(() -> insert.setSelected(true)); + assertFalse(FxTestSupport.call(recreate::isDisabled)); + assertEquals(Map.of(), build().getKeyColumnNames()); + // Existing rows are skipped by the same keys + FxTestSupport.run(() -> ((RadioButton) node("skipExistingRadio")).setSelected(true)); + assertTrue(FxTestSupport.call(() -> !recreate.isSelected() && recreate.isDisabled())); + assertFalse(FxTestSupport.call(keys::isDisabled)); + request = build(); + assertEquals(WriteMode.SKIP_EXISTING, request.getWriteMode()); + assertEquals(Map.of("T_ONE", List.of("name", "id")), request.getKeyColumnNames()); + } + + @Test + void offersNoUpsertForClickHouse() { + open(schemaNode()); + @SuppressWarnings("unchecked") + ComboBox dbType = (ComboBox) node("scriptDbTypeCombo"); + RadioButton upsert = node("upsertRadio"); + RadioButton skip = node("skipExistingRadio"); + FxTestSupport.run(() -> skip.setSelected(true)); + FxTestSupport.run(() -> dbType.setValue(DbType.CLICKHOUSE)); + assertTrue(FxTestSupport.call(() -> upsert.isDisabled() && !upsert.isSelected())); + assertTrue(FxTestSupport.call(() -> skip.isDisabled() && !skip.isSelected())); + assertEquals("ClickHouse can not match rows by key", FxTestSupport.call(() -> upsert.getTooltip().getText())); + FxTestSupport.run(() -> dbType.setValue(DbType.POSTGRESQL)); + assertFalse(FxTestSupport.call(upsert::isDisabled)); + assertFalse(FxTestSupport.call(skip::isDisabled)); + } + @Test @SuppressWarnings("unchecked") void groupsScriptDatabaseTypesByCategory() { @@ -560,4 +785,33 @@ void exportsRowsOfQuery() { assertFalse(FxTestSupport.call(() -> node("queryLabel").isVisible())); assertTrue(build().getTableQueries().isEmpty()); } + + /** + * Key columns of several tables, a table without primary key is inserted unless columns are chosen + */ + @Test + void choosesKeyColumnsOfTables() throws Exception { + try (Connection connection = DriverManager.getConnection(profile.getJdbcUrl(), "sa", ""); + Statement statement = connection.createStatement()) { + statement.execute("CREATE TABLE IF NOT EXISTS T_NO_KEY (CODE VARCHAR(10), NAME VARCHAR(20))"); + } + DatabaseSession session = context.getSessionManager().getSession(profile); + KeyColumnsDialog keys = FxTestSupport.call(() -> new KeyColumnsDialog(stage, session, UiDatabase.CATALOG, + UiDatabase.SCHEMA, List.of("T_ONE", "T_NO_KEY"), Map.of("T_ONE", List.of("NAME")))); + FxTestSupport.run(keys::show); + TextField one = FxTestSupport.call(() -> (TextField) keys.getDialogPane().lookup("#keyColumns-T_ONE")); + TextField noKey = FxTestSupport.call(() -> (TextField) keys.getDialogPane().lookup("#keyColumns-T_NO_KEY")); + FxTestSupport.waitUntil(() -> noKey.getPromptText().startsWith("No")); + assertEquals("No primary key, rows are inserted", FxTestSupport.call(noKey::getPromptText)); + // The chosen columns of before + assertEquals("NAME", FxTestSupport.call(one::getText)); + FxTestSupport.run(() -> noKey.setText("code")); + assertEquals(Map.of("T_ONE", List.of("NAME"), "T_NO_KEY", List.of("code")), FxTestSupport.call( + keys::getKeyColumns)); + // An empty field is the primary key + FxTestSupport.run(() -> one.setText(" ")); + assertEquals(Map.of("T_NO_KEY", List.of("code")), FxTestSupport.call(keys::getKeyColumns)); + assertEquals(List.of(), KeyColumnsDialog.parse(null)); + FxTestSupport.run(keys::close); + } } diff --git a/src/test/java/com/github/sqljam/face/view/ImportPackageDialogTest.java b/src/test/java/com/github/sqljam/face/view/ImportPackageDialogTest.java index 19c8ff9..ea36fac 100644 --- a/src/test/java/com/github/sqljam/face/view/ImportPackageDialogTest.java +++ b/src/test/java/com/github/sqljam/face/view/ImportPackageDialogTest.java @@ -19,6 +19,8 @@ import static org.junit.jupiter.api.Assertions.assertFalse; import static org.junit.jupiter.api.Assertions.assertTrue; +import com.github.sqljam.impexp.CsvImporter; +import com.github.sqljam.jdbc.ManualTransaction; import java.io.File; import java.nio.file.Files; import java.nio.file.StandardOpenOption; @@ -26,6 +28,7 @@ import java.sql.DriverManager; import java.sql.ResultSet; import java.util.List; +import java.util.concurrent.atomic.AtomicReference; import java.util.stream.Collectors; import org.junit.jupiter.api.AfterEach; @@ -48,9 +51,13 @@ import javafx.scene.Scene; import javafx.scene.control.Button; import javafx.scene.control.ButtonType; +import javafx.scene.control.CheckBox; import javafx.scene.control.ComboBox; +import javafx.scene.control.DialogPane; import javafx.scene.control.Label; +import javafx.scene.control.RadioButton; import javafx.scene.control.TableView; +import javafx.scene.control.Spinner; import javafx.scene.control.TextField; import javafx.scene.layout.StackPane; import javafx.stage.Stage; @@ -270,6 +277,9 @@ void importsPackage() throws Exception { assertEquals("Completed", status(progress)); assertEquals("Overall 100%", FxTestSupport.call(() -> ((Label) progress.getStage().getScene().getRoot() .lookup("#percentLabel")).getText())); + // Every table of the package is completed + assertEquals("Tables: 3 / 3", FxTestSupport.call(() -> ((Label) progress.getStage().getScene().getRoot() + .lookup("#overallLabel")).getText())); try (Connection connection = DriverManager.getConnection(target.getJdbcUrl(), "sa", ""); ResultSet rs = connection.createStatement().executeQuery("SELECT COUNT(*) FROM T_PAGED")) { rs.next(); @@ -304,6 +314,109 @@ void showsDataFormatOfPackages() throws Exception { assertEquals("COMPLETED", text("statusValue")); assertEquals("Parquet", text("formatValue")); assertFalse(startDisabled()); + FxTestSupport.run(dialog::close); + open(exportPackage("csv", DataFormat.CSV)); + assertEquals("CSV", text("formatValue")); + assertFalse(startDisabled()); + } + + @Test + void importsInOneTransactionByDefault() throws Exception { + open(exportPackage("csv", DataFormat.CSV)); + RadioButton one = node("oneTransactionRadio"); + RadioButton every = node("commitEveryRadio"); + CheckBox confirm = node("confirmCommitCheck"); + CheckBox skipErrors = node("skipErrorsCheck"); + assertTrue(FxTestSupport.call(one::isSelected)); + assertFalse(FxTestSupport.call(confirm::isSelected)); + // The first error stops the import by default + assertFalse(FxTestSupport.call(skipErrors::isSelected)); + assertTrue(FxTestSupport.call(dialog::buildRequest).isStopOnError()); + // Scripts are committed batch by batch, the rows per commit are typed in the dialog + Spinner commitRows = node("commitRowsSpinner"); + assertEquals(1000, FxTestSupport.call(commitRows::getValue)); + assertTrue(FxTestSupport.call(commitRows::isDisabled)); + assertEquals("Insert", text("writeModeValue")); + ImportPackageDialog.ImportRequest request = FxTestSupport.call(dialog::buildRequest); + assertTrue(request.isOneTransaction()); + assertFalse(request.isConfirmCommit()); + FxTestSupport.run(() -> confirm.setSelected(true)); + assertTrue(FxTestSupport.call(dialog::buildRequest).isConfirmCommit()); + // One transaction has no rows per commit + assertEquals(0, request.getCommitRows()); + FxTestSupport.run(() -> every.setSelected(true)); + FxTestSupport.run(() -> skipErrors.setSelected(true)); + assertFalse(FxTestSupport.call(commitRows::isDisabled)); + FxTestSupport.run(() -> commitRows.getEditor().setText("200")); + request = FxTestSupport.call(dialog::buildRequest); + assertFalse(request.isStopOnError()); + assertEquals(200, request.getCommitRows()); + assertFalse(request.isOneTransaction()); + assertFalse(request.isConfirmCommit()); + } + + @Test + void showsWriteModeOfPackage() throws Exception { + File packageDir = exportPackage("csv", DataFormat.CSV); + ExportManifest manifest = ExportManifest.read(packageDir); + manifest.getOptions().put("writeMode", "UPSERT"); + manifest.write(packageDir); + open(packageDir); + assertEquals("Update or insert by key", text("writeModeValue")); + manifest.getOptions().put("writeMode", "SKIP_EXISTING"); + manifest.write(packageDir); + FxTestSupport.closeAllDialogs(); + open(packageDir); + assertEquals("Skip existing by key", text("writeModeValue")); + } + + /** + * The package is imported in a transaction like the main window does: rolled back, then committed + */ + private ProgressDialog importInTransaction(File packageDir, String decision) { + ProgressDialog progress = FxTestSupport.call(() -> new ProgressDialog(stage, context, "Import", 3)); + AtomicReference transaction = new AtomicReference<>(); + FxTestSupport.run(() -> progress.run(() -> { + transaction.set(context.getTransferService().beginTransaction(target, null, null)); + return context.getTransferService().importScripts(target, null, null, packageDir, true, 0, + transaction.get(), progress.getListener()); + }, (AbstractFileImporter importer) -> importer.getClass().getSimpleName(), transaction::get, null)); + FxTestSupport.waitUntil(() -> FxTestSupport.getDialogPanes().stream().anyMatch(pane -> "decisionPane" + .equals(pane.getId()))); + DialogPane pane = FxTestSupport.getDialogPanes().stream().filter(item -> "decisionPane".equals(item.getId())) + .findFirst().orElseThrow(); + FxTestSupport.fireLater(FxTestSupport.call(() -> pane.getButtonTypes().stream().map(type -> (Button) pane + .lookupButton(type)).filter(button -> decision.equals(button.getText())).findFirst().orElseThrow())); + FxTestSupport.waitUntil(progress::isFinished); + return progress; + } + + private long countPaged() throws Exception { + try (Connection connection = DriverManager.getConnection(target.getJdbcUrl(), "sa", ""); + ResultSet rs = connection.createStatement().executeQuery("SELECT COUNT(*) FROM T_PAGED")) { + rs.next(); + return rs.getLong(1); + } + } + + @Test + void importsCsvPackageInTransaction() throws Exception { + File packageDir = exportPackage("csv", DataFormat.CSV); + assertTrue(AbstractFileImporter.forDirectory(packageDir, null, DbType.H2) instanceof CsvImporter); + ProgressDialog rolledBack = importInTransaction(packageDir, "Roll Back"); + assertEquals("Rolled back", status(rolledBack)); + // H2 commits DDL at once: the table stays, its rows are rolled back + assertEquals(0, countPaged()); + ProgressDialog committed = importInTransaction(packageDir, "Commit"); + assertEquals("Committed", status(committed)); + try (Connection connection = DriverManager.getConnection(target.getJdbcUrl(), "sa", ""); + ResultSet rs = connection.createStatement().executeQuery( + "SELECT COUNT(*), COUNT(DATA), MAX(LENGTH(NOTE)) FROM T_PAGED")) { + rs.next(); + assertEquals(UiDatabase.PAGED_ROWS, rs.getInt(1)); + assertEquals(1, rs.getInt(2)); + assertEquals(UiDatabase.LONG_TEXT_LENGTH, rs.getInt(3)); + } } @Test @@ -312,7 +425,8 @@ void guidesParquetFilesWithoutManifest() throws Exception { assertTrue(new File(parquetDir, ExportManifest.FILE_NAME).delete()); open(parquetDir); assertTrue(startDisabled()); - assertEquals("The directory contains Parquet files without manifest.json, please use Import Parquet Files", + assertEquals("The directory contains Parquet or CSV files without manifest.json, please use Import Parquet or CSV" + + " Files", text("warningLabel")); assertTrue(ImportPackageDialog.isParquetDirectory(parquetDir)); assertFalse(ImportPackageDialog.isParquetDirectory(dir)); diff --git a/src/test/java/com/github/sqljam/face/view/ImportParquetDialogTest.java b/src/test/java/com/github/sqljam/face/view/ImportParquetDialogTest.java index 106f4bf..4e3ccd7 100644 --- a/src/test/java/com/github/sqljam/face/view/ImportParquetDialogTest.java +++ b/src/test/java/com/github/sqljam/face/view/ImportParquetDialogTest.java @@ -21,6 +21,7 @@ import static org.junit.jupiter.api.Assertions.assertTrue; import java.io.File; +import java.nio.charset.Charset; import java.nio.file.Files; import java.sql.Connection; import java.sql.DriverManager; @@ -29,6 +30,7 @@ import java.util.List; import java.util.stream.Collectors; +import javafx.scene.control.CheckBox; import org.junit.jupiter.api.AfterEach; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.extension.ExtendWith; @@ -37,6 +39,7 @@ import org.testfx.framework.junit5.Start; import com.github.sqljam.face.model.ConnectionProfile; import com.github.sqljam.impexp.DbType; +import com.github.sqljam.impexp.DataFileImporter; import com.github.sqljam.impexp.ParquetImporter; import javafx.scene.Node; import javafx.scene.Scene; @@ -45,6 +48,7 @@ import javafx.scene.control.Label; import javafx.scene.control.ListView; import javafx.scene.control.RadioButton; +import javafx.scene.control.Spinner; import javafx.scene.control.TableView; import javafx.scene.control.TextField; import javafx.scene.layout.StackPane; @@ -108,6 +112,17 @@ private File writeParquet(String name, int from, int to) throws Exception { } @SuppressWarnings("unchecked") + private File writeCsv(String name, int from, int to) throws Exception { + File file = new File(dir, name); + try (Connection connection = DriverManager.getConnection("jdbc:duckdb:"); + Statement statement = connection.createStatement()) { + statement.execute(String.format("COPY (SELECT range::INTEGER AS id, 'name ' || range AS name," + + " DATE '2024-01-01' + range::INTEGER AS created FROM range(%d, %d)) TO '%s' (FORMAT csv, HEADER," + + " DELIMITER ';')", from, to, file.getAbsolutePath().replace("'", "''"))); + } + return file; + } + private T node(String id) { return (T) dialog.getDialogPane().lookup("#" + id); } @@ -146,7 +161,8 @@ private TableView> preview() { void requiresFilesAndTable() throws Exception { open(null); assertTrue(startDisabled()); - assertEquals("Files of one table are loaded together, columns and the first rows are previewed", + assertEquals("Files of one table are loaded together in this order, columns and the first rows are previewed." + + " Delimiters and column types of CSV files are detected", text("parquetPreviewLabel")); // The only data source is the target assertEquals("Target H2", FxTestSupport.call(() -> { @@ -192,7 +208,7 @@ void reportsUnreadableFiles() throws Exception { File broken = new File(dir, "broken.parquet"); Files.writeString(broken.toPath(), "not parquet"); addFiles(broken); - FxTestSupport.waitUntil(() -> text("warningLabel").startsWith("Unable to read Parquet files")); + FxTestSupport.waitUntil(() -> text("warningLabel").startsWith("Unable to read the files")); assertTrue(startDisabled()); } @@ -205,6 +221,49 @@ void derivesTableNames() { assertEquals("订单", ImportParquetDialog.getTableName(new File("订单.parquet"))); } + @Test + void loadsCsvPartsInOrder() throws Exception { + open(DbNode.connection(target)); + File second = writeCsv("orders_2.csv", ROWS, ROWS + 30); + File first = writeCsv("orders.csv", 0, ROWS); + // Added in any order, imported in the order of the parts + addFiles(second, first); + waitForPreview(); + assertEquals(List.of(first, second), FxTestSupport.call(() -> List.copyOf(((ListView) node( + "parquetFiles")).getItems()))); + // The table is named after the first part + assertEquals("orders", FxTestSupport.call(() -> ((TextField) node("tableField")).getText())); + // Delimiter and types of CSV files are detected + assertEquals("3 columns, " + (ROWS + 30) + " rows, the first " + ImportParquetDialog.PREVIEW_ROWS + + " are shown", text("parquetPreviewLabel")); + assertEquals("name 0", FxTestSupport.call(() -> preview().getItems().get(0).get(1))); + ImportParquetDialog.ParquetRequest request = FxTestSupport.call(dialog::buildRequest); + assertTrue(request.isOneTransaction()); + assertTrue(FxTestSupport.call(() -> ((RadioButton) node("oneTransactionRadio")).isSelected())); + // Options of CSV files are shown for CSV files, the semicolon is detected + assertTrue(FxTestSupport.call(() -> node("csvOptionsBox").isVisible())); + assertEquals("", request.getCsvOptions().getDelimiter()); + ProgressDialog loaded = load(request.getFiles(), ParquetImporter.Mode.CREATE); + assertEquals("Completed", FxTestSupport.call(() -> ((Label) loaded.getStage().getScene().getRoot() + .lookup("#progressStatus")).getText())); + try (Connection connection = DriverManager.getConnection(target.getJdbcUrl(), "sa", ""); + ResultSet rs = connection.createStatement().executeQuery("SELECT MIN(ID), MAX(ID), COUNT(*) FROM ORDERS")) { + rs.next(); + assertEquals(0, rs.getInt(1)); + assertEquals(ROWS + 29, rs.getInt(2)); + assertEquals(ROWS + 30, rs.getInt(3)); + } + } + + @Test + void rejectsMixedFormats() throws Exception { + open(null); + addFiles(writeParquet("orders.parquet", 0, 10), writeCsv("orders.csv", 10, 20)); + assertEquals("Files of one table must be all Parquet or all CSV", text("warningLabel")); + assertTrue(startDisabled()); + assertTrue(FxTestSupport.call(() -> preview().getColumns().isEmpty())); + } + @Test void buildsRequest() throws Exception { open(DbNode.connection(target)); @@ -225,7 +284,7 @@ void buildsRequest() throws Exception { private ProgressDialog load(List files, ParquetImporter.Mode mode) { ProgressDialog progress = FxTestSupport.call(() -> new ProgressDialog(stage, context, "Import", 1)); FxTestSupport.run(() -> progress.run(() -> context.getTransferService().importParquet(target, null, null, - files, "ORDERS", mode, progress.getListener()), (ParquetImporter importer) -> String.valueOf( + files, "ORDERS", mode, progress.getListener()), (DataFileImporter importer) -> String.valueOf( importer.getImportedRows()), null)); FxTestSupport.waitUntil(progress::isFinished); return progress; @@ -263,4 +322,74 @@ void loadsFilesIntoDatabase() throws Exception { assertEquals(ROWS, countRows()); FxTestSupport.run(() -> replaced.getStage().close()); } + + @Test + @SuppressWarnings("unchecked") + void upsertsRowsByKey() throws Exception { + open(DbNode.connection(target)); + addFiles(writeParquet("orders.parquet", 0, 10)); + TextField keys = node("keyColumnsField"); + assertTrue(FxTestSupport.call(keys::isDisabled)); + FxTestSupport.run(() -> ((RadioButton) node("upsertRadio")).setSelected(true)); + assertFalse(FxTestSupport.call(keys::isDisabled)); + assertEquals("Key columns, the primary key of the table if empty", FxTestSupport.call(keys::getPromptText)); + assertTrue(FxTestSupport.call(() -> keys.getTooltip().getText()).startsWith("Columns of a unique index or" + + " the primary key of the target table")); + ImportParquetDialog.ParquetRequest request = FxTestSupport.call(dialog::buildRequest); + assertEquals(ParquetImporter.Mode.UPSERT, request.getMode()); + assertEquals(List.of(), request.getKeyColumns()); + // Parquet files have no CSV options + assertNull(request.getCsvOptions()); + FxTestSupport.run(() -> keys.setText(" id ,name")); + assertEquals(List.of("id", "name"), FxTestSupport.call(dialog::buildRequest).getKeyColumns()); + // Existing rows are skipped by the same keys + RadioButton skip = node("skipExistingRadio"); + FxTestSupport.run(() -> skip.setSelected(true)); + assertFalse(FxTestSupport.call(keys::isDisabled)); + request = FxTestSupport.call(dialog::buildRequest); + assertEquals(ParquetImporter.Mode.SKIP_EXISTING, request.getMode()); + assertEquals(List.of("id", "name"), request.getKeyColumns()); + assertFalse(request.isSkipErrors()); + assertEquals(0, request.getCommitRows()); + FxTestSupport.run(() -> ((CheckBox) node("skipErrorsCheck")).setSelected(true)); + assertTrue(FxTestSupport.call(dialog::buildRequest).isSkipErrors()); + // Rows per commit typed in the dialog, the page size of the settings by default + FxTestSupport.run(() -> ((RadioButton) node("commitEveryRadio")).setSelected(true)); + assertEquals(context.getSettings().getTransferPageSize(), FxTestSupport.call(dialog::buildRequest) + .getCommitRows()); + FxTestSupport.run(() -> ((Spinner) node("commitRowsSpinner")).getEditor().setText("50")); + assertEquals(50, FxTestSupport.call(dialog::buildRequest).getCommitRows()); + // ClickHouse can not match rows by key + ConnectionProfile clickHouse = new ConnectionProfile(); + clickHouse.setName("ClickHouse"); + clickHouse.setDbType(DbType.CLICKHOUSE); + ComboBox targets = node("targetCombo"); + RadioButton upsert = node("upsertRadio"); + FxTestSupport.run(() -> { + targets.getItems().add(clickHouse); + targets.setValue(clickHouse); + }); + FxTestSupport.closeAllDialogs(); + assertTrue(FxTestSupport.call(() -> upsert.isDisabled() && !upsert.isSelected())); + assertTrue(FxTestSupport.call(() -> skip.isDisabled() && !skip.isSelected())); + assertTrue(FxTestSupport.call(() -> ((RadioButton) node("appendRadio")).isSelected())); + } + + @Test + void previewsCsvWithOptions() throws Exception { + open(null); + File file = new File(dir, "sales.txt"); + Files.write(file.toPath(), "1;北京;10.5\n2;上海;7\n".getBytes(Charset.forName("GBK"))); + addFiles(file); + assertTrue(FxTestSupport.call(() -> node("csvOptionsBox").isVisible())); + // Detected as UTF-8 the GBK bytes are not readable, the encoding and the header are chosen + FxTestSupport.waitUntil(() -> !text("warningLabel").isEmpty() || !preview().getItems().isEmpty()); + FxTestSupport.run(() -> { + ((ComboBox) node("csvEncodingCombo")).setValue("GBK"); + ((CheckBox) node("csvHeaderCheck")).setSelected(false); + }); + FxTestSupport.waitUntil(() -> preview().getItems().size() == 2); + assertEquals("北京", FxTestSupport.call(() -> preview().getItems().get(0).get(1))); + assertEquals("GBK", FxTestSupport.call(dialog::buildRequest).getCsvOptions().getEncoding()); + } } diff --git a/src/test/java/com/github/sqljam/face/view/MainViewTest.java b/src/test/java/com/github/sqljam/face/view/MainViewTest.java index 1a531f6..7c6c6a9 100644 --- a/src/test/java/com/github/sqljam/face/view/MainViewTest.java +++ b/src/test/java/com/github/sqljam/face/view/MainViewTest.java @@ -43,6 +43,7 @@ import javafx.scene.Scene; import javafx.scene.control.Button; import javafx.scene.control.ButtonType; +import javafx.scene.control.CheckBox; import javafx.scene.control.ComboBox; import javafx.scene.control.ContextMenu; import javafx.scene.control.DialogPane; @@ -50,6 +51,7 @@ import javafx.scene.control.Menu; import javafx.scene.control.MenuBar; import javafx.scene.control.MenuItem; +import javafx.scene.control.RadioButton; import javafx.scene.control.Tab; import javafx.scene.control.TabPane; import javafx.scene.control.TableView; @@ -58,6 +60,7 @@ import javafx.scene.control.TreeCell; import javafx.scene.control.TreeItem; import javafx.stage.Stage; +import javafx.stage.Window; /** * @Description: MainViewTest verifies the data source tree and table tabs against an embedded database @@ -282,11 +285,11 @@ private MenuItem fileMenuItem(String id) { @Test void opensParquetImportFromMenuAndTree() { MenuItem importParquet = fileMenuItem("importParquetMenuItem"); - assertEquals("Import Parquet Files...", importParquet.getText()); + assertEquals("Import Parquet or CSV Files...", importParquet.getText()); assertEquals("Open Export Package...", fileMenuItem("openPackageMenuItem").getText()); Platform.runLater(importParquet::fire); DialogPane pane = FxTestSupport.waitForDialogPane(); - assertEquals("Load Parquet files into a table of a database", FxTestSupport.call(pane::getHeaderText)); + assertEquals("Load Parquet or CSV files into a table of a database", FxTestSupport.call(pane::getHeaderText)); assertTrue(FxTestSupport.call(() -> pane.lookupButton(ButtonType.OK).isDisabled())); FxTestSupport.closeDialog(ButtonType.CANCEL); // The context menu of a data source offers both imports @@ -301,7 +304,7 @@ void opensParquetImportFromMenuAndTree() { .flatMap(cell -> cell.getContextMenu().getItems().stream()).map(MenuItem::getText) .filter(text -> text != null).collect(Collectors.toList()); }); - assertTrue(items.containsAll(List.of("Import Export Package...", "Import Parquet Files...")), + assertTrue(items.containsAll(List.of("Import Export Package...", "Import Parquet or CSV Files...")), items.toString()); } @@ -424,4 +427,79 @@ void showsExitButton() { .getItems().stream().filter(item -> "Exit".equals(item.getText())).findFirst().orElseThrow()); assertNotNull(exitItem.getGraphic()); } + + /** + * Copies the schema into the same database by the export dialog of the main window, it starts without a + * question about the rows, like the import wizards of other tools + * + * @param commitEvery whether rows are committed every N rows instead of one transaction + */ + @SuppressWarnings("unchecked") + private void startCopy(String schema, boolean commitEvery, boolean confirm) { + TreeItem schemaItem = expandToSchema(); + FxTestSupport.run(() -> tree().getSelectionModel().select(schemaItem)); + FxTestSupport.fireLater((Button) FxTestSupport.call(() -> mainView.lookup("#action-export"))); + DialogPane export = FxTestSupport.waitForDialogPane(); + FxTestSupport.run(() -> ((RadioButton) export.lookup("#databaseRadio")).setSelected(true)); + ComboBox target = FxTestSupport.call(() -> (ComboBox) export + .lookup("#targetCombo")); + FxTestSupport.run(() -> target.setValue(target.getItems().stream().filter(item -> "Embedded H2".equals(item + .getName())).findFirst().orElseThrow())); + ComboBox targetSchema = FxTestSupport.call(() -> (ComboBox) export + .lookup("#targetSchemaCombo")); + FxTestSupport.waitUntil(() -> !targetSchema.getItems().isEmpty()); + FxTestSupport.run(() -> targetSchema.getEditor().setText(schema)); + // One transaction by default + assertTrue(FxTestSupport.call(() -> ((RadioButton) export.lookup("#oneTransactionRadio")).isSelected())); + if (commitEvery) { + FxTestSupport.run(() -> ((RadioButton) export.lookup("#commitEveryRadio")).setSelected(true)); + } + FxTestSupport.run(() -> ((CheckBox) export.lookup("#confirmCommitCheck")).setSelected(confirm)); + FxTestSupport.fireLater((Button) FxTestSupport.call(() -> export.lookupButton(ButtonType.OK))); + } + + private static void press(DialogPane pane, String text) { + FxTestSupport.fireLater(FxTestSupport.call(() -> pane.getButtonTypes().stream().map(type -> (Button) pane + .lookupButton(type)).filter(button -> text.equals(button.getText())).findFirst().orElseThrow())); + } + + private static Label progressStatus() { + FxTestSupport.waitUntil(() -> Window.getWindows().stream().anyMatch(window -> window.isShowing() + && window.getScene() != null && window.getScene().getRoot().lookup("#progressStatus") != null)); + return FxTestSupport.call(() -> (Label) Window.getWindows().stream().filter(window -> window.isShowing() + && window.getScene() != null && window.getScene().getRoot().lookup("#progressStatus") != null) + .findFirst().orElseThrow().getScene().getRoot().lookup("#progressStatus")); + } + + @Test + void commitsCopyEveryRows() { + startCopy("PAGES", true, false); + Label status = progressStatus(); + FxTestSupport.waitUntil(() -> "Completed".equals(status.getText())); + // No question after the import, the rows are committed already + assertTrue(FxTestSupport.getDialogPanes().stream().noneMatch(pane -> "decisionPane".equals(pane.getId()))); + FxTestSupport.run(() -> status.getScene().getWindow().hide()); + } + + @Test + void commitsCopyInOneTransaction() { + startCopy("ONE_TX", false, false); + Label status = progressStatus(); + // Committed when the import succeeds, like the import wizards of other tools + FxTestSupport.waitUntil(() -> "Completed".equals(status.getText())); + assertTrue(FxTestSupport.getDialogPanes().stream().noneMatch(pane -> "decisionPane".equals(pane.getId()))); + FxTestSupport.run(() -> status.getScene().getWindow().hide()); + } + + @Test + void confirmsCopyBeforeCommit() { + startCopy("CONFIRMED", false, true); + Label status = progressStatus(); + FxTestSupport.waitUntil(() -> FxTestSupport.getDialogPanes().stream().anyMatch(pane -> "decisionPane" + .equals(pane.getId()))); + press(FxTestSupport.getDialogPanes().stream().filter(pane -> "decisionPane".equals(pane.getId())) + .findFirst().orElseThrow(), "Commit"); + FxTestSupport.waitUntil(() -> "Committed".equals(status.getText())); + FxTestSupport.run(() -> status.getScene().getWindow().hide()); + } } diff --git a/src/test/java/com/github/sqljam/face/view/ProgressDialogTest.java b/src/test/java/com/github/sqljam/face/view/ProgressDialogTest.java index 7f1d642..a8be569 100644 --- a/src/test/java/com/github/sqljam/face/view/ProgressDialogTest.java +++ b/src/test/java/com/github/sqljam/face/view/ProgressDialogTest.java @@ -20,8 +20,15 @@ import static org.junit.jupiter.api.Assertions.assertTrue; import java.io.File; +import java.lang.reflect.Field; import java.nio.file.Files; +import java.sql.Connection; +import java.sql.DriverManager; +import java.sql.ResultSet; +import java.sql.Statement; +import java.util.List; import java.util.concurrent.atomic.AtomicReference; +import java.util.stream.Collectors; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.extension.ExtendWith; @@ -30,15 +37,21 @@ import org.testfx.framework.junit5.Start; import com.github.sqljam.impexp.ExportCancelledException; import com.github.sqljam.impexp.ExportListener; +import com.github.sqljam.jdbc.ManualTransaction; +import javafx.application.Platform; import javafx.scene.Parent; import javafx.scene.Scene; import javafx.scene.control.Button; +import javafx.scene.control.ButtonType; +import javafx.scene.control.CheckBox; +import javafx.scene.control.DialogPane; import javafx.scene.control.Label; import javafx.scene.control.ListView; import javafx.scene.control.ProgressBar; import javafx.scene.layout.StackPane; import javafx.stage.Stage; import javafx.stage.WindowEvent; +import javafx.util.Duration; /** * @Description: ProgressDialogTest verifies progress, errors, cancellation and finish states @@ -264,4 +277,353 @@ void cancelledByListenerWithoutException() { // Failures after cancelling are reported as cancelled assertEquals("Cancelled", text(dialog, "progressStatus")); } + + private String newDatabase() throws Exception { + String url = "jdbc:h2:mem:pd" + System.nanoTime() + ";DB_CLOSE_DELAY=-1"; + try (Connection connection = DriverManager.getConnection(url); + Statement statement = connection.createStatement()) { + statement.execute("CREATE TABLE t (id INT)"); + } + return url; + } + + private static long count(String url) throws Exception { + try (Connection connection = DriverManager.getConnection(url); + Statement statement = connection.createStatement(); + ResultSet rs = statement.executeQuery("SELECT COUNT(*) FROM t")) { + rs.next(); + return rs.getLong(1); + } + } + + /** + * Runs an import of one row in a manual transaction, an index after the row is deferred (H2 commits DDL) + */ + private static ManualTransaction runInTransaction(ProgressDialog dialog, String url, boolean error, + AtomicReference completed) throws Exception { + ManualTransaction transaction = new ManualTransaction(DriverManager.getConnection(url), false); + FxTestSupport.run(() -> dialog.run(() -> { + try (Statement statement = transaction.getConnection().createStatement()) { + statement.executeUpdate("INSERT INTO t VALUES (1)"); + statement.execute("CREATE INDEX idx_t ON t (id)"); + } + if (error) { + dialog.getListener().onError("value too long", null); + } + return 1; + }, rows -> "Imported " + rows + " rows", () -> transaction, () -> completed.set(true))); + return transaction; + } + + private static DialogPane decisionPane() { + FxTestSupport.waitUntil(() -> FxTestSupport.getDialogPanes().stream().anyMatch(pane -> "decisionPane" + .equals(pane.getId()))); + return FxTestSupport.getDialogPanes().stream().filter(pane -> "decisionPane".equals(pane.getId())) + .findFirst().orElseThrow(); + } + + private static void press(DialogPane pane, String text) { + Button button = FxTestSupport.call(() -> pane.getButtonTypes().stream().map(type -> (Button) pane + .lookupButton(type)).filter(item -> text.equals(item.getText())).findFirst().orElseThrow()); + FxTestSupport.fireLater(button); + } + + @Test + void commitsAfterQuestion() throws Exception { + String url = newDatabase(); + ProgressDialog dialog = create(1); + AtomicReference completed = new AtomicReference<>(false); + ManualTransaction transaction = runInTransaction(dialog, url, false, completed); + DialogPane pane = decisionPane(); + assertEquals("Commit the Import?", FxTestSupport.call(pane::getHeaderText)); + assertTrue(FxTestSupport.call(pane::getContentText).contains("\"Test\"")); + assertTrue(dialog.isPending()); + assertEquals("Waiting for commit", text(dialog, "progressStatus")); + assertFalse(dialog.isFinished()); + assertEquals("Tables: 1 / 1", text(dialog, "overallLabel")); + // Nothing is visible before the commit, the index waits for it + assertEquals(0, count(url)); + assertTrue(logs(dialog).stream().anyMatch(log -> log.startsWith("This database commits DDL at once"))); + assertTrue(logs(dialog).contains("1 statements (foreign keys, identity values) run after the commit.")); + assertFalse(completed.get()); + press(pane, "Commit"); + FxTestSupport.waitUntil(dialog::isFinished); + assertEquals("Committed", text(dialog, "progressStatus")); + assertEquals(1, count(url)); + assertTrue(completed.get()); + assertFalse(dialog.isPending()); + assertTrue(transaction.isFinished()); + assertTrue(FxTestSupport.call(() -> root(dialog).lookup("#cancelButton").isVisible())); + assertFalse(FxTestSupport.call(() -> root(dialog).lookup("#commitButton").isVisible())); + assertEquals(0, dialog.getErrorLogCount()); + } + + @Test + void rollsBackFromQuestion() throws Exception { + String url = newDatabase(); + ProgressDialog dialog = create(1); + AtomicReference completed = new AtomicReference<>(false); + runInTransaction(dialog, url, false, completed); + press(decisionPane(), "Roll Back"); + FxTestSupport.waitUntil(dialog::isFinished); + assertEquals("Rolled back", text(dialog, "progressStatus")); + assertEquals(0, count(url)); + assertTrue(completed.get()); + // A roll back by the user is not an error + assertEquals(0, dialog.getErrorLogCount()); + } + + @Test + void decidesLaterWithButtons() throws Exception { + String url = newDatabase(); + ProgressDialog dialog = create(1); + runInTransaction(dialog, url, false, new AtomicReference<>(false)); + press(decisionPane(), "Decide Later"); + FxTestSupport.waitUntil(() -> FxTestSupport.getDialogPanes().isEmpty()); + // The import still waits, the window offers both decisions + assertTrue(dialog.isPending()); + assertTrue(FxTestSupport.call(() -> root(dialog).lookup("#commitButton").isVisible())); + assertTrue(FxTestSupport.call(() -> root(dialog).lookup("#rollbackButton").isVisible())); + assertFalse(FxTestSupport.call(() -> root(dialog).lookup("#cancelButton").isVisible())); + FxTestSupport.fireLater((Button) FxTestSupport.call(() -> root(dialog).lookup("#commitButton"))); + FxTestSupport.waitUntil(dialog::isFinished); + assertEquals("Committed", text(dialog, "progressStatus")); + assertEquals(1, count(url)); + } + + @Test + void closingPendingWindowRollsBack() throws Exception { + String url = newDatabase(); + ProgressDialog dialog = create(1); + runInTransaction(dialog, url, false, new AtomicReference<>(false)); + press(decisionPane(), "Decide Later"); + FxTestSupport.waitUntil(() -> FxTestSupport.getDialogPanes().isEmpty()); + // Cancelled: the window stays and still waits + Platform.runLater(() -> dialog.getStage().fireEvent(new WindowEvent(dialog.getStage(), + WindowEvent.WINDOW_CLOSE_REQUEST))); + FxTestSupport.closeDialog(ButtonType.CANCEL); + assertTrue(dialog.isPending()); + assertTrue(FxTestSupport.call(() -> dialog.getStage().isShowing())); + // Confirmed: rolled back and closed + Platform.runLater(() -> dialog.getStage().fireEvent(new WindowEvent(dialog.getStage(), + WindowEvent.WINDOW_CLOSE_REQUEST))); + FxTestSupport.closeDialog(ButtonType.OK); + FxTestSupport.waitUntil(() -> !dialog.getStage().isShowing()); + assertEquals("Rolled back", text(dialog, "progressStatus")); + assertEquals(0, count(url)); + } + + @Test + void rollsBackImportWithErrors() throws Exception { + String url = newDatabase(); + ProgressDialog dialog = create(1); + AtomicReference completed = new AtomicReference<>(false); + ManualTransaction transaction = runInTransaction(dialog, url, true, completed); + FxTestSupport.waitUntil(dialog::isFinished); + // No question, nothing of the import is kept + assertTrue(FxTestSupport.getDialogPanes().isEmpty()); + assertEquals("Rolled back because of errors", text(dialog, "progressStatus")); + assertEquals(0, count(url)); + assertTrue(transaction.isFinished()); + assertTrue(completed.get()); + } + + @Test + void rollsBackFailedImport() throws Exception { + String url = newDatabase(); + ProgressDialog dialog = create(1); + ManualTransaction transaction = new ManualTransaction(DriverManager.getConnection(url), false); + FxTestSupport.run(() -> dialog.run(() -> { + try (Statement statement = transaction.getConnection().createStatement()) { + statement.executeUpdate("INSERT INTO t VALUES (1)"); + } + throw new IllegalStateException("connection lost"); + }, null, () -> transaction, null)); + FxTestSupport.waitUntil(dialog::isFinished); + assertEquals("Failed", text(dialog, "progressStatus")); + FxTestSupport.waitUntil(() -> logs(dialog).contains("Rolled back")); + assertEquals(0, count(url)); + assertTrue(transaction.isFinished()); + } + + @SuppressWarnings("unchecked") + private static List logs(ProgressDialog dialog) { + return FxTestSupport.call(() -> ((ListView) root(dialog).lookup("#logList")).getItems().stream() + .map(item -> { + try { + Field field = item.getClass().getDeclaredField("message"); + field.setAccessible(true); + return (String) field.get(item); + } catch (ReflectiveOperationException e) { + throw new IllegalStateException(e); + } + }).collect(Collectors.toList())); + } + + @Test + void remindsWhileWaitingForCommit() throws Exception { + String url = newDatabase(); + ProgressDialog dialog = create(1); + dialog.setReminderInterval(Duration.millis(400)); + runInTransaction(dialog, url, false, new AtomicReference<>(false)); + press(decisionPane(), "Decide Later"); + FxTestSupport.waitUntil(() -> FxTestSupport.getDialogPanes().stream().anyMatch(pane -> "reminderPane" + .equals(pane.getId()))); + DialogPane reminder = FxTestSupport.getDialogPanes().stream().filter(pane -> "reminderPane".equals(pane + .getId())).findFirst().orElseThrow(); + assertTrue(FxTestSupport.call(reminder::getContentText).contains("holds locks, undo and log space")); + // The reminder does not block the buttons of the window, later reminders are logged only + FxTestSupport.waitUntil(() -> logs(dialog).stream().filter(log -> log.startsWith("Waiting for commit for")) + .count() >= 2); + assertEquals(1, FxTestSupport.getDialogPanes().stream().filter(pane -> "reminderPane".equals(pane.getId())) + .count()); + assertTrue(text(dialog, "progressStatus").startsWith("Waiting for commit (")); + FxTestSupport.run(() -> reminder.getScene().getWindow().hide()); + FxTestSupport.fireLater((Button) FxTestSupport.call(() -> root(dialog).lookup("#commitButton"))); + FxTestSupport.waitUntil(dialog::isFinished); + assertEquals("Committed", text(dialog, "progressStatus")); + long reminders = logs(dialog).stream().filter(log -> log.startsWith("Waiting for commit for")).count(); + Thread.sleep(1000); + // No reminder after the decision + assertEquals(reminders, logs(dialog).stream().filter(log -> log.startsWith("Waiting for commit for")) + .count()); + assertTrue(FxTestSupport.getDialogPanes().isEmpty()); + } + + @Test + void remindsNotWhenTurnedOff() throws Exception { + context.getSettings().setConfirmCommitReminderMinutes(0); + String url = newDatabase(); + ProgressDialog dialog = create(1); + runInTransaction(dialog, url, false, new AtomicReference<>(false)); + press(decisionPane(), "Roll Back"); + FxTestSupport.waitUntil(dialog::isFinished); + assertTrue(logs(dialog).stream().noneMatch(log -> log.startsWith("Waiting for commit for"))); + } + + /** + * One transaction without confirm: committed when the import succeeds, like the import wizards of other tools + */ + @Test + void commitsOneTransactionAutomatically() throws Exception { + String url = newDatabase(); + ProgressDialog dialog = create(1); + ManualTransaction transaction = new ManualTransaction(DriverManager.getConnection(url), false); + AtomicReference completed = new AtomicReference<>(false); + FxTestSupport.run(() -> dialog.run(() -> { + try (Statement statement = transaction.getConnection().createStatement()) { + statement.executeUpdate("INSERT INTO t VALUES (1)"); + } + return 1; + }, rows -> "Imported " + rows + " rows", () -> transaction, false, () -> completed.set(true))); + FxTestSupport.waitUntil(dialog::isFinished); + assertTrue(FxTestSupport.getDialogPanes().isEmpty()); + assertEquals("Completed", text(dialog, "progressStatus")); + assertEquals(1, count(url)); + assertTrue(transaction.isFinished()); + assertTrue(completed.get()); + } + + @Test + void rollsBackOneTransactionWithErrors() throws Exception { + String url = newDatabase(); + ProgressDialog dialog = create(1); + ManualTransaction transaction = new ManualTransaction(DriverManager.getConnection(url), false); + FxTestSupport.run(() -> dialog.run(() -> { + try (Statement statement = transaction.getConnection().createStatement()) { + statement.executeUpdate("INSERT INTO t VALUES (1)"); + } + dialog.getListener().onError("value too long", null); + return 1; + }, null, () -> transaction, false, null)); + FxTestSupport.waitUntil(dialog::isFinished); + assertEquals("Rolled back because of errors", text(dialog, "progressStatus")); + assertEquals(0, count(url)); + } + + /** + * Rows with errors were skipped: one transaction is committed with the other rows + */ + @Test + void commitsSkippedErrorsAutomatically() throws Exception { + String url = newDatabase(); + ProgressDialog dialog = create(1); + FxTestSupport.run(() -> dialog.setSkipErrors(true)); + ManualTransaction transaction = new ManualTransaction(DriverManager.getConnection(url), false); + FxTestSupport.run(() -> dialog.run(() -> { + try (Statement statement = transaction.getConnection().createStatement()) { + statement.executeUpdate("INSERT INTO t VALUES (1)"); + } + dialog.getListener().onError("Skipped row of table 'T' (ID=2): value too long", null); + return 1; + }, null, () -> transaction, false, null)); + FxTestSupport.waitUntil(dialog::isFinished); + assertEquals("Completed with 1 errors", text(dialog, "progressStatus")); + assertEquals(1, count(url)); + assertTrue(logs(dialog).contains("Skipped row of table 'T' (ID=2): value too long")); + } + + /** + * Confirm before commit with skipped rows: the user is asked, commit keeps the other rows, roll back is not + * caused by the errors + */ + @Test + void confirmsImportWithSkippedErrors() throws Exception { + for (String decision : new String[]{"Commit", "Roll Back"}) { + String url = newDatabase(); + ProgressDialog dialog = create(1); + FxTestSupport.run(() -> dialog.setSkipErrors(true)); + AtomicReference completed = new AtomicReference<>(false); + runInTransaction(dialog, url, true, completed); + press(decisionPane(), decision); + FxTestSupport.waitUntil(dialog::isFinished); + if ("Commit".equals(decision)) { + assertEquals("Committed with 1 errors skipped", text(dialog, "progressStatus")); + assertEquals(1, count(url)); + } else { + assertEquals("Rolled back", text(dialog, "progressStatus")); + assertEquals(0, count(url)); + } + assertTrue(completed.get()); + FxTestSupport.run(() -> dialog.getStage().close()); + } + } + + /** + * The window shows the last entries, errors only on demand, the saved log keeps every entry + */ + @Test + @SuppressWarnings("unchecked") + void filtersAndSavesLog(@org.junit.jupiter.api.io.TempDir File dir) throws Exception { + ProgressDialog dialog = create(1); + FxTestSupport.run(() -> dialog.run(() -> { + for (int i = 1; i <= 6000; i++) { + dialog.getListener().onMessage("Row " + i); + } + dialog.getListener().onError("Skipped row of table 'T' (ID=7): bad value", null); + return 1; + }, null, null, false, null)); + FxTestSupport.waitUntil(dialog::isFinished); + ListView list = (ListView) root(dialog).lookup("#logList"); + assertEquals(5000, FxTestSupport.call(() -> list.getItems().size())); + CheckBox errorsOnly = (CheckBox) root(dialog).lookup("#errorsOnlyCheck"); + FxTestSupport.run(() -> errorsOnly.setSelected(true)); + assertEquals(List.of("Skipped row of table 'T' (ID=7): bad value"), logs(dialog)); + FxTestSupport.run(() -> errorsOnly.setSelected(false)); + assertEquals(5000, logs(dialog).size()); + assertTrue(FxTestSupport.call(() -> root(dialog).lookup("#saveLogButton").isVisible())); + File file = new File(dir, "import.log"); + FxTestSupport.run(() -> { + try { + dialog.writeLog(file); + } catch (java.io.IOException e) { + throw new IllegalStateException(e); + } + }); + List lines = Files.readAllLines(file.toPath()); + assertEquals("Status: Completed with 1 errors, errors and skipped items: 1", lines.get(1)); + assertTrue(lines.stream().anyMatch(line -> line.endsWith(" INFO Row 1"))); + assertTrue(lines.stream().anyMatch(line -> line.endsWith(" ERROR Skipped row of table 'T' (ID=7): bad value"))); + assertTrue(lines.size() > 6000); + } } diff --git a/src/test/java/com/github/sqljam/face/view/TutorialVideo.java b/src/test/java/com/github/sqljam/face/view/TutorialVideo.java index 95dfd3f..421674e 100644 --- a/src/test/java/com/github/sqljam/face/view/TutorialVideo.java +++ b/src/test/java/com/github/sqljam/face/view/TutorialVideo.java @@ -50,6 +50,7 @@ import javafx.scene.control.ButtonBase; import javafx.scene.control.CheckBox; import javafx.scene.control.ComboBox; +import javafx.scene.control.DialogPane; import javafx.scene.control.Label; import javafx.scene.control.Labeled; import javafx.scene.control.ListCell; @@ -69,8 +70,9 @@ /** * @Description: TutorialVideo records the tutorial video of SqlJam with the example database: connect, query rows, - * export into files, import the package into PostgreSQL, and export directly into SQL Server (created - * in the export dialog) and Oracle. Subtitles are in English, the background music is generated. + * export into CSV files, import the package into PostgreSQL, and export directly into SQL Server + * (created in the export dialog) and Oracle. Imports run in one transaction, the package import is + * confirmed before its commit. Subtitles are in English, the background music is generated. * It is recorded once per major version, it is not a test of the build (the name does not * end with Test), run it by * mvn test -Dtest=TutorialVideo -Dvideo.file=docs/assets/sqljam-tutorial.mp4 @@ -166,7 +168,7 @@ void record() throws Exception { FileUtils.deleteQuietly(packageDir); File output = new File(System.getProperty("video.file", "target/video/sqljam-tutorial.mp4")); recorder = FxTestSupport.call(() -> new VideoRecorder(stage, System.getProperty("ffmpeg", "ffmpeg"), 1.25, - 24, 88)); + 24, 100)); recorder.start(new File(work, "raw.mp4")); intro(); @@ -341,28 +343,30 @@ private void openExport() throws Exception { } private void exportFiles() throws Exception { - recorder.caption("Export to files: all tables of the schema, written for PostgreSQL"); + recorder.caption("Export to CSV files: all tables of the schema, written for PostgreSQL"); openExport(); scrollDialog(1); type((TextField) topNode("directoryField"), packageDir.getAbsolutePath()); choose(topNode("scriptDbTypeCombo"), DbType.POSTGRESQL); + RadioButton csv = (RadioButton) topNode("csvFormatRadio"); + press(csv, () -> csv.setSelected(true)); pause(1000); press(topNode("startButton")); - recorder.caption("The package holds the schema, data, constraints and a manifest"); + recorder.caption("The package holds the schema, CSV data files, constraints and a manifest"); Window progress = waitForProgress(); pause(1500); press(node(progress.getScene().getRoot(), "viewPackageButton")); Node viewer = waitFor(() -> topNode("packageViewer")); fitDialog(onFx(() -> viewer.getScene().getWindow())); recorder.caption("The package viewer shows every file of the package"); - pause(4000); + pause(3200); FxTestSupport.run(() -> viewer.getScene().getWindow().hide()); pause(500); closeProgress(progress); } private void importPackage() throws Exception { - recorder.caption("Import the package into the shop schema of PostgreSQL"); + recorder.caption("Import the CSV package into the shop schema of PostgreSQL"); select(connection(ExampleDatabase.PROFILE_NAME)); press(node(mainView, "action-importScripts")); TextField directory = waitFor(() -> (TextField) topNode("directoryField")); @@ -389,15 +393,19 @@ private void importPackage() throws Exception { }); choose(schema, SCHEMA); pause(800); + // One transaction rolls back on error, confirm before commit lets the user review the import first + CheckBox confirm = (CheckBox) topNode("confirmCommitCheck"); + recorder.caption("One transaction: an error rolls back everything, confirm to review first"); + press(confirm, () -> confirm.setSelected(true)); + pause(1200); press(topNode("startButton")); - Window progress = waitForProgress(); - pause(1800); + Window progress = commitImport("Review the import, then commit or roll back", 2600); closeProgress(progress); recorder.caption("The tables are in PostgreSQL now"); TreeItem postgresql = expand(connection("PostgreSQL demo")); TreeItem shop = expand(child(expand(child(postgresql, "demo")), SCHEMA)); assertEquals(TABLES.size(), FxTestSupport.call(() -> shop.getChildren().size()).intValue()); - pause(3000); + pause(2400); collapse(postgresql); } @@ -437,12 +445,13 @@ private void exportSqlServer() throws Exception { assertTrue(FxTestSupport.call(() -> ((CheckBox) topNode("createSchemaCheck")).isSelected())); pause(1000); press(topNode("startButton")); + // Committed when the import succeeds Window progress = waitForProgress(); - pause(1800); + pause(1500); closeProgress(progress); TreeItem sqlServer = expand(waitFor(() -> connection("SQL Server demo"))); expand(child(expand(child(sqlServer, "demo")), SCHEMA)); - pause(3000); + pause(2400); collapse(sqlServer); } @@ -466,7 +475,7 @@ private void exportOracle() throws Exception { press(topNode("startButton")); Window progress = waitForProgress(); recorder.caption("Types are translated, keys, indexes, comments and identities come along"); - pause(2500); + pause(2200); closeProgress(progress); pause(800); } @@ -487,6 +496,32 @@ private Window waitForProgress() throws InterruptedException { return window; } + /** + * Waits until the import into a database is complete and asks to commit, then commits it + * + * @param caption subtitle while the question is shown, null to keep the subtitle + */ + private Window commitImport(String caption, long think) throws InterruptedException { + Window window = waitFor(() -> Window.getWindows().stream().filter(Window::isShowing).filter(item -> item + .getScene() != null && item.getScene().getRoot().lookup("#progressStatus") != null).findFirst() + .orElse(null)); + fitDialog(window); + DialogPane question = waitFor(() -> (DialogPane) topNode("decisionPane")); + fitDialog(onFx(() -> question.getScene().getWindow())); + if (caption != null) { + recorder.caption(caption); + } + pause(think); + press(onFx(() -> question.getButtonTypes().stream().map(type -> (Button) question.lookupButton(type)) + .filter(button -> "Commit".equals(button.getText())).findFirst().orElseThrow())); + waitFor(() -> "Close".equals(((Labeled) window.getScene().getRoot().lookup("#cancelButton")).getText()) + ? true : null); + assertEquals("Committed", FxTestSupport.call(() -> ((Labeled) window.getScene().getRoot() + .lookup("#progressStatus")).getText())); + pause(1000); + return window; + } + /** * Dialogs are placed by the window system, the video shows them inside the main window: a higher dialog is * made smaller and every dialog is centered diff --git a/src/test/java/com/github/sqljam/face/view/VideoRecorder.java b/src/test/java/com/github/sqljam/face/view/VideoRecorder.java index cc0e633..168b9c8 100644 --- a/src/test/java/com/github/sqljam/face/view/VideoRecorder.java +++ b/src/test/java/com/github/sqljam/face/view/VideoRecorder.java @@ -273,7 +273,7 @@ void finish(File output, File music, String font) throws IOException, Interrupte private String toAss(String font) { int band = height - sceneHeight; - int size = Math.round(band * 0.34f); + int size = Math.round(band * 0.40f); StringBuilder ass = new StringBuilder(); ass.append("[Script Info]\nScriptType: v4.00+\nPlayResX: ").append(width).append("\nPlayResY: ") .append(height).append("\nWrapStyle: 0\n\n[V4+ Styles]\n") diff --git a/src/test/java/com/github/sqljam/impexp/CsvSplitterTest.java b/src/test/java/com/github/sqljam/impexp/CsvSplitterTest.java new file mode 100644 index 0000000..2c982b0 --- /dev/null +++ b/src/test/java/com/github/sqljam/impexp/CsvSplitterTest.java @@ -0,0 +1,134 @@ +/* + * Copyright 2023-2026 Fred Feng + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.github.sqljam.impexp; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.io.File; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.util.ArrayList; +import java.util.List; + +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +/** + * @Description: CsvSplitterTest verifies parts of large CSV files: named like data files of sql scripts, each with + * the header row, records are never cut (also with line breaks and quotes inside fields), and the + * order of importing by part numbers + * @Author: Fred Feng + * @Date: 26/03/2023 + * @Version 1.0.0 + */ +class CsvSplitterTest { + + private static final String HEADER = "id,name,note\n"; + + @TempDir + File dir; + + private File write(String name, int rows) throws Exception { + StringBuilder csv = new StringBuilder(HEADER); + for (int i = 1; i <= rows; i++) { + // Line breaks, delimiters and escaped quotes inside quoted fields + csv.append(i).append(",\"name ").append(i).append("\",\"line one\nline \"\"two\"\", ") + .append("中文 ").append(i).append("\"\n"); + } + File file = new File(dir, name); + Files.writeString(file.toPath(), csv.toString(), StandardCharsets.UTF_8); + return file; + } + + /** + * Records of the parts without their headers + */ + private static List records(List parts) throws Exception { + List records = new ArrayList<>(); + for (File part : parts) { + String text = Files.readString(part.toPath(), StandardCharsets.UTF_8); + assertTrue(text.startsWith(HEADER), "Header of " + part.getName()); + String body = text.substring(HEADER.length()); + // Records end with a quote and a line break + for (String record : body.split("(?<=\")\n")) { + if (!record.isEmpty()) { + records.add(record); + } + } + } + return records; + } + + @Test + void splitsAtRecords() throws Exception { + File file = write("orders.csv", 200); + String original = Files.readString(file.toPath(), StandardCharsets.UTF_8); + List parts = CsvSplitter.split(file, 1024); + assertTrue(parts.size() > 5, "Parts: " + parts.size()); + assertEquals("orders.csv", parts.get(0).getName()); + assertEquals("orders_2.csv", parts.get(1).getName()); + assertEquals("orders_" + parts.size() + ".csv", parts.get(parts.size() - 1).getName()); + for (File part : parts) { + assertTrue(part.length() <= 1024, part.getName() + " " + part.length()); + } + // Nothing is lost or reordered, the temporary file is removed + List records = records(parts); + assertEquals(200, records.size()); + assertEquals(original.substring(HEADER.length()), String.join("\n", records) + "\n"); + assertEquals(parts.size(), dir.listFiles().length); + } + + @Test + void keepsSmallFiles() throws Exception { + File file = write("small.csv", 3); + long size = file.length(); + assertEquals(List.of(file), CsvSplitter.split(file, size)); + assertEquals(List.of(file), CsvSplitter.split(file, 0)); + assertEquals(size, file.length()); + } + + @Test + void keepsLargeRecordsWhole() throws Exception { + // Each record is larger than the maximum size: one record per part + File file = write("large.csv", 5); + List parts = CsvSplitter.split(file, 10); + assertEquals(5, parts.size()); + assertEquals(5, records(parts).size()); + } + + @Test + void splitsFileWithoutTrailingLineBreak() throws Exception { + File file = new File(dir, "plain.csv"); + Files.writeString(file.toPath(), HEADER + "1,\"a\",\"x\"\n2,\"b\",\"y\"", StandardCharsets.UTF_8); + List parts = CsvSplitter.split(file, HEADER.length() + 12); + assertEquals(2, parts.size()); + assertEquals(HEADER + "2,\"b\",\"y\"", Files.readString(parts.get(1).toPath(), StandardCharsets.UTF_8)); + } + + @Test + void sortsPartsByNumber() { + List files = new ArrayList<>(); + for (String name : new String[]{"orders_10.csv", "orders_2.csv", "items.csv", "orders.csv", "orders_9.csv", + "items_2.csv", "orders_x.csv"}) { + files.add(new File(dir, name)); + } + List names = new ArrayList<>(); + CsvSplitter.sortParts(files).forEach(file -> names.add(file.getName())); + assertEquals(List.of("items.csv", "items_2.csv", "orders.csv", "orders_2.csv", "orders_9.csv", + "orders_10.csv", "orders_x.csv"), names); + } +} diff --git a/src/test/java/com/github/sqljam/impexp/DataFileBoundaryTest.java b/src/test/java/com/github/sqljam/impexp/DataFileBoundaryTest.java new file mode 100644 index 0000000..4833885 --- /dev/null +++ b/src/test/java/com/github/sqljam/impexp/DataFileBoundaryTest.java @@ -0,0 +1,343 @@ +/* + * Copyright 2023-2026 Fred Feng + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.github.sqljam.impexp; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.io.File; +import java.nio.charset.Charset; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.sql.Connection; +import java.sql.DriverManager; +import java.sql.ResultSet; +import java.sql.Statement; +import java.util.ArrayList; +import java.util.List; +import java.util.UUID; + +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +/** + * @Description: DataFileBoundaryTest verifies CSV and Parquet packages at their boundaries with H2 databases: empty + * tables (a header only), names with delimiters, quotes and spaces, NULL and empty strings, a table of + * one row which is not split, and parts of a large table in the order of its rows + * @Author: Fred Feng + * @Date: 26/03/2023 + * @Version 1.0.0 + */ +class DataFileBoundaryTest { + + @TempDir + File dir; + + private static String newUrl() { + return "jdbc:h2:mem:df" + UUID.randomUUID().toString().replace("-", "") + ";DB_CLOSE_DELAY=-1"; + } + + private static void execute(String url, String... sqls) throws Exception { + try (Connection connection = DriverManager.getConnection(url, "sa", ""); + Statement statement = connection.createStatement()) { + for (String sql : sqls) { + statement.execute(sql); + } + } + } + + private static List> rows(String url, String sql) throws Exception { + List> rows = new ArrayList<>(); + try (Connection connection = DriverManager.getConnection(url, "sa", ""); + Statement statement = connection.createStatement(); ResultSet rs = statement.executeQuery(sql)) { + int columns = rs.getMetaData().getColumnCount(); + while (rs.next()) { + List row = new ArrayList<>(); + for (int i = 1; i <= columns; i++) { + row.add(rs.getString(i)); + } + rows.add(row); + } + } + return rows; + } + + private static String sourceUrl() throws Exception { + String url = newUrl(); + execute(url, "CREATE TABLE \"EMPTY\" (ID INT PRIMARY KEY, NAME VARCHAR(20))", + "CREATE TABLE \"ODD, \"\"NAMES\"\"\" (ID INT PRIMARY KEY, \"A;B|C\" VARCHAR(50), \"QUOTE\"\"D\" " + + "VARCHAR(50), \"TAB\tCOL\" VARCHAR(50))", + "INSERT INTO \"ODD, \"\"NAMES\"\"\" VALUES (1, '', NULL, 'tab\there'), (2, 'a,b;c|d', '\"q\"', " + + "'line\nbreak'), (3, NULL, '', ' ')", + "CREATE TABLE ONE_ROW (ID INT PRIMARY KEY, NOTE VARCHAR(20))", + "INSERT INTO ONE_ROW VALUES (1, 'only')", + "CREATE TABLE MANY (ID INT PRIMARY KEY, NOTE VARCHAR(100))", + "INSERT INTO MANY SELECT X, REPEAT('x', 80) || X FROM SYSTEM_RANGE(1, 3000)"); + return url; + } + + private static void configure(Exporter.ExportConfiguration configuration, String url) { + configuration.setDbType(DbType.H2); + configuration.setUrl(url); + configuration.setUsername("sa"); + configuration.setPassword(""); + configuration.setIncludedSchemaNames(new String[]{"PUBLIC"}); + configuration.setIdReused(true); + configuration.setConnectionPoolEnabled(false); + } + + private static void importPackage(File packageDir, String targetUrl) throws Exception { + try (Connection connection = DriverManager.getConnection(targetUrl, "sa", "")) { + AbstractFileImporter importer = AbstractFileImporter.forDirectory(packageDir, connection, DbType.H2); + importer.importDirectory(packageDir); + assertEquals(0, importer.getFailedCount()); + } + } + + private static void verify(String source, String target) throws Exception { + assertEquals(List.of(), rows(target, "SELECT * FROM \"EMPTY\"")); + String odd = "SELECT * FROM \"ODD, \"\"NAMES\"\"\" ORDER BY ID"; + assertEquals(rows(source, odd), rows(target, odd)); + // NULL and empty strings stay apart + assertNull(rows(target, odd).get(0).get(2)); + assertEquals("", rows(target, odd).get(0).get(1)); + assertEquals("", rows(target, odd).get(2).get(2)); + assertEquals(List.of(List.of("1", "only")), rows(target, "SELECT * FROM ONE_ROW")); + String many = "SELECT * FROM MANY ORDER BY ID"; + assertEquals(rows(source, many), rows(target, many)); + } + + @Test + void csvPackageAtBoundaries() throws Exception { + String source = sourceUrl(); + File packageDir = new File(dir, "csv"); + CsvExporter exporter = new CsvExporter(packageDir); + configure(exporter.getConfiguration(), source); + exporter.setDelimiter(";"); + exporter.setMaxFileSize(32 * 1024); + exporter.exportDdlAndData(); + ExportManifest manifest = ExportManifest.read(packageDir); + assertEquals(";", manifest.getOptions().get("delimiter")); + ExportManifest.TableEntry empty = table(manifest, "EMPTY"); + assertEquals(0, empty.getRows()); + assertEquals(1, empty.getDataFiles().size()); + // An empty table is a header only + assertEquals("ID;NAME\n", Files.readString(new File(packageDir, empty.getDataFiles().iterator().next()) + .toPath(), StandardCharsets.UTF_8)); + assertEquals(List.of("ID", "NAME"), table(manifest, "EMPTY").getColumns().stream() + .map(ExportManifest.ColumnEntry::getName).toList()); + assertEquals(1, table(manifest, "ONE_ROW").getDataFiles().size()); + List manyFiles = List.copyOf(table(manifest, "MANY").getDataFiles()); + assertTrue(manyFiles.size() > 3, manyFiles.toString()); + assertTrue(manyFiles.get(1).endsWith("MANY_2.csv"), manyFiles.toString()); + String target = newUrl(); + importPackage(packageDir, target); + verify(source, target); + } + + @Test + void parquetPackageAtBoundaries() throws Exception { + String source = sourceUrl(); + File packageDir = new File(dir, "parquet"); + ParquetExporter exporter = new ParquetExporter(packageDir); + configure(exporter.getConfiguration(), source); + exporter.setMaxFileSize(8 * 1024); + exporter.exportDdlAndData(); + ExportManifest manifest = ExportManifest.read(packageDir); + assertEquals(8 * 1024, ((Number) manifest.getOptions().get("maxFileSize")).longValue()); + assertEquals(1, table(manifest, "EMPTY").getDataFiles().size()); + // One row is never split + assertEquals(1, table(manifest, "ONE_ROW").getDataFiles().size()); + List manyFiles = List.copyOf(table(manifest, "MANY").getDataFiles()); + assertTrue(manyFiles.size() > 1, manyFiles.toString()); + assertTrue(manyFiles.get(1).endsWith("MANY_2.parquet"), manyFiles.toString()); + // Each part holds the next rows of the table + try (Connection connection = DriverManager.getConnection("jdbc:duckdb:"); + Statement statement = connection.createStatement()) { + long previousMax = 0; + for (String path : manyFiles) { + try (ResultSet rs = statement.executeQuery("SELECT MIN(ID), MAX(ID) FROM read_parquet('" + + new File(packageDir, path).getAbsolutePath() + "')")) { + rs.next(); + assertEquals(previousMax + 1, rs.getLong(1), path); + previousMax = rs.getLong(2); + } + } + assertEquals(3000, previousMax); + } + String target = newUrl(); + importPackage(packageDir, target); + verify(source, target); + } + + @Test + void emptyCsvFileOfOtherTools() throws Exception { + File file = new File(dir, "empty.csv"); + Files.writeString(file.toPath(), "id,name\n", StandardCharsets.UTF_8); + String target = newUrl(); + try (Connection connection = DriverManager.getConnection(target, "sa", "")) { + CsvImporter importer = new CsvImporter(connection, DbType.H2); + importer.setDelimiter(","); + assertEquals(0, importer.importFiles(List.of(file), "EMPTY_CSV", DataFileImporter.Mode.CREATE)); + } + assertEquals(List.of(List.of("0")), rows(target, "SELECT COUNT(*) FROM EMPTY_CSV")); + } + + private static ExportManifest.TableEntry table(ExportManifest manifest, String name) { + return manifest.getTables().stream().filter(table -> name.equals(table.getName())).findFirst() + .orElseThrow(() -> new AssertionError("No table " + name)); + } + + private static String optionsSource() throws Exception { + String url = newUrl(); + execute(url, "CREATE TABLE NOTES (ID INT PRIMARY KEY, TXT VARCHAR(50), D DATE, TS TIMESTAMP)", + "INSERT INTO NOTES VALUES (1, 'NULL', DATE '2024-01-31', TIMESTAMP '2024-01-31 08:09:10')," + + " (2, NULL, NULL, NULL), (3, '', DATE '1999-12-31', TIMESTAMP '1999-12-31 23:59:59')," + + " (4, 'it''s 中文; \"q\"', DATE '2000-02-29', TIMESTAMP '2000-02-29 00:00:00')"); + return url; + } + + private File exportNotes(String source, CsvOptions options, String name) throws Exception { + File packageDir = new File(dir, name); + CsvExporter exporter = new CsvExporter(packageDir); + configure(exporter.getConfiguration(), source); + exporter.setOptions(options); + exporter.exportDdlAndData(); + return packageDir; + } + + /** + * Options other than the defaults keep every value: the text of NULL apart from the same string, single quotes, + * a semicolon, no header, GBK and a date format without fractions of seconds + */ + @Test + void csvOptionsKeepValues() throws Exception { + String source = optionsSource(); + CsvOptions options = new CsvOptions(); + options.setDelimiter(";"); + options.setQuote("'"); + options.setNullString("NULL"); + options.setHeader(false); + options.setEncoding("GBK"); + options.setDateFormat("%d.%m.%Y"); + File packageDir = exportNotes(source, options, "options"); + ExportManifest manifest = ExportManifest.read(packageDir); + assertEquals(options, CsvOptions.fromMap(manifest.getOptions())); + File data = new File(packageDir, table(manifest, "NOTES").getDataFiles().iterator().next()); + String text = new String(Files.readAllBytes(data.toPath()), Charset.forName("GBK")); + // No header, dates in the format, NULL as text and the string NULL quoted + assertTrue(text.startsWith("1;'NULL';31.01.2024;31.01.2024 08:09:10"), text); + assertTrue(text.contains("2;NULL;NULL;NULL"), text); + assertTrue(text.contains("中文"), text); + String target = newUrl(); + importPackage(packageDir, target); + String query = "SELECT * FROM NOTES ORDER BY ID"; + assertEquals(rows(source, query), rows(target, query)); + assertEquals("NULL", rows(target, query).get(0).get(1)); + assertNull(rows(target, query).get(1).get(1)); + assertEquals("", rows(target, query).get(2).get(1)); + } + + @Test + void csvWithByteOrderMark() throws Exception { + String source = optionsSource(); + CsvOptions options = new CsvOptions(); + options.setEncoding(CsvOptions.UTF_8_BOM); + File packageDir = exportNotes(source, options, "bom"); + ExportManifest manifest = ExportManifest.read(packageDir); + byte[] bytes = Files.readAllBytes(new File(packageDir, table(manifest, "NOTES").getDataFiles().iterator() + .next()).toPath()); + // Excel recognizes UTF-8 by the byte order mark + assertEquals(List.of((byte) 0xEF, (byte) 0xBB, (byte) 0xBF), List.of(bytes[0], bytes[1], bytes[2])); + String target = newUrl(); + importPackage(packageDir, target); + assertEquals(rows(source, "SELECT * FROM NOTES ORDER BY ID"), rows(target, "SELECT * FROM NOTES ORDER BY ID")); + } + + /** + * Characters which the encoding can not hold stop the export instead of being replaced + */ + @Test + void refusesCharactersOutsideTheEncoding() throws Exception { + String source = optionsSource(); + CsvOptions options = new CsvOptions(); + options.setEncoding("ISO-8859-1"); + ImpExpException e = assertThrows(ImpExpException.class, () -> exportNotes(source, options, "latin")); + assertTrue(e.getMessage().contains("can not be converted from UTF-8 to ISO-8859-1"), e.getMessage()); + } + + @Test + void importsCsvFileOfOtherToolWithOptions() throws Exception { + File file = new File(dir, "sales.txt"); + Files.write(file.toPath(), "1;北京;10.5\n2;上海;\n3;'a;b';7\n".getBytes(Charset.forName("GBK"))); + CsvOptions options = new CsvOptions(); + options.setDelimiter(";"); + options.setQuote("'"); + options.setHeader(false); + options.setEncoding("GBK"); + String target = newUrl(); + try (Connection connection = DriverManager.getConnection(target, "sa", "")) { + CsvImporter importer = new CsvImporter(connection, DbType.H2); + importer.setOptions(options); + assertEquals(3, importer.importFiles(List.of(file), "SALES", DataFileImporter.Mode.CREATE)); + } + List> rows = rows(target, "SELECT * FROM SALES ORDER BY 1"); + assertEquals("北京", rows.get(0).get(1)); + assertNull(rows.get(1).get(2)); + assertEquals("a;b", rows.get(2).get(1)); + } + + /** + * CSV files of other tools into an existing table: rows of existing keys are kept, a row with an error is + * skipped and reported when errors are skipped, otherwise it stops the import + */ + @Test + void skipsExistingRowsAndErrorsOfFiles() throws Exception { + File file = new File(dir, "items.csv"); + Files.write(file.toPath(), "id,note\n1,new\n2,b\n3,toolongvalue\n4,d\n".getBytes(StandardCharsets.UTF_8)); + String target = newUrl(); + try (Connection connection = DriverManager.getConnection(target, "sa", ""); + Statement statement = connection.createStatement()) { + statement.execute("CREATE TABLE ITEMS (ID INT PRIMARY KEY, NOTE VARCHAR(5))"); + statement.execute("INSERT INTO ITEMS VALUES (1, 'old')"); + } + try (Connection connection = DriverManager.getConnection(target, "sa", "")) { + CsvImporter importer = new CsvImporter(connection, DbType.H2); + importer.setPageSize(2); + assertThrows(Exception.class, () -> importer.importFiles(List.of(file), "ITEMS", + DataFileImporter.Mode.SKIP_EXISTING)); + } + // The first page is committed before the error stops the import + assertEquals(List.of(List.of("1", "old"), List.of("2", "b")), rows(target, "SELECT * FROM ITEMS ORDER BY 1")); + try (Connection connection = DriverManager.getConnection(target, "sa", "")) { + CsvImporter importer = new CsvImporter(connection, DbType.H2); + importer.setPageSize(2); + importer.setStopOnError(false); + com.github.sqljam.it.CollectingListener listener = new com.github.sqljam.it.CollectingListener(); + importer.setExportListener(listener); + // Rows without errors are counted, the skipped row is not + assertEquals(3, importer.importFiles(List.of(file), "ITEMS", DataFileImporter.Mode.SKIP_EXISTING)); + assertEquals(1, listener.getErrors().size(), listener.getErrors().toString()); + // The row is named by the columns of the target table + assertTrue(listener.getErrors().get(0).startsWith("Skipped row of table 'ITEMS' (ID=3): "), + listener.getErrors().get(0)); + } + assertEquals(List.of(List.of("1", "old"), List.of("2", "b"), List.of("4", "d")), rows(target, + "SELECT * FROM ITEMS ORDER BY 1")); + } +} diff --git a/src/test/java/com/github/sqljam/impexp/ImportBoundaryTest.java b/src/test/java/com/github/sqljam/impexp/ImportBoundaryTest.java new file mode 100644 index 0000000..70a2702 --- /dev/null +++ b/src/test/java/com/github/sqljam/impexp/ImportBoundaryTest.java @@ -0,0 +1,320 @@ +/* + * Copyright 2023-2026 Fred Feng + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.github.sqljam.impexp; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.sql.Connection; +import java.sql.DriverManager; +import java.sql.ResultSet; +import java.sql.Statement; +import java.util.ArrayList; +import java.util.List; +import java.util.UUID; + +import org.junit.jupiter.api.Test; +import com.github.sqljam.it.CollectingListener; +import com.github.sqljam.jdbc.ManualTransaction; +import com.github.sqljam.jdbc.SharedConnectionFactory; + +/** + * @Description: ImportBoundaryTest verifies upserts and transactions of imports at their boundaries with H2: empty + * tables, NULL values, missing key columns, key columns narrowed away by a query, new tables, an error + * in the middle of commits every N rows and in one transaction, rows with errors which are skipped, + * existing rows which are skipped, and a transaction without rows + * @Author: Fred Feng + * @Date: 26/03/2023 + * @Version 1.0.0 + */ +class ImportBoundaryTest { + + private static String newUrl() { + return "jdbc:h2:mem:ib" + UUID.randomUUID().toString().replace("-", "") + ";DB_CLOSE_DELAY=-1"; + } + + private static void execute(String url, String... sqls) throws Exception { + try (Connection connection = DriverManager.getConnection(url, "sa", ""); + Statement statement = connection.createStatement()) { + for (String sql : sqls) { + statement.execute(sql); + } + } + } + + private static List column(String url, String sql) throws Exception { + List values = new ArrayList<>(); + try (Connection connection = DriverManager.getConnection(url, "sa", ""); + Statement statement = connection.createStatement(); ResultSet rs = statement.executeQuery(sql)) { + while (rs.next()) { + values.add(rs.getString(1)); + } + } + return values; + } + + private static ImportExporter importer(String source, String target, WriteMode writeMode) { + ImportExporter importExporter = new ImportExporter(); + Exporter.ExportConfiguration configuration = importExporter.getExportConfiguration(); + configuration.setDbType(DbType.H2); + configuration.setUrl(source); + configuration.setUsername("sa"); + configuration.setPassword(""); + configuration.setIncludedSchemaNames(new String[]{"PUBLIC"}); + configuration.setIdReused(true); + configuration.setConnectionPoolEnabled(false); + configuration.setTableRecreated(false); + configuration.setWriteMode(writeMode); + configuration.setPageSize(2); + importExporter.getImportConfiguration().setDbType(DbType.H2); + importExporter.getImportConfiguration().setUrl(target); + importExporter.getImportConfiguration().setUsername("sa"); + importExporter.getImportConfiguration().setPassword(""); + importExporter.getImportConfiguration().setConnectionPoolEnabled(false); + return importExporter; + } + + @Test + void upsertsEmptyTablesAndNulls() throws Exception { + String source = newUrl(); + String target = newUrl(); + execute(source, "CREATE TABLE EMPTY (ID INT PRIMARY KEY)", + "CREATE TABLE ITEMS (ID INT PRIMARY KEY, NOTE VARCHAR(20))", "INSERT INTO ITEMS VALUES (1, NULL)"); + execute(target, "CREATE TABLE ITEMS (ID INT PRIMARY KEY, NOTE VARCHAR(20))", + "INSERT INTO ITEMS VALUES (1, 'old')"); + CollectingListener listener = new CollectingListener(); + ImportExporter importExporter = importer(source, target, WriteMode.UPSERT); + importExporter.setExportListener(listener); + importExporter.exportDdlAndData(); + assertEquals(List.of(), listener.getErrors()); + // The empty table is created, NULL replaces the old value + assertEquals(List.of("0"), column(target, "SELECT COUNT(*) FROM EMPTY")); + assertEquals(1, column(target, "SELECT NOTE FROM ITEMS").size()); + assertNull(column(target, "SELECT NOTE FROM ITEMS").get(0)); + } + + @Test + void refusesKeysWhichAreNotImported() throws Exception { + String source = newUrl(); + String target = newUrl(); + execute(source, "CREATE TABLE ITEMS (ID INT PRIMARY KEY, NOTE VARCHAR(20))", "INSERT INTO ITEMS VALUES (1, 'a')"); + // A chosen column which does not exist + ImportExporter missing = importer(source, target, WriteMode.UPSERT); + missing.getExportConfiguration().getKeyColumnNames().put("ITEMS", List.of("CODE")); + ImpExpException e = assertThrows(ImpExpException.class, missing::exportDdlAndData); + assertTrue(e.getMessage().contains("Key column CODE of table ITEMS is not imported"), e.getMessage()); + // The primary key is not a column of the query + ImportExporter narrowed = importer(source, target, WriteMode.UPSERT); + narrowed.getExportConfiguration().getTableQueries().put("ITEMS", new TableQuery(List.of("NOTE"), null, null, + null)); + e = assertThrows(ImpExpException.class, narrowed::exportDdlAndData); + assertTrue(e.getMessage().contains("Key column ID of table ITEMS is not imported"), e.getMessage()); + } + + private static String failingSource() throws Exception { + String source = newUrl(); + execute(source, "CREATE TABLE ITEMS (ID INT PRIMARY KEY, AMOUNT INT)", + "INSERT INTO ITEMS SELECT X, X FROM SYSTEM_RANGE(1, 6)"); + return source; + } + + /** + * The fifth row breaks a check of the target: pages of two rows before it are committed, one transaction keeps + * nothing + */ + @Test + void errorInTheMiddleOfAnImport() throws Exception { + String source = failingSource(); + String everyRows = newUrl(); + String oneTransaction = newUrl(); + for (String target : new String[]{everyRows, oneTransaction}) { + execute(target, "CREATE TABLE ITEMS (ID INT PRIMARY KEY, AMOUNT INT CHECK (AMOUNT <> 5))"); + } + ImportExporter pages = importer(source, everyRows, WriteMode.INSERT); + pages.setExportListener(new CollectingListener()); + assertThrows(Exception.class, pages::exportDdlAndData); + assertEquals(List.of("1", "2", "3", "4"), column(everyRows, "SELECT ID FROM ITEMS ORDER BY ID")); + + ImportExporter transactional = importer(source, oneTransaction, WriteMode.INSERT); + transactional.setExportListener(new CollectingListener()); + try (ManualTransaction transaction = new ManualTransaction(DriverManager.getConnection(oneTransaction, "sa", + ""), false)) { + transactional.setTargetConnectionFactory(new SharedConnectionFactory(transaction.getConnection())); + assertThrows(Exception.class, transactional::exportDdlAndData); + assertTrue(transaction.isRolledBack()); + } + assertEquals(List.of(), column(oneTransaction, "SELECT ID FROM ITEMS")); + } + + @Test + void commitsTransactionWithoutRows() throws Exception { + String source = newUrl(); + String target = newUrl(); + execute(source, "CREATE TABLE EMPTY (ID INT PRIMARY KEY)"); + ImportExporter importExporter = importer(source, target, WriteMode.INSERT); + try (ManualTransaction transaction = new ManualTransaction(DriverManager.getConnection(target, "sa", ""), + false)) { + importExporter.setTargetConnectionFactory(new SharedConnectionFactory(transaction.getConnection())); + importExporter.exportDdlAndData(); + assertEquals(List.of(), transaction.commit()); + } + assertEquals(List.of("0"), column(target, "SELECT COUNT(*) FROM EMPTY")); + } + + /** + * The fifth row breaks a check of the target and is skipped: the other rows are written by commits every N + * rows and in one transaction, each skipped row is reported with its key + */ + @Test + void skipsRowsWithErrors() throws Exception { + String source = failingSource(); + String everyRows = newUrl(); + String oneTransaction = newUrl(); + for (String target : new String[]{everyRows, oneTransaction}) { + execute(target, "CREATE TABLE ITEMS (ID INT PRIMARY KEY, AMOUNT INT CHECK (AMOUNT <> 5))"); + } + ImportExporter pages = importer(source, everyRows, WriteMode.INSERT); + pages.getExportConfiguration().setFailFast(false); + List tableRows = new ArrayList<>(); + CollectingListener listener = new CollectingListener() { + @Override + public void onTableEnd(String catalogName, String schemaName, String tableName, long processedRows) { + tableRows.add(processedRows); + } + }; + pages.setExportListener(listener); + pages.exportDdlAndData(); + assertEquals(List.of("1", "2", "3", "4", "6"), column(everyRows, "SELECT ID FROM ITEMS ORDER BY ID")); + assertEquals(1, listener.getErrors().size()); + // The table reports the written rows, the skipped row is not counted + assertEquals(List.of(5L), tableRows); + assertTrue(listener.getErrors().get(0).startsWith("Skipped row of table 'ITEMS' (ID=5): "), + listener.getErrors().get(0)); + + ImportExporter transactional = importer(source, oneTransaction, WriteMode.INSERT); + transactional.getExportConfiguration().setFailFast(false); + listener = new CollectingListener(); + transactional.setExportListener(listener); + try (ManualTransaction transaction = new ManualTransaction(DriverManager.getConnection(oneTransaction, "sa", + ""), false)) { + transactional.setTargetConnectionFactory(new SharedConnectionFactory(transaction.getConnection())); + transactional.exportDdlAndData(); + // The savepoint of the page keeps the transaction + assertTrue(!transaction.isRolledBack()); + assertEquals(List.of(), transaction.commit()); + } + assertEquals(List.of("1", "2", "3", "4", "6"), column(oneTransaction, "SELECT ID FROM ITEMS ORDER BY ID")); + assertEquals(1, listener.getErrors().size()); + } + + /** + * DuckDB has no savepoints: commits every N rows write the rows of a failed page one by one, one transaction + * can not go on and is rolled back + */ + @Test + void skipsRowsWithoutSavepoints(@org.junit.jupiter.api.io.TempDir java.io.File dir) throws Exception { + String source = failingSource(); + for (boolean managed : new boolean[]{false, true}) { + String target = "jdbc:duckdb:" + new java.io.File(dir, "skip" + managed + ".duckdb").getAbsolutePath(); + try (Connection connection = DriverManager.getConnection(target); + Statement statement = connection.createStatement()) { + // Tables of the source schema PUBLIC go into the schema public + statement.execute("CREATE SCHEMA public"); + statement.execute("CREATE TABLE public.items (id INT PRIMARY KEY, amount INT CHECK (amount <> 5))"); + } + ImportExporter importExporter = importer(source, target, WriteMode.INSERT); + importExporter.getImportConfiguration().setUsername(null); + importExporter.getImportConfiguration().setPassword(null); + importExporter.getExportConfiguration().setFailFast(false); + CollectingListener listener = new CollectingListener(); + importExporter.setExportListener(listener); + if (!managed) { + importExporter.exportDdlAndData(); + assertEquals(List.of("1", "2", "3", "4", "6"), duckColumn(target)); + assertEquals(1, listener.getErrors().size()); + continue; + } + try (ManualTransaction transaction = new ManualTransaction(DriverManager.getConnection(target), true)) { + importExporter.setTargetConnectionFactory(new SharedConnectionFactory(transaction.getConnection())); + importExporter.exportDdlAndData(); + assertTrue(transaction.isRolledBack()); + assertTrue(listener.getErrors().stream().anyMatch(error -> error.contains( + "can not be skipped in one transaction without savepoints")), listener.getErrors().toString()); + } + assertEquals(List.of(), duckColumn(target)); + } + } + + private static List duckColumn(String url) throws Exception { + List values = new ArrayList<>(); + try (Connection connection = DriverManager.getConnection(url); + Statement statement = connection.createStatement(); + ResultSet rs = statement.executeQuery("SELECT id FROM public.items ORDER BY id")) { + while (rs.next()) { + values.add(rs.getString(1)); + } + } + return values; + } + + /** + * Rows of existing keys are kept, rows of new keys are inserted, a table without key is inserted + */ + @Test + void skipsExistingRows() throws Exception { + String source = newUrl(); + String target = newUrl(); + execute(source, "CREATE TABLE ITEMS (ID INT PRIMARY KEY, NOTE VARCHAR(20))", + "INSERT INTO ITEMS VALUES (1, 'new'), (2, 'b'), (3, 'c')", + "CREATE TABLE LOGS (NOTE VARCHAR(20))", "INSERT INTO LOGS VALUES ('x')"); + execute(target, "CREATE TABLE ITEMS (ID INT PRIMARY KEY, NOTE VARCHAR(20))", + "INSERT INTO ITEMS VALUES (1, 'old')", "CREATE TABLE LOGS (NOTE VARCHAR(20))", + "INSERT INTO LOGS VALUES ('x')"); + CollectingListener listener = new CollectingListener(); + ImportExporter importExporter = importer(source, target, WriteMode.SKIP_EXISTING); + importExporter.setExportListener(listener); + importExporter.exportDdlAndData(); + assertEquals(List.of(), listener.getErrors()); + assertEquals(List.of("old", "b", "c"), column(target, "SELECT NOTE FROM ITEMS ORDER BY ID")); + assertEquals(List.of("x", "x"), column(target, "SELECT NOTE FROM LOGS")); + // Chosen key columns: NOTE of the first row is new, the row is inserted and breaks the primary key + ImportExporter chosen = importer(source, target, WriteMode.SKIP_EXISTING); + chosen.getExportConfiguration().setIncludedTableNames(new String[]{"ITEMS"}); + chosen.getExportConfiguration().getKeyColumnNames().put("ITEMS", List.of("NOTE")); + chosen.getExportConfiguration().setFailFast(false); + listener = new CollectingListener(); + chosen.setExportListener(listener); + chosen.exportDdlAndData(); + assertEquals(1, listener.getErrors().size()); + assertTrue(listener.getErrors().get(0).startsWith("Skipped row of table 'ITEMS' (NOTE=new): "), + listener.getErrors().get(0)); + assertEquals(List.of("old", "b", "c"), column(target, "SELECT NOTE FROM ITEMS ORDER BY ID")); + } + + @Test + void describesRows() { + java.util.Map row = new java.util.LinkedHashMap<>(); + row.put("ID", 7); + row.put("DATA", new byte[3]); + row.put("NOTE", "x".repeat(80)); + assertEquals("id=7, data=<3 bytes>, NOTE=" + "x".repeat(47) + "...", ImportExportHandler.describeRow(row, + List.of("id", "data", "NOTE"))); + assertEquals("MISSING=null", ImportExportHandler.describeRow(row, List.of("MISSING"))); + } +} + diff --git a/src/test/java/com/github/sqljam/impexp/SqlScriptRunnerTest.java b/src/test/java/com/github/sqljam/impexp/SqlScriptRunnerTest.java index a026b15..5c3821a 100644 --- a/src/test/java/com/github/sqljam/impexp/SqlScriptRunnerTest.java +++ b/src/test/java/com/github/sqljam/impexp/SqlScriptRunnerTest.java @@ -212,4 +212,35 @@ private static int count(Connection connection) throws SQLException { return rs.getInt(1); } } + + /** + * Statements with errors are skipped in one transaction: the savepoint of a failed batch keeps the statements + * before it, each skipped statement is reported + */ + @Test + void skipsStatementsInTransaction() throws Exception { + String url = "jdbc:h2:mem:skipTx;DB_CLOSE_DELAY=-1"; + try (Connection setup = DriverManager.getConnection(url)) { + setup.createStatement().execute("CREATE TABLE s (id INT PRIMARY KEY)"); + } + try (com.github.sqljam.jdbc.ManualTransaction transaction = new com.github.sqljam.jdbc.ManualTransaction( + DriverManager.getConnection(url), false)) { + SqlScriptRunner runner = new SqlScriptRunner(transaction.getConnection()); + runner.setBatchSize(2); + runner.setStopOnError(false); + CollectingListener listener = new CollectingListener(); + runner.setExportListener(listener); + runner.runScript("INSERT INTO s VALUES (1);\nINSERT INTO s VALUES (2);\nINSERT INTO s VALUES (3);\n" + + "INSERT INTO s VALUES (1);\nINSERT INTO s VALUES (4);"); + assertFalse(transaction.isRolledBack()); + assertEquals(List.of(), transaction.commit()); + assertEquals(1, runner.getFailedCount()); + assertEquals(1, listener.getErrors().size()); + assertTrue(listener.getErrors().get(0).startsWith("Skipped sql: INSERT INTO s VALUES (1), cause: "), + listener.getErrors().get(0)); + } + try (Connection check = DriverManager.getConnection(url)) { + assertEquals(4, rows(check, "SELECT COUNT(*) FROM s")); + } + } } diff --git a/src/test/java/com/github/sqljam/it/AbstractParquetIT.java b/src/test/java/com/github/sqljam/it/AbstractParquetIT.java index 35e151b..5c7d476 100644 --- a/src/test/java/com/github/sqljam/it/AbstractParquetIT.java +++ b/src/test/java/com/github/sqljam/it/AbstractParquetIT.java @@ -20,15 +20,25 @@ import static org.junit.jupiter.api.Assumptions.assumeTrue; import java.io.File; +import java.nio.file.Files; import java.sql.Connection; import java.sql.ResultSet; import java.sql.Statement; +import java.util.ArrayList; +import java.util.Collections; import java.util.List; +import java.util.regex.Pattern; import org.apache.commons.io.FileUtils; import org.junit.jupiter.api.Test; import org.junit.jupiter.params.ParameterizedTest; import org.junit.jupiter.params.provider.EnumSource; +import com.github.sqljam.impexp.AbstractFileImporter; +import com.github.sqljam.impexp.CsvExporter; +import com.github.sqljam.impexp.CsvImporter; +import com.github.sqljam.impexp.CsvOptions; +import com.github.sqljam.impexp.DataFileExporter; +import com.github.sqljam.impexp.DataFileImporter; import com.github.sqljam.impexp.DataFormat; import com.github.sqljam.impexp.ExportManifest; import com.github.sqljam.impexp.ExportMode; @@ -37,14 +47,21 @@ import com.github.sqljam.impexp.ScriptExportHandler; /** - * @Description: AbstractParquetIT exports fixture tables of a source database as Parquet packages and imports them - * into every database, and loads Parquet files into new and existing tables + * @Description: AbstractParquetIT exports fixture tables of a source database as Parquet and CSV packages (split + * into parts) and imports them into every database, and loads Parquet and CSV files into new and + * existing tables * @Author: Fred Feng * @Date: 26/03/2023 * @Version 1.0.0 */ public abstract class AbstractParquetIT { + /** + * Small parts, so that tables of the fixture are split: emp.csv, emp_2.csv ... imported in order + */ + private static final long CSV_PART_SIZE = 8 * 1024; + private static final long PARQUET_PART_SIZE = 16 * 1024; + protected final ItDatabase source; protected AbstractParquetIT(ItDatabase source) { @@ -57,35 +74,64 @@ protected File getExportDirectory(String name) { return dir; } - private ParquetExporter newExporter(File dir, ItDatabase target) { - ParquetExporter parquetExporter = new ParquetExporter(dir); - source.configureSource(parquetExporter.getConfiguration()); - parquetExporter.getConfiguration().setIdReused(true); - parquetExporter.setTargetDbType(target.getDbType()); - parquetExporter.setTargetCatalogName(target.getTargetCatalog()); - parquetExporter.setTargetSchemaName(target.getTargetSchema()); - // Compressions are rotated over the matrix - parquetExporter.setCompression(ParquetExporter.COMPRESSIONS[(source.ordinal() + target.ordinal()) - % ParquetExporter.COMPRESSIONS.length]); - return parquetExporter; + private DataFileExporter newExporter(File dir, ItDatabase target, DataFormat format) { + int rotation = source.ordinal() + target.ordinal(); + DataFileExporter exporter; + if (format == DataFormat.CSV) { + CsvExporter csvExporter = new CsvExporter(dir); + // Options are rotated over the matrix: delimiter, encoding, quote, text of NULL and header + CsvOptions options = new CsvOptions(); + options.setDelimiter(CsvExporter.DELIMITERS[rotation % CsvExporter.DELIMITERS.length]); + // Encodings which hold the Chinese text of the fixture, others stop the export + String[] encodings = {"UTF-8", CsvOptions.UTF_8_BOM, "GBK", "GB18030", "UTF-16"}; + options.setEncoding(encodings[rotation % encodings.length]); + options.setQuote(CsvOptions.QUOTES[rotation % CsvOptions.QUOTES.length]); + options.setNullString(rotation % 3 == 0 ? "NULL" : ""); + options.setHeader(rotation % 4 != 1); + csvExporter.setOptions(options); + csvExporter.setMaxFileSize(CSV_PART_SIZE); + exporter = csvExporter; + } else { + ParquetExporter parquetExporter = new ParquetExporter(dir); + // Compressions are rotated over the matrix + parquetExporter.setCompression(ParquetExporter.COMPRESSIONS[rotation % ParquetExporter.COMPRESSIONS + .length]); + parquetExporter.setMaxFileSize(PARQUET_PART_SIZE); + exporter = parquetExporter; + } + source.configureSource(exporter.getConfiguration()); + exporter.getConfiguration().setIdReused(true); + exporter.setTargetDbType(target.getDbType()); + exporter.setTargetCatalogName(target.getTargetCatalog()); + exporter.setTargetSchemaName(target.getTargetSchema()); + return exporter; } @ParameterizedTest(name = "parquet for {0}") @EnumSource(ItDatabase.class) void exportAndImport(ItDatabase target) throws Exception { - // Oracle test user has only one schema + exportAndImport(target, DataFormat.PARQUET); + } + + @ParameterizedTest(name = "csv for {0}") + @EnumSource(ItDatabase.class) + void exportAndImportCsv(ItDatabase target) throws Exception { + exportAndImport(target, DataFormat.CSV); + } + + private void exportAndImport(ItDatabase target, DataFormat format) throws Exception { // Test users of Oracle and MariaDB have one schema, the same database type is verified by copyIntoSameSchema assumeTrue(!(source == target && source.isSingleSchema())); assumeTrue(AbstractImportIT.isTargetSelected(target)); assumeTrue(source.isAvailable() && target.isAvailable()); source.loadFixture(); - File dir = getExportDirectory(target.name()); - ParquetExporter parquetExporter = newExporter(dir, target); + File dir = getExportDirectory(format.name().toLowerCase() + "-" + target.name()); + DataFileExporter exporter = newExporter(dir, target, format); CollectingListener listener = new CollectingListener(); - parquetExporter.setExportListener(listener); + exporter.setExportListener(listener); try { - parquetExporter.exportDdlAndData(); + exporter.exportDdlAndData(); } catch (Exception e) { throw new AssertionError(e.getMessage() + "\nErrors: " + listener.getErrors(), e); } @@ -93,7 +139,7 @@ void exportAndImport(ItDatabase target) throws Exception { assertEquals(Boolean.TRUE, listener.getSuccessful()); ExportManifest manifest = ExportManifest.read(dir); - assertEquals(DataFormat.PARQUET, manifest.getDataFormat()); + assertEquals(format, manifest.getDataFormat()); assertEquals(ExportManifest.Status.COMPLETED, manifest.getStatus()); assertEquals(target.getDbType(), manifest.getTarget().getDbType()); assertEquals(5, manifest.getTables().size(), "Tables in manifest: " + manifest.getTables().size()); @@ -103,12 +149,32 @@ void exportAndImport(ItDatabase target) throws Exception { assertEquals(ItDatabase.EMP_COUNT, emp.getRows()); // ClickHouse has no identity columns assertEquals(source == ItDatabase.CLICKHOUSE ? 0 : 1, emp.getIdentityColumns().size(), "Identity column of emp"); - assertTrue(manifest.getFiles().stream().anyMatch(f -> f.getType() == ExportManifest.FileType.PARQUET)); + ExportManifest.FileType fileType = format == DataFormat.CSV ? ExportManifest.FileType.CSV + : ExportManifest.FileType.PARQUET; + assertTrue(manifest.getFiles().stream().anyMatch(f -> f.getType() == fileType)); assertEquals(ExportManifest.FileType.SCHEMA, manifest.getFiles().get(0).getType()); - assertTrue(new File(dir, emp.getDataFiles().iterator().next()).isFile(), "Parquet file of emp"); + List empFiles = List.copyOf(emp.getDataFiles()); + for (String path : empFiles) { + assertTrue(new File(dir, path).isFile(), "Data file of emp: " + path); + } + if (format == DataFormat.CSV) { + // Every column with its type, parts named like data files of sql scripts, each with the header + CsvOptions options = CsvOptions.fromMap(manifest.getOptions()); + if (options.isHeader() && options.isUtf8()) { + assertEquals(emp.getColumns().size(), countColumns(new File(dir, empFiles.get(0)))); + } + assertTrue(empFiles.size() > 1, "emp is split: " + empFiles); + assertTrue(empFiles.get(1).endsWith(source.getPrefix().toLowerCase() + "emp_2.csv") + || empFiles.get(1).toLowerCase().endsWith("emp_2.csv"), empFiles.toString()); + // Parts are split as UTF-8, UTF-16 doubles their size + for (String path : empFiles) { + assertTrue(new File(dir, path).length() <= (CSV_PART_SIZE + 4096) * 2, "Size of " + path); + } + } try (Connection connection = target.getTargetConnection()) { - ParquetImporter importer = new ParquetImporter(connection, target.getDbType()); + AbstractFileImporter importer = AbstractFileImporter.forDirectory(dir, connection, target.getDbType()); + assertEquals(format == DataFormat.CSV ? CsvImporter.class : ParquetImporter.class, importer.getClass()); listener = new CollectingListener(); importer.setExportListener(listener); try { @@ -122,40 +188,80 @@ void exportAndImport(ItDatabase target) throws Exception { } } + /** + * Fields of the header row of a CSV file + */ + private static int countColumns(File csvFile) throws Exception { + String header = Files.readAllLines(csvFile.toPath()).get(0); + for (String delimiter : CsvExporter.DELIMITERS) { + if (header.contains(delimiter)) { + return header.split(Pattern.quote(delimiter), -1).length; + } + } + return 1; + } + /** * Parquet files are loaded into a new table (created from the Parquet schema) and appended to it */ @Test void loadParquetFiles() throws Exception { + loadFiles(DataFormat.PARQUET); + } + + /** + * CSV files are loaded into a new table (delimiter and types detected) and appended to it, parts in order + */ + @Test + void loadCsvFiles() throws Exception { + loadFiles(DataFormat.CSV); + } + + private void loadFiles(DataFormat format) throws Exception { ItDatabase target = ItDatabase.H2; assumeTrue(source.isAvailable() && target.isAvailable()); source.loadFixture(); - File dir = getExportDirectory("files"); - ParquetExporter parquetExporter = newExporter(dir, target); - parquetExporter.export(ExportMode.DATA); + File dir = getExportDirectory(format.name().toLowerCase() + "-files"); + DataFileExporter exporter = newExporter(dir, target, format); + exporter.export(ExportMode.DATA); ExportManifest manifest = ExportManifest.read(dir); assertEquals(ExportMode.DATA, manifest.getExportMode()); ExportManifest.TableEntry emp = manifest.getTables().stream() .filter(table -> table.getName().equalsIgnoreCase(source.getPrefix() + "emp")).findFirst() .orElseThrow(); - File file = new File(dir, emp.getDataFiles().iterator().next()); - assertTrue(file.getParentFile().getName().equals(ScriptExportHandler.DATA_DIR_NAME)); + List files = new ArrayList<>(); + for (String path : emp.getDataFiles()) { + files.add(new File(dir, path)); + } + assertTrue(files.get(0).getParentFile().getName().equals(ScriptExportHandler.DATA_DIR_NAME)); + // The parts are given in any order, they are imported in the order of their numbers + Collections.reverse(files); - String tableName = source.getPrefix() + "pq_emp"; + String tableName = source.getPrefix() + (format == DataFormat.CSV ? "csv_emp" : "pq_emp"); try (Connection connection = target.getTargetConnection()) { - ParquetImporter importer = new ParquetImporter(connection, target.getDbType()); + DataFileImporter importer; + if (format == DataFormat.CSV) { + // The options of the files are chosen by the user, the delimiter is detected + CsvImporter csvImporter = new CsvImporter(connection, target.getDbType()); + CsvOptions options = CsvOptions.fromMap(manifest.getOptions()); + options.setDelimiter(""); + csvImporter.setOptions(options); + importer = csvImporter; + } else { + importer = new ParquetImporter(connection, target.getDbType()); + } importer.setTargetSchemaName(target.getTargetSchema()); CollectingListener listener = new CollectingListener(); importer.setExportListener(listener); - assertEquals(ItDatabase.EMP_COUNT, importer.importFiles(List.of(file), tableName, - ParquetImporter.Mode.CREATE)); + assertEquals(ItDatabase.EMP_COUNT, importer.importFiles(files, tableName, + DataFileImporter.Mode.CREATE)); assertEquals(List.of(), listener.getErrors()); assertTrue(listener.isCompleted(), "Import progress 100%"); // Rows are appended to the existing table listener = new CollectingListener(); importer.setExportListener(listener); - assertEquals(ItDatabase.EMP_COUNT, importer.importFiles(List.of(file), tableName, - ParquetImporter.Mode.APPEND)); + assertEquals(ItDatabase.EMP_COUNT, importer.importFiles(files, tableName, + DataFileImporter.Mode.APPEND)); assertEquals(List.of(), listener.getErrors()); assertTrue(listener.isCompleted(), "Append progress 100%"); try (Statement statement = connection.createStatement(); @@ -164,6 +270,18 @@ void loadParquetFiles() throws Exception { rs.next(); assertEquals(ItDatabase.EMP_COUNT * 2, rs.getLong(1)); } + if (format == DataFormat.CSV && files.size() > 1) { + // Rows of the parts keep the order of the source: the first part holds the first rows + try (Statement statement = connection.createStatement(); + ResultSet rs = statement.executeQuery("SELECT * FROM \"" + target.getTargetSchema() + "\".\"" + + tableName.toUpperCase() + "\" LIMIT 1")) { + assertTrue(rs.next()); + } + } + try (Statement statement = connection.createStatement()) { + statement.execute("DROP TABLE \"" + target.getTargetSchema() + "\".\"" + tableName.toUpperCase() + + "\""); + } } } } diff --git a/src/test/java/com/github/sqljam/it/ItVerifier.java b/src/test/java/com/github/sqljam/it/ItVerifier.java index faf33e8..25e6e7e 100644 --- a/src/test/java/com/github/sqljam/it/ItVerifier.java +++ b/src/test/java/com/github/sqljam/it/ItVerifier.java @@ -114,7 +114,7 @@ public String table(String name) { return tableName; } - private String ref(String name) { + public String ref(String name) { return dialect.getSourceTableName(catalog, schema, table(name)); } diff --git a/src/test/java/com/github/sqljam/it/ManualCommitBenchmark.java b/src/test/java/com/github/sqljam/it/ManualCommitBenchmark.java new file mode 100644 index 0000000..26da88c --- /dev/null +++ b/src/test/java/com/github/sqljam/it/ManualCommitBenchmark.java @@ -0,0 +1,199 @@ +/* + * Copyright 2023-2026 Fred Feng + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.github.sqljam.it; + +import java.io.File; +import java.nio.file.Files; +import java.sql.Connection; +import java.sql.DriverManager; +import java.sql.ResultSet; +import java.sql.Statement; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; +import java.util.Locale; + +import org.apache.commons.io.FileUtils; +import org.junit.jupiter.api.Test; +import com.github.sqljam.impexp.DbType; +import com.github.sqljam.impexp.ImportExporter; +import com.github.sqljam.jdbc.ManualTransaction; +import com.github.sqljam.jdbc.SharedConnectionFactory; + +/** + * @Description: ManualCommitBenchmark measures imports of many rows into each database: commits page by page, one + * manual transaction with its import and commit times, and the roll back of one transaction. It is + * not a test of the build (the name does not end with Test or IT), run it by + * mvn test -Dtest=ManualCommitBenchmark -Dbench.rows=1000000 -Dbench.targets=POSTGRESQL,MYSQL + * The results are written to target/bench/manual-commit.md. The table bench_orders is dropped from + * each database right after its measurement, the source database is deleted at the end. + * @Author: Fred Feng + * @Date: 26/03/2023 + * @Version 1.0.0 + */ +class ManualCommitBenchmark { + + private static final String TABLE = "BENCH_ORDERS"; + private static final File DIR = new File("target/bench"); + + /** + * A new source database with the rows, the source of an earlier run is deleted first + */ + private static String createSource(long rows) throws Exception { + File file = new File(DIR, "source"); + FileUtils.deleteQuietly(new File(file.getPath() + ".mv.db")); + String url = "jdbc:h2:file:" + file.getAbsolutePath(); + try (Connection connection = DriverManager.getConnection(url, "sa", ""); + Statement statement = connection.createStatement()) { + statement.execute("CREATE TABLE " + TABLE + " (ID BIGINT PRIMARY KEY, ORDER_NO VARCHAR(20) NOT NULL," + + " CUSTOMER_ID INT NOT NULL, STATUS VARCHAR(12) NOT NULL, TOTAL DECIMAL(12,2) NOT NULL," + + " NOTE VARCHAR(100), CREATED_AT TIMESTAMP NOT NULL)"); + statement.execute("INSERT INTO " + TABLE + " SELECT X, 'SO-' || LPAD(X, 12, '0'), MOD(X, 5000)," + + " CASEWHEN(MOD(X, 3) = 0, 'PAID', 'NEW'), MOD(X, 100000) / 100.0, REPEAT('n', MOD(X, 60))," + + " TIMESTAMP '2024-01-01 00:00:00' + X * INTERVAL '1' SECOND FROM SYSTEM_RANGE(1, " + rows + ")"); + } + return url; + } + + private static ImportExporter newImporter(String sourceUrl, ItDatabase target) { + ImportExporter importExporter = new ImportExporter(); + importExporter.getExportConfiguration().setDbType(DbType.H2); + importExporter.getExportConfiguration().setUrl(sourceUrl); + importExporter.getExportConfiguration().setUsername("sa"); + importExporter.getExportConfiguration().setPassword(""); + importExporter.getExportConfiguration().setIncludedSchemaNames(new String[]{"PUBLIC"}); + importExporter.getExportConfiguration().setIncludedTableNames(new String[]{TABLE}); + importExporter.getExportConfiguration().setIdReused(true); + importExporter.getExportConfiguration().setConnectionPoolEnabled(false); + target.configureTarget(importExporter.getImportConfiguration()); + return importExporter; + } + + /** + * Seconds of the import committed page by page, the default before manual commit + */ + private static double pageByPage(String sourceUrl, ItDatabase target) throws Exception { + ImportExporter importExporter = newImporter(sourceUrl, target); + importExporter.getImportConfiguration().setConnectionPoolEnabled(true); + long start = System.nanoTime(); + importExporter.exportDdlAndData(); + return (System.nanoTime() - start) / 1e9; + } + + /** + * Seconds of the import in one transaction and of its end: commit or roll back + */ + private static double[] oneTransaction(String sourceUrl, ItDatabase target, boolean commit) throws Exception { + try (ManualTransaction transaction = new ManualTransaction(target.getTargetConnection(), + target.getDbType().isTransactionalDdl())) { + ImportExporter importExporter = newImporter(sourceUrl, target); + importExporter.getExportConfiguration().setFailFast(true); + importExporter.setTargetConnectionFactory(new SharedConnectionFactory(transaction.getConnection())); + long start = System.nanoTime(); + importExporter.exportDdlAndData(); + double imported = (System.nanoTime() - start) / 1e9; + start = System.nanoTime(); + if (commit) { + List errors = transaction.commit(); + if (!errors.isEmpty()) { + throw new IllegalStateException("Deferred statements failed: " + errors); + } + } else { + transaction.rollback(); + } + return new double[]{imported, (System.nanoTime() - start) / 1e9}; + } + } + + /** + * The imported table in the target schema (or database) with the case of the target + */ + private static String targetTable(ItDatabase target) { + boolean upper = target.getDbType() == DbType.ORACLE || target.getDbType() == DbType.H2; + String table = upper ? TABLE : TABLE.toLowerCase(Locale.ENGLISH); + String owner = target.getDbType() == DbType.MYSQL || target.getDbType() == DbType.MARIADB + ? target.getTargetCatalog() : target.getTargetSchema(); + return owner != null ? owner + "." + table : table; + } + + private static long targetRows(ItDatabase target) { + try (Connection connection = target.getTargetConnection(); + Statement statement = connection.createStatement(); + ResultSet rs = statement.executeQuery("SELECT COUNT(*) FROM " + targetTable(target))) { + rs.next(); + return rs.getLong(1); + } catch (Exception e) { + return -1; + } + } + + private static void drop(ItDatabase target) { + try (Connection connection = target.getTargetConnection(); + Statement statement = connection.createStatement()) { + statement.execute("DROP TABLE " + targetTable(target)); + } catch (Exception ignored) { + // The table is not created + } + } + + @Test + void run() throws Exception { + long rows = Long.getLong("bench.rows", 1_000_000); + List names = Arrays.asList(System.getProperty("bench.targets", + "POSTGRESQL,MYSQL,MARIADB,ORACLE,SQLSERVER,H2").split(",")); + DIR.mkdirs(); + String sourceUrl = createSource(rows); + List lines = new ArrayList<>(); + lines.add(String.format(Locale.ENGLISH, "Rows: %,d (7 columns, about 100 bytes), page size 5000", rows)); + lines.add(""); + lines.add("| Target | Page by page | One transaction: import | Commit | Roll back after import | Roll back |"); + lines.add("|---|---|---|---|---|---|"); + for (String name : names) { + ItDatabase target = ItDatabase.valueOf(name.trim()); + if (!target.isAvailable()) { + lines.add("| " + target + " | unavailable | | | | |"); + continue; + } + String row; + try { + drop(target); + double pages = pageByPage(sourceUrl, target); + drop(target); + double[] committed = oneTransaction(sourceUrl, target, true); + long committedRows = targetRows(target); + drop(target); + double[] rolledBack = oneTransaction(sourceUrl, target, false); + long remaining = targetRows(target); + row = String.format(Locale.ENGLISH, "| %s | %.1f s | %.1f s | %.2f s | %.1f s | %.2f s |", target, + pages, committed[0], committed[1], rolledBack[0], rolledBack[1]); + if (committedRows != rows || remaining > 0) { + row += " rows after commit " + committedRows + ", after roll back " + remaining; + } + } catch (Exception e) { + row = "| " + target + " | failed: " + e.getMessage().replace('\n', ' ').replace('|', '/') + " | | | | |"; + } finally { + drop(target); + } + System.out.println(row); + lines.add(row); + } + Files.write(new File(DIR, "manual-commit.md").toPath(), lines); + lines.forEach(System.out::println); + // The source database is test data as well + FileUtils.deleteQuietly(new File(DIR, "source.mv.db")); + FileUtils.deleteQuietly(new File(DIR, "source.trace.db")); + } +} diff --git a/src/test/java/com/github/sqljam/it/ManualCommitIT.java b/src/test/java/com/github/sqljam/it/ManualCommitIT.java new file mode 100644 index 0000000..4986cb3 --- /dev/null +++ b/src/test/java/com/github/sqljam/it/ManualCommitIT.java @@ -0,0 +1,242 @@ +/* + * Copyright 2023-2026 Fred Feng + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.github.sqljam.it; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.junit.jupiter.api.Assumptions.assumeTrue; + +import java.io.File; +import java.sql.Connection; +import java.sql.SQLException; +import java.util.List; + +import org.apache.commons.io.FileUtils; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.EnumSource; +import com.github.sqljam.impexp.AbstractFileImporter; +import com.github.sqljam.impexp.CsvExporter; +import com.github.sqljam.impexp.DataFileStrategy; +import com.github.sqljam.impexp.DataFormat; +import com.github.sqljam.impexp.Exporter; +import com.github.sqljam.impexp.ImportExporter; +import com.github.sqljam.impexp.ScriptExporter; +import com.github.sqljam.jdbc.ManualTransaction; +import com.github.sqljam.jdbc.SharedConnectionFactory; + +/** + * @Description: ManualCommitIT imports the fixture tables of H2 into every database in one manual transaction: + * a roll back keeps nothing of the rows (databases with transactional DDL keep the tables as they + * were, others keep the created tables empty), a commit keeps everything and runs the deferred DDL + * (foreign keys, identity values) afterwards. ClickHouse has no transactions. + * @Author: Fred Feng + * @Date: 26/03/2023 + * @Version 1.0.0 + */ +class ManualCommitIT { + + private static final ItDatabase SOURCE = ItDatabase.H2; + private static final String[] TABLES = {"dept", "emp", "emp_tag", "sales", "types"}; + + /** + * Imports the fixture tables in the transaction, the transaction is still open afterwards + */ + private static ManualTransaction importInTransaction(ItDatabase target) throws Exception { + ManualTransaction transaction = new ManualTransaction(target.getTargetConnection(), + target.getDbType().isTransactionalDdl()); + ImportExporter importExporter = new ImportExporter(); + SOURCE.configureSource(importExporter.getExportConfiguration()); + importExporter.getExportConfiguration().setIdReused(true); + importExporter.getExportConfiguration().setPageSize(100); + importExporter.getExportConfiguration().setFailFast(true); + target.configureTarget(importExporter.getImportConfiguration()); + importExporter.getImportConfiguration().setConnectionPoolEnabled(false); + importExporter.setTargetConnectionFactory(new SharedConnectionFactory(transaction.getConnection())); + CollectingListener listener = new CollectingListener(); + importExporter.setExportListener(listener); + try { + importExporter.exportDdlAndData(); + } catch (Exception e) { + transaction.close(); + throw new AssertionError(e.getMessage() + "\nErrors: " + listener.getErrors(), e); + } + assertEquals(List.of(), listener.getErrors(), "Errors of " + SOURCE + " -> " + target); + assertFalse(transaction.isRolledBack()); + return transaction; + } + + /** + * Rows of the fixture tables in the target, -1 for a missing table + */ + private static long[] counts(ItDatabase target) throws SQLException { + long[] counts = new long[TABLES.length]; + try (Connection connection = target.getTargetConnection()) { + ItVerifier verifier = new ItVerifier(SOURCE, target, connection); + for (int i = 0; i < TABLES.length; i++) { + try { + counts[i] = verifier.count(TABLES[i]); + } catch (SQLException | RuntimeException e) { + counts[i] = -1; + } + } + } + return counts; + } + + @ParameterizedTest(name = "manual commit into {0}") + @EnumSource(ItDatabase.class) + void rollBackThenCommit(ItDatabase target) throws Exception { + assumeTrue(target.getDbType().isTransactionSupported()); + assumeTrue(AbstractImportIT.isTargetSelected(target)); + assumeTrue(SOURCE.isAvailable() && target.isAvailable()); + SOURCE.loadFixture(); + long[] before = counts(target); + + // Rolled back: nothing of the rows is kept + try (ManualTransaction transaction = importInTransaction(target)) { + if (!target.getDbType().isTransactionalDdl()) { + // Foreign keys and identity values wait for the commit, they would commit the rows + assertFalse(transaction.getDeferredStatements().isEmpty(), "Deferred statements of " + target); + } + transaction.rollback(); + assertTrue(transaction.isFinished()); + } + long[] afterRollback = counts(target); + for (int i = 0; i < TABLES.length; i++) { + if (target.getDbType().isTransactionalDdl()) { + assertEquals(before[i], afterRollback[i], TABLES[i] + " is as before the import in " + target); + } else { + assertEquals(0, afterRollback[i], TABLES[i] + " is created and empty in " + target); + } + } + + // Committed: rows, keys, identity values and the deferred statements + try (ManualTransaction transaction = importInTransaction(target)) { + assertEquals(List.of(), transaction.commit(), "Deferred statements of " + target); + assertThrows(IllegalStateException.class, transaction::rollback); + } + try (Connection connection = target.getTargetConnection()) { + new ItVerifier(SOURCE, target, connection).verifyAll(); + } + } + + /** + * Exports the fixture tables of H2 as a package for the target: sql scripts with LOB files, or CSV files + */ + private static File exportPackage(ItDatabase target, DataFormat format) throws Exception { + File dir = new File(ItDatabase.DATA_DIR, "manual-commit/" + format + "-" + target); + FileUtils.deleteQuietly(dir); + Exporter.ExportConfiguration configuration; + CollectingListener listener = new CollectingListener(); + if (format == DataFormat.CSV) { + CsvExporter exporter = new CsvExporter(dir); + exporter.setMaxFileSize(8 * 1024); + configuration = exporter.getConfiguration(); + SOURCE.configureSource(configuration); + configuration.setIdReused(true); + exporter.setTargetDbType(target.getDbType()); + exporter.setTargetCatalogName(target.getTargetCatalog()); + exporter.setTargetSchemaName(target.getTargetSchema()); + exporter.setExportListener(listener); + exporter.exportDdlAndData(); + } else { + ScriptExporter exporter = new ScriptExporter(dir, DataFileStrategy.SINGLE_FILE, 8 * 1024); + configuration = exporter.getConfiguration(); + SOURCE.configureSource(configuration); + configuration.setIdReused(true); + exporter.setTargetDbType(target.getDbType()); + exporter.setTargetCatalogName(target.getTargetCatalog()); + exporter.setTargetSchemaName(target.getTargetSchema()); + exporter.setExportListener(listener); + exporter.exportDdlAndData(); + } + assertEquals(List.of(), listener.getErrors()); + return dir; + } + + private static ManualTransaction importPackageInTransaction(ItDatabase target, File dir) throws Exception { + ManualTransaction transaction = new ManualTransaction(target.getTargetConnection(), + target.getDbType().isTransactionalDdl()); + AbstractFileImporter importer = AbstractFileImporter.forDirectory(dir, transaction.getConnection(), + target.getDbType()); + CollectingListener listener = new CollectingListener(); + importer.setExportListener(listener); + try { + importer.importDirectory(dir); + } catch (Exception e) { + transaction.close(); + throw new AssertionError(e.getMessage() + "\nErrors: " + listener.getErrors(), e); + } + assertEquals(List.of(), listener.getErrors(), "Errors of the package for " + target); + assertFalse(transaction.isRolledBack()); + return transaction; + } + + /** + * Packages of sql scripts (batches, LOB files) and of CSV files are imported in one transaction as well + */ + @ParameterizedTest(name = "manual commit of packages into {0}") + @EnumSource(ItDatabase.class) + void packageRollBackThenCommit(ItDatabase target) throws Exception { + assumeTrue(target.getDbType().isTransactionSupported()); + assumeTrue(AbstractImportIT.isTargetSelected(target)); + assumeTrue(SOURCE.isAvailable() && target.isAvailable()); + SOURCE.loadFixture(); + for (DataFormat format : new DataFormat[]{DataFormat.SQL, DataFormat.CSV}) { + File dir = exportPackage(target, format); + long[] before = counts(target); + try (ManualTransaction transaction = importPackageInTransaction(target, dir)) { + transaction.rollback(); + } + long[] afterRollback = counts(target); + for (int i = 0; i < TABLES.length; i++) { + if (target.getDbType().isTransactionalDdl()) { + assertEquals(before[i], afterRollback[i], format + " " + TABLES[i] + " as before in " + target); + } else { + assertEquals(0, afterRollback[i], format + " " + TABLES[i] + " is empty in " + target); + } + } + try (ManualTransaction transaction = importPackageInTransaction(target, dir)) { + assertEquals(List.of(), transaction.commit(), format + " deferred statements of " + target); + } + try (Connection connection = target.getTargetConnection()) { + ItVerifier verifier = new ItVerifier(SOURCE, target, connection); + if (format == DataFormat.CSV) { + verifier.withInstantEquality(); + } + verifier.verifyAll(); + } + } + } + + /** + * An open transaction is rolled back when it is closed, e.g. when the user closes the window + */ + @Test + void closeRollsBack() throws Exception { + ItDatabase target = ItDatabase.POSTGRESQL; + assumeTrue(SOURCE.isAvailable() && target.isAvailable()); + SOURCE.loadFixture(); + long[] before = counts(target); + ManualTransaction transaction = importInTransaction(target); + transaction.close(); + assertTrue(transaction.isFinished()); + assertEquals(before[1], counts(target)[1]); + } +} diff --git a/src/test/java/com/github/sqljam/it/UpsertIT.java b/src/test/java/com/github/sqljam/it/UpsertIT.java new file mode 100644 index 0000000..7634187 --- /dev/null +++ b/src/test/java/com/github/sqljam/it/UpsertIT.java @@ -0,0 +1,452 @@ +/* + * Copyright 2023-2026 Fred Feng + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.github.sqljam.it; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.junit.jupiter.api.Assumptions.assumeTrue; + +import java.io.File; +import java.sql.Connection; +import java.sql.DriverManager; +import java.sql.ResultSet; +import java.sql.Statement; +import java.util.List; +import java.util.UUID; + +import org.apache.commons.io.FileUtils; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.EnumSource; +import com.github.sqljam.impexp.AbstractFileImporter; +import com.github.sqljam.impexp.CsvExporter; +import com.github.sqljam.impexp.DataFileStrategy; +import com.github.sqljam.impexp.DataFormat; +import com.github.sqljam.impexp.DbType; +import com.github.sqljam.impexp.Exporter; +import com.github.sqljam.impexp.ImpExpException; +import com.github.sqljam.impexp.ImportExporter; +import com.github.sqljam.impexp.ScriptExporter; +import com.github.sqljam.impexp.WriteMode; +import com.github.sqljam.jdbc.ManualTransaction; +import com.github.sqljam.jdbc.SharedConnectionFactory; + +/** + * @Description: UpsertIT upserts the fixture tables of H2 into every database: rows changed and removed in the + * target are written again by their primary keys (composite keys included), kept tables, indexes and + * constraints are skipped. Direct imports, packages of sql scripts and packages of CSV files are + * verified, ClickHouse refuses upserts, a table without key is inserted and chosen key columns are + * used. + * @Author: Fred Feng + * @Date: 26/03/2023 + * @Version 1.0.0 + */ +class UpsertIT { + + private static final ItDatabase SOURCE = ItDatabase.H2; + + private static ImportExporter newImporter(ItDatabase target, WriteMode writeMode, boolean recreated) { + ImportExporter importExporter = new ImportExporter(); + SOURCE.configureSource(importExporter.getExportConfiguration()); + Exporter.ExportConfiguration source = importExporter.getExportConfiguration(); + source.setIdReused(true); + source.setPageSize(100); + source.setWriteMode(writeMode); + source.setTableRecreated(recreated); + target.configureTarget(importExporter.getImportConfiguration()); + return importExporter; + } + + private static void importDirectly(ItDatabase target, WriteMode writeMode, boolean recreated) throws Exception { + ImportExporter importExporter = newImporter(target, writeMode, recreated); + CollectingListener listener = new CollectingListener(); + importExporter.setExportListener(listener); + importExporter.exportDdlAndData(); + assertEquals(List.of(), listener.getErrors(), writeMode + " of " + SOURCE + " -> " + target); + } + + /** + * Rows of the target are changed and removed: notes of every employee, all tags and one sale + */ + private static void tamper(ItDatabase target) throws Exception { + try (Connection connection = target.getTargetConnection(); Statement statement = connection.createStatement()) { + ItVerifier verifier = new ItVerifier(SOURCE, target, connection); + statement.executeUpdate("UPDATE " + verifier.ref("emp") + " SET note = 'tampered'"); + statement.executeUpdate("DELETE FROM " + verifier.ref("emp_tag")); + statement.executeUpdate("DELETE FROM " + verifier.ref("sales") + " WHERE id = 1"); + // The row inserted by the verification of identity values + statement.executeUpdate("DELETE FROM " + verifier.ref("dept") + " WHERE name = 'New Dept'"); + if (!connection.getAutoCommit()) { + connection.commit(); + } + assertEquals(0, verifier.count("emp_tag")); + } + } + + private static void verify(ItDatabase target, boolean instantEquality) throws Exception { + try (Connection connection = target.getTargetConnection()) { + ItVerifier verifier = new ItVerifier(SOURCE, target, connection); + if (instantEquality) { + verifier.withInstantEquality(); + } + verifier.verifyAll(); + } + } + + @ParameterizedTest(name = "upsert into {0}") + @EnumSource(ItDatabase.class) + void upsertDirectly(ItDatabase target) throws Exception { + assumeTrue(target.getDbType().createDialect().isUpsertSupported()); + assumeTrue(AbstractImportIT.isTargetSelected(target)); + assumeTrue(SOURCE.isAvailable() && target.isAvailable()); + SOURCE.loadFixture(); + importDirectly(target, WriteMode.INSERT, true); + tamper(target); + // Kept tables: existing tables, indexes and constraints are skipped, rows are updated or inserted + importDirectly(target, WriteMode.UPSERT, false); + verify(target, false); + // Again with nothing changed but the verification row + tamper(target); + importDirectly(target, WriteMode.UPSERT, false); + verify(target, false); + } + + private static File exportPackage(ItDatabase target, DataFormat format) throws Exception { + return exportPackage(target, format, WriteMode.UPSERT); + } + + private static File exportPackage(ItDatabase target, DataFormat format, WriteMode writeMode) throws Exception { + File dir = new File(ItDatabase.DATA_DIR, "upsert/" + format + "-" + writeMode + "-" + target); + FileUtils.deleteQuietly(dir); + CollectingListener listener = new CollectingListener(); + Exporter.ExportConfiguration configuration; + if (format == DataFormat.CSV) { + CsvExporter exporter = new CsvExporter(dir); + configuration = exporter.getConfiguration(); + SOURCE.configureSource(configuration); + exporter.setTargetDbType(target.getDbType()); + exporter.setTargetCatalogName(target.getTargetCatalog()); + exporter.setTargetSchemaName(target.getTargetSchema()); + exporter.setExportListener(listener); + configure(configuration, writeMode); + exporter.exportDdlAndData(); + } else { + ScriptExporter exporter = new ScriptExporter(dir, DataFileStrategy.SINGLE_FILE, 8 * 1024); + configuration = exporter.getConfiguration(); + SOURCE.configureSource(configuration); + exporter.setTargetDbType(target.getDbType()); + exporter.setTargetCatalogName(target.getTargetCatalog()); + exporter.setTargetSchemaName(target.getTargetSchema()); + exporter.setExportListener(listener); + configure(configuration, writeMode); + exporter.exportDdlAndData(); + } + assertEquals(List.of(), listener.getErrors()); + return dir; + } + + private static void configure(Exporter.ExportConfiguration configuration, WriteMode writeMode) { + configuration.setIdReused(true); + configuration.setWriteMode(writeMode); + configuration.setTableRecreated(false); + } + + @ParameterizedTest(name = "upsert packages into {0}") + @EnumSource(ItDatabase.class) + void upsertPackages(ItDatabase target) throws Exception { + assumeTrue(target.getDbType().createDialect().isUpsertSupported()); + assumeTrue(AbstractImportIT.isTargetSelected(target)); + assumeTrue(SOURCE.isAvailable() && target.isAvailable()); + SOURCE.loadFixture(); + for (DataFormat format : new DataFormat[]{DataFormat.SQL, DataFormat.CSV}) { + File dir = exportPackage(target, format); + importDirectly(target, WriteMode.INSERT, true); + tamper(target); + try (Connection connection = target.getTargetConnection()) { + AbstractFileImporter importer = AbstractFileImporter.forDirectory(dir, connection, target.getDbType()); + CollectingListener listener = new CollectingListener(); + importer.setExportListener(listener); + importer.importDirectory(dir); + assertEquals(List.of(), listener.getErrors(), format + " package of " + target); + } + verify(target, format == DataFormat.CSV); + } + } + + /** + * Rows of the target are kept, missing rows are inserted: the notes of the employees stay tampered, removed + * tags and sales come back, by a direct import and by SQL and CSV packages + */ + @ParameterizedTest(name = "skip existing rows of {0}") + @EnumSource(ItDatabase.class) + void skipExistingRows(ItDatabase target) throws Exception { + assumeTrue(target.getDbType().createDialect().isUpsertSupported()); + assumeTrue(AbstractImportIT.isTargetSelected(target)); + assumeTrue(SOURCE.isAvailable() && target.isAvailable()); + SOURCE.loadFixture(); + for (DataFormat format : new DataFormat[]{null, DataFormat.SQL, DataFormat.CSV}) { + importDirectly(target, WriteMode.INSERT, true); + tamper(target); + if (format == null) { + importDirectly(target, WriteMode.SKIP_EXISTING, false); + } else { + File dir = exportPackage(target, format, WriteMode.SKIP_EXISTING); + try (Connection connection = target.getTargetConnection()) { + AbstractFileImporter importer = AbstractFileImporter.forDirectory(dir, connection, + target.getDbType()); + CollectingListener listener = new CollectingListener(); + importer.setExportListener(listener); + importer.importDirectory(dir); + assertEquals(List.of(), listener.getErrors(), format + " package of " + target); + } + } + try (Connection connection = target.getTargetConnection()) { + ItVerifier verifier = new ItVerifier(SOURCE, target, connection); + String label = (format == null ? "direct" : format) + " import into " + target; + assertEquals(ItDatabase.EMP_COUNT, verifier.count("emp"), label); + try (Statement statement = connection.createStatement(); ResultSet rs = statement.executeQuery( + "SELECT COUNT(*) FROM " + verifier.ref("emp") + " WHERE note = 'tampered'")) { + rs.next(); + assertEquals(ItDatabase.EMP_COUNT, rs.getLong(1), label); + } + try (Connection source = SOURCE.getSourceConnection()) { + ItVerifier sourceVerifier = new ItVerifier(SOURCE, SOURCE, source, "PUBLIC"); + for (String table : new String[]{"emp_tag", "sales"}) { + assertEquals(sourceVerifier.count(table), verifier.count(table), table + " of " + label); + } + } + } + } + } + + /** + * Rows of the same key exist already and fail: they are skipped and reported, the missing rows are written + * by commits every N rows and in one transaction. One transaction of DuckDB has no savepoints and is rolled + * back. + */ + @ParameterizedTest(name = "skip rows with errors of {0}") + @EnumSource(ItDatabase.class) + void skipRowsWithErrors(ItDatabase target) throws Exception { + assumeTrue(target.getDbType().isTransactionSupported()); + assumeTrue(AbstractImportIT.isTargetSelected(target)); + assumeTrue(SOURCE.isAvailable() && target.isAvailable()); + try (Connection source = SOURCE.getSourceConnection(); Statement statement = source.createStatement()) { + statement.execute("DROP TABLE IF EXISTS sjh_skip"); + statement.execute("CREATE TABLE sjh_skip (id INT PRIMARY KEY, note VARCHAR(20))"); + statement.execute("INSERT INTO sjh_skip SELECT X, 'row ' || X FROM SYSTEM_RANGE(1, 6)"); + } + ImportExporter create = newImporter(target, WriteMode.INSERT, true); + create.getExportConfiguration().setIncludedTableNames(new String[]{"SJH_SKIP"}); + create.setExportListener(new CollectingListener()); + create.exportDdlAndData(); + for (boolean oneTransaction : new boolean[]{false, true}) { + String label = (oneTransaction ? "one transaction of " : "commits every N rows of ") + target; + try (Connection connection = target.getTargetConnection(); Statement statement = connection + .createStatement()) { + ItVerifier verifier = new ItVerifier(SOURCE, target, connection); + statement.executeUpdate("DELETE FROM " + verifier.ref("skip") + " WHERE id IN (2, 4)"); + if (!connection.getAutoCommit()) { + connection.commit(); + } + } + ImportExporter importExporter = newImporter(target, WriteMode.INSERT, false); + importExporter.getExportConfiguration().setIncludedTableNames(new String[]{"SJH_SKIP"}); + importExporter.getExportConfiguration().setPageSize(2); + importExporter.getExportConfiguration().setFailFast(false); + CollectingListener listener = new CollectingListener(); + importExporter.setExportListener(listener); + boolean rolledBack = false; + if (oneTransaction) { + try (ManualTransaction transaction = new ManualTransaction(target.getTargetConnection(), + target.getDbType().isTransactionalDdl())) { + importExporter.setTargetConnectionFactory(new SharedConnectionFactory( + transaction.getConnection())); + importExporter.exportData(); + rolledBack = transaction.isRolledBack(); + if (!rolledBack) { + assertEquals(List.of(), transaction.commit(), label); + } + } + } else { + importExporter.exportData(); + } + try (Connection connection = target.getTargetConnection()) { + ItVerifier verifier = new ItVerifier(SOURCE, target, connection); + if (rolledBack) { + // Without savepoints the transaction can not go on (DuckDB) + assertEquals(DbType.DUCKDB, target.getDbType(), label); + assertEquals(4, verifier.count("skip"), label); + continue; + } + assertEquals(6, verifier.count("skip"), label); + } + List skipped = listener.getErrors(); + assertEquals(4, skipped.size(), label + ": " + skipped); + assertTrue(skipped.stream().allMatch(error -> error.toLowerCase().startsWith( + "skipped row of table 'sjh_skip' (id=")), label + ": " + skipped); + } + } + + /** + * The target table exists with names which differ in case from the source: the table, the columns and the + * keys are matched without case, rows are inserted and then upserted + */ + @ParameterizedTest(name = "names in other case of {0}") + @EnumSource(ItDatabase.class) + void matchesNamesWhichDifferInCase(ItDatabase target) throws Exception { + assumeTrue(target.getDbType().createDialect().isUpsertSupported()); + assumeTrue(AbstractImportIT.isTargetSelected(target)); + assumeTrue(SOURCE.isAvailable() && target.isAvailable()); + try (Connection source = SOURCE.getSourceConnection(); Statement statement = source.createStatement()) { + statement.execute("DROP TABLE IF EXISTS \"sjh_case\""); + statement.execute("CREATE TABLE \"sjh_case\" (\"OrderId\" INT PRIMARY KEY, \"customerName\" VARCHAR(20)," + + " \"amount\" INT)"); + statement.execute("INSERT INTO \"sjh_case\" VALUES (1, 'a', 10), (2, 'b', 20), (3, 'c', 30)"); + } + com.github.sqljam.impexp.Dialect dialect = target.getDbType().createDialect(); + String table = (target.getTargetSchema() != null && target.getDbType().isSchemaSupported() + ? quote(dialect, target.getTargetSchema()) + "." : "") + quote(dialect, "SJH_CASE"); + try (Connection connection = target.getTargetConnection(); Statement statement = connection.createStatement()) { + // Tables of both cases, e.g. a copy of the source table by an earlier run + String prefix = table.substring(0, table.length() - quote(dialect, "SJH_CASE").length()); + for (String name : new String[]{table, prefix + quote(dialect, "sjh_case")}) { + // DuckDB closes a statement when it fails + try (Statement drop = connection.createStatement()) { + drop.execute("DROP TABLE " + name); + } catch (Exception e) { + // The table does not exist + } + } + statement.execute("CREATE TABLE " + table + " (" + quote(dialect, "orderid") + " INT PRIMARY KEY, " + + quote(dialect, "CUSTOMERNAME") + " VARCHAR(20), " + quote(dialect, "Amount") + " INT)"); + if (!connection.getAutoCommit()) { + connection.commit(); + } + } + try { + for (WriteMode writeMode : new WriteMode[]{WriteMode.INSERT, WriteMode.UPSERT}) { + if (writeMode == WriteMode.UPSERT) { + try (Connection source = SOURCE.getSourceConnection(); + Statement statement = source.createStatement()) { + statement.execute("UPDATE \"sjh_case\" SET \"amount\" = \"amount\" + 1"); + statement.execute("INSERT INTO \"sjh_case\" VALUES (4, 'd', 40)"); + } + } + ImportExporter importExporter = newImporter(target, writeMode, false); + importExporter.getExportConfiguration().setIncludedTableNames(new String[]{"sjh_case"}); + CollectingListener listener = new CollectingListener(); + importExporter.setExportListener(listener); + importExporter.exportData(); + assertEquals(List.of(), listener.getErrors(), writeMode + " into " + target); + } + try (Connection connection = target.getTargetConnection(); Statement statement = connection + .createStatement(); ResultSet rs = statement.executeQuery("SELECT " + quote(dialect, "orderid") + + ", " + quote(dialect, "CUSTOMERNAME") + ", " + quote(dialect, "Amount") + " FROM " + table + + " ORDER BY 1")) { + List rows = new java.util.ArrayList<>(); + while (rs.next()) { + rows.add(rs.getInt(1) + " " + rs.getString(2) + " " + rs.getInt(3)); + } + assertEquals(List.of("1 a 11", "2 b 21", "3 c 31", "4 d 40"), rows, target.toString()); + } + } finally { + try (Connection connection = target.getTargetConnection(); Statement statement = connection + .createStatement()) { + statement.execute("DROP TABLE " + table); + if (!connection.getAutoCommit()) { + connection.commit(); + } + } + try (Connection source = SOURCE.getSourceConnection(); Statement statement = source.createStatement()) { + statement.execute("DROP TABLE IF EXISTS \"sjh_case\""); + } + } + } + + private static String quote(com.github.sqljam.impexp.Dialect dialect, String name) { + return dialect.getOpenQuote() + name + dialect.getCloseQuote(); + } + + @Test + void refusesWithoutUpsert() { + ItDatabase target = ItDatabase.CLICKHOUSE; + assumeTrue(SOURCE.isAvailable() && target.isAvailable()); + ImpExpException e = assertThrows(ImpExpException.class, () -> newImporter(target, WriteMode.UPSERT, false) + .exportDdlAndData()); + assertTrue(e.getMessage().contains("can not match rows by their key"), e.getMessage()); + } + + private static long count(String url, String sql) throws Exception { + try (Connection connection = DriverManager.getConnection(url, "sa", ""); + Statement statement = connection.createStatement(); ResultSet rs = statement.executeQuery(sql)) { + rs.next(); + return rs.getLong(1); + } + } + + /** + * Tables without primary key are inserted, chosen key columns identify the rows of other tables + */ + @Test + void usesChosenKeysAndInsertsTablesWithoutKey() throws Exception { + String source = "jdbc:h2:mem:up" + UUID.randomUUID().toString().replace("-", "") + ";DB_CLOSE_DELAY=-1"; + String target = "jdbc:h2:mem:up" + UUID.randomUUID().toString().replace("-", "") + ";DB_CLOSE_DELAY=-1"; + try (Connection connection = DriverManager.getConnection(source, "sa", ""); + Statement statement = connection.createStatement()) { + statement.execute("CREATE TABLE LOGS (MESSAGE VARCHAR(20))"); + statement.execute("INSERT INTO LOGS VALUES ('a'), ('b')"); + statement.execute("CREATE TABLE USERS (ID INT PRIMARY KEY, EMAIL VARCHAR(40), NAME VARCHAR(20))"); + statement.execute("INSERT INTO USERS VALUES (1, 'a@x.io', 'A'), (2, 'b@x.io', 'B')"); + } + try (Connection connection = DriverManager.getConnection(target, "sa", ""); + Statement statement = connection.createStatement()) { + // The same email with another id + statement.execute("CREATE TABLE USERS (ID INT, EMAIL VARCHAR(40), NAME VARCHAR(20))"); + statement.execute("INSERT INTO USERS VALUES (7, 'a@x.io', 'old')"); + } + for (int i = 0; i < 2; i++) { + ImportExporter importExporter = new ImportExporter(); + Exporter.ExportConfiguration configuration = importExporter.getExportConfiguration(); + configuration.setDbType(DbType.H2); + configuration.setUrl(source); + configuration.setUsername("sa"); + configuration.setPassword(""); + configuration.setIncludedSchemaNames(new String[]{"PUBLIC"}); + configuration.setIdReused(true); + configuration.setConnectionPoolEnabled(false); + configuration.setTableRecreated(false); + configuration.setWriteMode(WriteMode.UPSERT); + configuration.getKeyColumnNames().put("USERS", List.of("EMAIL")); + importExporter.getImportConfiguration().setDbType(DbType.H2); + importExporter.getImportConfiguration().setUrl(target); + importExporter.getImportConfiguration().setUsername("sa"); + importExporter.getImportConfiguration().setPassword(""); + importExporter.getImportConfiguration().setConnectionPoolEnabled(false); + CollectingListener listener = new CollectingListener(); + importExporter.setExportListener(listener); + importExporter.exportDdlAndData(); + assertEquals(List.of(), listener.getErrors()); + assertTrue(listener.getMessages().stream().anyMatch(message -> message.startsWith("Table LOGS has no " + + "primary key")), listener.getMessages().toString()); + } + // Inserted twice without key + assertEquals(4, count(target, "SELECT COUNT(*) FROM LOGS")); + // Matched by email: the row of a@x.io is updated with the source id, b@x.io is inserted once + assertEquals(2, count(target, "SELECT COUNT(*) FROM USERS")); + assertEquals(1, count(target, "SELECT COUNT(*) FROM USERS WHERE EMAIL = 'a@x.io' AND ID = 1 AND NAME = 'A'")); + } +} diff --git a/src/test/java/com/github/sqljam/jdbc/ManualTransactionTest.java b/src/test/java/com/github/sqljam/jdbc/ManualTransactionTest.java new file mode 100644 index 0000000..1abc17d --- /dev/null +++ b/src/test/java/com/github/sqljam/jdbc/ManualTransactionTest.java @@ -0,0 +1,228 @@ +/* + * Copyright 2023-2026 Fred Feng + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.github.sqljam.jdbc; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.io.File; +import java.sql.Connection; +import java.sql.DriverManager; +import java.sql.PreparedStatement; +import java.sql.ResultSet; +import java.sql.SQLException; +import java.sql.Statement; +import java.util.List; +import java.util.UUID; + +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +/** + * @Description: ManualTransactionTest verifies the transaction of a manual commit with in-memory databases: commits + * of importers are ignored, DDL after written rows is deferred on databases which commit around DDL + * (H2), a rollback of an importer rolls back everything, and the deferred statements after the commit + * @Author: Fred Feng + * @Date: 26/03/2023 + * @Version 1.0.0 + */ +class ManualTransactionTest { + + @TempDir + File dir; + + private static String url() { + return "jdbc:h2:mem:mt" + UUID.randomUUID().toString().replace("-", "") + ";DB_CLOSE_DELAY=-1"; + } + + private static long count(String url, String table) throws SQLException { + try (Connection connection = DriverManager.getConnection(url); + Statement statement = connection.createStatement(); + ResultSet rs = statement.executeQuery("SELECT COUNT(*) FROM " + table)) { + rs.next(); + return rs.getLong(1); + } + } + + @Test + void classifiesStatements() { + assertTrue(ManualTransaction.isDdl(" create table t (id int)")); + assertTrue(ManualTransaction.isDdl("ALTER TABLE t ADD CONSTRAINT fk FOREIGN KEY (a) REFERENCES p (id)")); + assertTrue(ManualTransaction.isDdl("COMMENT ON TABLE t IS 'x'")); + assertTrue(ManualTransaction.isDdl("BEGIN EXECUTE IMMEDIATE 'DROP TABLE T'; EXCEPTION WHEN OTHERS THEN " + + "NULL; END;")); + assertFalse(ManualTransaction.isDdl("INSERT INTO t VALUES (1)")); + assertFalse(ManualTransaction.isDdl("SET IDENTITY_INSERT t ON")); + assertTrue(ManualTransaction.isDml("insert into t values (1)")); + assertTrue(ManualTransaction.isDml("MERGE INTO t USING s ON (1 = 1)")); + // Batches of Oracle scripts + assertTrue(ManualTransaction.isDml("BEGIN\nINSERT INTO t VALUES (1);\nINSERT INTO t VALUES (2);\nEND;")); + assertFalse(ManualTransaction.isDml("SELECT setval('s', 10)")); + assertFalse(ManualTransaction.isDml("CREATE TABLE inserted (id int)")); + } + + @Test + void defersDdlAfterRowsAndCommits() throws Exception { + String url = url(); + try (ManualTransaction transaction = new ManualTransaction(DriverManager.getConnection(url), false)) { + Connection connection = transaction.getConnection(); + assertFalse(connection.getAutoCommit()); + // DDL before rows runs at once, it commits nothing of the import + try (Statement statement = connection.createStatement()) { + statement.execute("CREATE TABLE parent (id INT PRIMARY KEY)"); + statement.execute("CREATE TABLE child (id INT PRIMARY KEY, parent_id INT)"); + } + connection.setAutoCommit(true); + assertFalse(connection.getAutoCommit()); + try (PreparedStatement ps = connection.prepareStatement("INSERT INTO parent VALUES (?)")) { + ps.setInt(1, 1); + ps.executeUpdate(); + } + try (Statement statement = connection.createStatement()) { + statement.executeUpdate("INSERT INTO child VALUES (1, 1)"); + // Commits of importers are ignored + connection.commit(); + assertEquals(0, statement.executeUpdate("ALTER TABLE child ADD CONSTRAINT fk_child FOREIGN KEY " + + "(parent_id) REFERENCES parent (id)")); + statement.addBatch("CREATE INDEX idx_child ON child (parent_id)"); + assertSame(connection, statement.getConnection()); + } + connection.close(); + assertFalse(transaction.getConnection().isClosed()); + assertEquals(2, transaction.getDeferredStatements().size()); + // Nothing is visible to other connections before the commit + assertEquals(0, count(url, "parent")); + assertEquals(List.of(), transaction.commit()); + assertTrue(transaction.isFinished()); + assertThrows(IllegalStateException.class, transaction::commit); + } + assertEquals(1, count(url, "parent")); + assertEquals(1, count(url, "child")); + try (Connection connection = DriverManager.getConnection(url); + Statement statement = connection.createStatement(); + ResultSet rs = statement.executeQuery("SELECT COUNT(*) FROM INFORMATION_SCHEMA.TABLE_CONSTRAINTS " + + "WHERE CONSTRAINT_NAME = 'FK_CHILD'")) { + rs.next(); + assertEquals(1, rs.getLong(1)); + } + } + + private static void assertSame(Connection expected, Connection actual) { + assertTrue(expected == actual, "Statements return the transaction connection"); + } + + @Test + void rollbackOfImporterRollsBackEverything() throws Exception { + String url = url(); + try (Connection setup = DriverManager.getConnection(url); Statement statement = setup.createStatement()) { + statement.execute("CREATE TABLE t (id INT PRIMARY KEY)"); + } + try (ManualTransaction transaction = new ManualTransaction(DriverManager.getConnection(url), true)) { + Connection connection = transaction.getConnection(); + try (Statement statement = connection.createStatement()) { + statement.executeUpdate("INSERT INTO t VALUES (1)"); + // Commit of a batch, then a failed batch is rolled back by the importer + connection.commit(); + statement.executeUpdate("INSERT INTO t VALUES (2)"); + connection.rollback(); + } + assertTrue(transaction.isRolledBack()); + transaction.rollback(); + assertThrows(IllegalStateException.class, transaction::rollback); + } + assertEquals(0, count(url, "t")); + } + + /** + * DDL of databases with transactional DDL is part of the transaction and rolled back with it + */ + @Test + void keepsDdlInTransaction() throws Exception { + String url = "jdbc:duckdb:" + new File(dir, "manual.duckdb").getAbsolutePath(); + try (ManualTransaction transaction = new ManualTransaction(DriverManager.getConnection(url), true)) { + try (Statement statement = transaction.getConnection().createStatement()) { + statement.execute("CREATE TABLE t (id INT)"); + statement.executeUpdate("INSERT INTO t VALUES (1)"); + statement.execute("CREATE INDEX idx_t ON t (id)"); + } + assertEquals(List.of(), transaction.getDeferredStatements()); + transaction.rollback(); + } + try (Connection connection = DriverManager.getConnection(url); + Statement statement = connection.createStatement(); + ResultSet rs = statement.executeQuery("SELECT COUNT(*) FROM duckdb_tables() WHERE table_name = 't'")) { + rs.next(); + assertEquals(0, rs.getLong(1)); + } + } + + @Test + void reportsFailedDeferredStatements() throws Exception { + String url = url(); + try (ManualTransaction transaction = new ManualTransaction(DriverManager.getConnection(url), false)) { + Connection connection = transaction.getConnection(); + try (Statement statement = connection.createStatement()) { + statement.execute("CREATE TABLE t (id INT)"); + statement.executeUpdate("INSERT INTO t VALUES (1)"); + statement.execute("ALTER TABLE missing ADD COLUMN x INT"); + statement.execute("CREATE INDEX idx_t ON t (id)"); + } + List errors = transaction.commit(); + // The failure does not stop the next statement, the rows are committed + assertEquals(1, errors.size()); + assertTrue(errors.get(0).startsWith("ALTER TABLE missing"), errors.toString()); + assertTrue(transaction.toString().contains("deferred=2")); + } + assertEquals(1, count(url, "t")); + } + + @Test + void closeRollsBackOpenTransaction() throws Exception { + String url = url(); + try (Connection setup = DriverManager.getConnection(url); Statement statement = setup.createStatement()) { + statement.execute("CREATE TABLE t (id INT)"); + } + ManualTransaction transaction = new ManualTransaction(DriverManager.getConnection(url), true); + assertTrue(transaction.isTransactionalDdl()); + try (Statement statement = transaction.getConnection().createStatement()) { + statement.executeUpdate("INSERT INTO t VALUES (1)"); + } + assertSame(transaction.getConnection(), transaction.getConnection().unwrap(Connection.class)); + transaction.close(); + assertTrue(transaction.isFinished()); + assertEquals(0, count(url, "t")); + // Closing twice does nothing + transaction.close(); + } + + /** + * The connection of a transaction is known, also when a connection factory shares it + */ + @Test + void knowsConnectionsOfTransactions() throws Exception { + String url = url(); + try (Connection plain = DriverManager.getConnection(url); + ManualTransaction transaction = new ManualTransaction(DriverManager.getConnection(url), false)) { + assertTrue(ManualTransaction.isManaged(transaction.getConnection())); + assertTrue(ManualTransaction.isManaged(new SharedConnectionFactory(transaction.getConnection()) + .getConnection())); + assertFalse(ManualTransaction.isManaged(plain)); + assertFalse(ManualTransaction.isManaged(new SharedConnectionFactory(plain).getConnection())); + } + } +} diff --git a/target/maven-status/maven-compiler-plugin/compile/default-compile/createdFiles.lst b/target/maven-status/maven-compiler-plugin/compile/default-compile/createdFiles.lst index 95a8bf3..fcf356c 100644 --- a/target/maven-status/maven-compiler-plugin/compile/default-compile/createdFiles.lst +++ b/target/maven-status/maven-compiler-plugin/compile/default-compile/createdFiles.lst @@ -37,9 +37,9 @@ com/github/sqljam/page/SimplePageResponse$PageIterator.class com/github/sqljam/impexp/DuckDBWorkspace.class com/github/sqljam/page/PageResponse.class com/github/sqljam/face/view/ConnectionDialog.class -com/github/sqljam/impexp/ParquetImporter$Mode.class com/github/sqljam/face/service/ConnectionStore.class com/github/sqljam/face/view/ExportDialog$1.class +com/github/sqljam/impexp/CsvImporter.class com/github/sqljam/impexp/ExportHandler.class com/github/sqljam/impexp/db/MariaDBMetaDataOperations.class com/github/sqljam/face/view/ExportDialog$2.class @@ -53,6 +53,7 @@ com/github/sqljam/impexp/DbType$3.class com/github/sqljam/face/view/ProgressDialog$2.class com/github/sqljam/impexp/DbType$5.class com/github/sqljam/impexp/DataFileStrategy.class +com/github/sqljam/impexp/DataFileExporter.class com/github/sqljam/face/service/DdlGenerator$CapturingHandler.class com/github/sqljam/impexp/db/DuckDBDialect.class com/github/sqljam/face/view/ThemeManager.class @@ -62,6 +63,7 @@ com/github/sqljam/page/ReadonlyPage.class com/github/sqljam/impexp/db/Oracle12cDialect.class com/github/sqljam/jdbc/DataAccessException.class com/github/sqljam/face/view/ImportPackageDialog$ImportRequest.class +com/github/sqljam/jdbc/ManualTransaction$Managed.class com/github/sqljam/jdbc/JdbcUtils$MapCursor.class com/github/sqljam/face/view/SplashView.class com/github/sqljam/impexp/SqlScriptRunner.class @@ -71,6 +73,7 @@ com/github/sqljam/face/view/ProfileRegistry.class com/github/sqljam/face/view/ExportDialog$TableChoice.class com/github/sqljam/page/DefaultPageContent.class com/github/sqljam/impexp/db/SQLiteMetaDataOperations.class +com/github/sqljam/impexp/DataFileImporter.class com/github/sqljam/impexp/db/MySQLMetaDataOperations.class com/github/sqljam/impexp/ExportManifest.class com/github/sqljam/impexp/DbType$8.class @@ -80,11 +83,12 @@ com/github/sqljam/impexp/DdlScripter.class com/github/sqljam/impexp/TransactionIsolationLevel.class com/github/sqljam/face/view/ConnectionTree.class com/github/sqljam/impexp/db/SQLServer2008Dialect.class +com/github/sqljam/impexp/ImportExporter$WrittenRowsListener.class com/github/sqljam/face/view/DbTypeCells$2.class com/github/sqljam/face/view/AppContext.class com/github/sqljam/jdbc/JdbcUtils.class com/github/sqljam/impexp/ParquetImporter.class -com/github/sqljam/impexp/ParquetImporter$ProgressListener.class +com/github/sqljam/impexp/DataFileExporter$PhaseListener.class com/github/sqljam/impexp/PartitionMetaData.class com/github/sqljam/impexp/AbstractFileImporter$ForwardingListener.class com/github/sqljam/face/SplashPreloader$Ready.class @@ -97,7 +101,6 @@ com/github/sqljam/face/service/DatabaseSession.class com/github/sqljam/face/view/ImportParquetDialog$1.class com/github/sqljam/impexp/CatalogMetaData.class com/github/sqljam/impexp/PooledConnectionFactory.class -com/github/sqljam/impexp/ParquetExporter$PhaseListener$1.class com/github/sqljam/impexp/TableQuery.class com/github/sqljam/impexp/CommentMetaData.class com/github/sqljam/impexp/DbType$7.class @@ -117,8 +120,10 @@ com/github/sqljam/jdbc/ConnectionFactory.class com/github/sqljam/face/view/DataPane.class com/github/sqljam/page/PageRequest.class com/github/sqljam/impexp/ExportManifest$FileEntry.class +com/github/sqljam/face/view/TransactionOption.class com/github/sqljam/face/view/ConnectionTree$DbNodeCell.class com/github/sqljam/impexp/ImportExportHandler$ImportConfiguration.class +com/github/sqljam/impexp/CsvOptions.class com/github/sqljam/jdbc/Cursor.class com/github/sqljam/impexp/TypeNames.class com/github/sqljam/face/view/Icons.class @@ -137,20 +142,23 @@ com/github/sqljam/impexp/StringHelper.class com/github/sqljam/impexp/AbstractFileImporter$1.class com/github/sqljam/face/model/TransferRequest$Target.class com/github/sqljam/impexp/LobManifestWriter.class +com/github/sqljam/face/view/CsvOptionsPane$1.class com/github/sqljam/face/view/ProgressDialog.class com/github/sqljam/page/PageReader.class -com/github/sqljam/impexp/ParquetExporter$PhaseListener.class com/github/sqljam/impexp/db/PostgreSQL10Dialect.class com/github/sqljam/impexp/DbType$1.class com/github/sqljam/face/view/ImportParquetDialog$ParquetRequest.class com/github/sqljam/face/SqlJamApplication.class com/github/sqljam/impexp/db/SQLiteDialect.class +com/github/sqljam/impexp/CsvSplitter.class com/github/sqljam/face/view/Dialogs.class com/github/sqljam/face/view/DbTypeCells$1.class +com/github/sqljam/impexp/DataFileImporter$ProgressListener.class com/github/sqljam/face/view/ImportParquetDialog.class com/github/sqljam/utils/KeyConvertibleMap.class com/github/sqljam/face/Banner.class com/github/sqljam/impexp/ExportListener$1.class +com/github/sqljam/impexp/ExportManifest$ColumnEntry.class com/github/sqljam/face/service/DatabaseSession$1.class com/github/sqljam/impexp/ExportCancelledException.class com/github/sqljam/face/JvmOptions.class @@ -162,6 +170,7 @@ com/github/sqljam/impexp/db/MySQLDialect.class com/github/sqljam/face/model/ConnectionProfile.class com/github/sqljam/face/service/JsonStore.class com/github/sqljam/impexp/IndexMetaData.class +com/github/sqljam/impexp/DataFileImporter$1.class com/github/sqljam/face/view/DbTypeCells.class com/github/sqljam/jdbc/SharedConnectionFactory.class com/github/sqljam/impexp/ColumnMetaData.class @@ -173,25 +182,33 @@ com/github/sqljam/utils/ConvertUtils.class com/github/sqljam/face/view/ImportPackageDialog.class com/github/sqljam/impexp/db/ClickHouseDialect.class com/github/sqljam/impexp/MetaData.class +com/github/sqljam/impexp/DataFileExporter$PhaseListener$1.class com/github/sqljam/impexp/DbType$2.class com/github/sqljam/face/model/TransferRequest.class com/github/sqljam/impexp/db/MariaDBDialect.class +com/github/sqljam/face/view/KeyColumnsDialog.class com/github/sqljam/impexp/MetaDataOperations.class +com/github/sqljam/impexp/CsvExporter.class com/github/sqljam/impexp/ExportManifest$Database.class com/github/sqljam/impexp/MetaDataVisitor.class +com/github/sqljam/impexp/ImportExportHandler$RowBinder.class com/github/sqljam/impexp/TableMetaData.class com/github/sqljam/impexp/db/H2MetaDataOperations.class com/github/sqljam/page/PageContent.class com/github/sqljam/impexp/ExportManifest$Status.class com/github/sqljam/impexp/db/DuckDBMetaDataOperations.class com/github/sqljam/impexp/SequenceMetaData.class +com/github/sqljam/impexp/DbType$10.class com/github/sqljam/impexp/db/MySQL55Dialect.class +com/github/sqljam/impexp/DataFileImporter$Mode.class com/github/sqljam/impexp/db/H2Dialect.class com/github/sqljam/impexp/DataFormat.class +com/github/sqljam/jdbc/ManualTransaction.class com/github/sqljam/impexp/DdlScripter$Schema.class com/github/sqljam/impexp/Exporter.class com/github/sqljam/page/SimplePageResponse.class com/github/sqljam/impexp/db/PostgreSQL9Dialect.class +com/github/sqljam/impexp/WriteMode.class com/github/sqljam/face/view/DbNode.class com/github/sqljam/face/SplashPreloader.class com/github/sqljam/impexp/DdlScripter$PartitionTable.class @@ -201,8 +218,8 @@ com/github/sqljam/impexp/DdlScripter$Catalog.class com/github/sqljam/jdbc/JdbcUtils$ObjectCursor.class com/github/sqljam/utils/Observer.class com/github/sqljam/config/Config.class +com/github/sqljam/face/view/CsvOptionsPane.class com/github/sqljam/impexp/ExportListener.class -com/github/sqljam/impexp/ParquetImporter$1.class com/github/sqljam/impexp/ScriptExporter.class com/github/sqljam/impexp/Dialect.class com/github/sqljam/jdbc/PreparedStatementCallback.class diff --git a/target/maven-status/maven-compiler-plugin/compile/default-compile/inputFiles.lst b/target/maven-status/maven-compiler-plugin/compile/default-compile/inputFiles.lst index f514284..8cfd97f 100644 --- a/target/maven-status/maven-compiler-plugin/compile/default-compile/inputFiles.lst +++ b/target/maven-status/maven-compiler-plugin/compile/default-compile/inputFiles.lst @@ -24,6 +24,7 @@ /Users/fred/work/repo/sqljam/src/main/java/com/github/sqljam/face/view/ConnectionDialog.java /Users/fred/work/repo/sqljam/src/main/java/com/github/sqljam/face/view/ConnectionForm.java /Users/fred/work/repo/sqljam/src/main/java/com/github/sqljam/face/view/ConnectionTree.java +/Users/fred/work/repo/sqljam/src/main/java/com/github/sqljam/face/view/CsvOptionsPane.java /Users/fred/work/repo/sqljam/src/main/java/com/github/sqljam/face/view/DataPane.java /Users/fred/work/repo/sqljam/src/main/java/com/github/sqljam/face/view/DbNode.java /Users/fred/work/repo/sqljam/src/main/java/com/github/sqljam/face/view/DbTypeCells.java @@ -33,6 +34,7 @@ /Users/fred/work/repo/sqljam/src/main/java/com/github/sqljam/face/view/Icons.java /Users/fred/work/repo/sqljam/src/main/java/com/github/sqljam/face/view/ImportPackageDialog.java /Users/fred/work/repo/sqljam/src/main/java/com/github/sqljam/face/view/ImportParquetDialog.java +/Users/fred/work/repo/sqljam/src/main/java/com/github/sqljam/face/view/KeyColumnsDialog.java /Users/fred/work/repo/sqljam/src/main/java/com/github/sqljam/face/view/LoginView.java /Users/fred/work/repo/sqljam/src/main/java/com/github/sqljam/face/view/MainView.java /Users/fred/work/repo/sqljam/src/main/java/com/github/sqljam/face/view/Messages.java @@ -44,11 +46,18 @@ /Users/fred/work/repo/sqljam/src/main/java/com/github/sqljam/face/view/TableTab.java /Users/fred/work/repo/sqljam/src/main/java/com/github/sqljam/face/view/TaskRunner.java /Users/fred/work/repo/sqljam/src/main/java/com/github/sqljam/face/view/ThemeManager.java +/Users/fred/work/repo/sqljam/src/main/java/com/github/sqljam/face/view/TransactionOption.java /Users/fred/work/repo/sqljam/src/main/java/com/github/sqljam/impexp/AbstractFileImporter.java /Users/fred/work/repo/sqljam/src/main/java/com/github/sqljam/impexp/CatalogMetaData.java /Users/fred/work/repo/sqljam/src/main/java/com/github/sqljam/impexp/ColumnMetaData.java /Users/fred/work/repo/sqljam/src/main/java/com/github/sqljam/impexp/CombinedIndexMetaData.java /Users/fred/work/repo/sqljam/src/main/java/com/github/sqljam/impexp/CommentMetaData.java +/Users/fred/work/repo/sqljam/src/main/java/com/github/sqljam/impexp/CsvExporter.java +/Users/fred/work/repo/sqljam/src/main/java/com/github/sqljam/impexp/CsvImporter.java +/Users/fred/work/repo/sqljam/src/main/java/com/github/sqljam/impexp/CsvOptions.java +/Users/fred/work/repo/sqljam/src/main/java/com/github/sqljam/impexp/CsvSplitter.java +/Users/fred/work/repo/sqljam/src/main/java/com/github/sqljam/impexp/DataFileExporter.java +/Users/fred/work/repo/sqljam/src/main/java/com/github/sqljam/impexp/DataFileImporter.java /Users/fred/work/repo/sqljam/src/main/java/com/github/sqljam/impexp/DataFileStrategy.java /Users/fred/work/repo/sqljam/src/main/java/com/github/sqljam/impexp/DataFormat.java /Users/fred/work/repo/sqljam/src/main/java/com/github/sqljam/impexp/DbCategory.java @@ -97,6 +106,7 @@ /Users/fred/work/repo/sqljam/src/main/java/com/github/sqljam/impexp/TransactionIsolationLevel.java /Users/fred/work/repo/sqljam/src/main/java/com/github/sqljam/impexp/TypeMappingException.java /Users/fred/work/repo/sqljam/src/main/java/com/github/sqljam/impexp/TypeNames.java +/Users/fred/work/repo/sqljam/src/main/java/com/github/sqljam/impexp/WriteMode.java /Users/fred/work/repo/sqljam/src/main/java/com/github/sqljam/impexp/db/ClickHouseDialect.java /Users/fred/work/repo/sqljam/src/main/java/com/github/sqljam/impexp/db/ClickHouseMetaDataOperations.java /Users/fred/work/repo/sqljam/src/main/java/com/github/sqljam/impexp/db/DuckDBDialect.java @@ -127,6 +137,7 @@ /Users/fred/work/repo/sqljam/src/main/java/com/github/sqljam/jdbc/Cursor.java /Users/fred/work/repo/sqljam/src/main/java/com/github/sqljam/jdbc/DataAccessException.java /Users/fred/work/repo/sqljam/src/main/java/com/github/sqljam/jdbc/JdbcUtils.java +/Users/fred/work/repo/sqljam/src/main/java/com/github/sqljam/jdbc/ManualTransaction.java /Users/fred/work/repo/sqljam/src/main/java/com/github/sqljam/jdbc/PreparedStatementCallback.java /Users/fred/work/repo/sqljam/src/main/java/com/github/sqljam/jdbc/SharedConnectionFactory.java /Users/fred/work/repo/sqljam/src/main/java/com/github/sqljam/jdbc/SimpleConnectionFactory.java