Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@
- **Linux/macOS SSH Tunnel**:跨平台連線設定可勾選「透過 SSH Tunnel 連線」,填入 SSH 主機、port、使用者、密碼或 OpenSSH/PEM 私鑰(含密語)與必填的主機金鑰 SHA256 指紋;連線時以 SSH.NET 建立只綁定 127.0.0.1 動態 port 的本機轉送,MySQL/MariaDB、PostgreSQL、SQL Server 的所有查詢與資料表操作都改走轉送端點,session 中斷或程式關閉時一併關閉;SSH 連線被跳板機中斷(閒置逾時、重啟、網路中斷)後,下一次資料庫操作會以同樣的指紋規則自動重建轉送,而不是把死掉的端點交給驅動程式。主機金鑰與指紋完全比對、不符即中止並回報伺服器實際指紋;MD5 或格式錯誤的指紋、缺主機/使用者/指紋、非絕對路徑私鑰、私鑰權限過寬與 MySQL/PostgreSQL 的 VerifyFull(127.0.0.1 主機名稱必定不符)一律 fail closed,SQL Server 則改以原始主機名稱驗證憑證。SSH 密碼與私鑰密語不寫入 `connections.json`(設定檔含這兩個欄位會拒絕載入),勾選系統密碼庫時與資料庫密碼一樣以獨立項目存入 Linux Secret Service/macOS Keychain,否則只保留到本次程式關閉;smoke test 會以本機臨時 sshd 實機驗證金鑰登入、加密私鑰、錯誤密語、指紋不符、權限過寬與 provider session 整合,實機矩陣另會經 tunnel 對 MySQL 8/MariaDB 11.4(Required)、PostgreSQL 16(VerifyCA+CA)與 SQL Server 2022(Mandatory+伺服器憑證,以原始主機名稱驗證)實際查詢與列出資料庫。

### 🛠️ 問題修正與優化
- PostgreSQL 邏輯搬移保留並校驗主鍵 constraint/索引名稱;跨 SQLite 無法保留的自訂名稱與 PostgreSQL 目標名稱碰撞改在預檢阻擋,避免已驗證卻靜默更名。
- **PostgreSQL 索引鍵預檢**:讀取實際 block_size,標準 8 KiB page 下逐列檢查 PK 與普通索引的保守 2048-byte tuple 預算,包含 UTF-8、複合鍵與對齊成本。超限在確認前拒絕,不依賴壓縮或等 INSERT 才失敗;未索引文字仍保留原欄位限額。
- **SQL Server 搬移參數**:TimeSpan 不再被 DbType.Time 轉成 DateTime,NULL time 明確保留 SQL time;binary 欄位的 NULL 不再推導成 nvarchar。每批資料參數改為 2098,保留 RPC 額度。新增正式來源 SQL Server 2019 ↔ 2022 隔離雙向回歸與末批失敗/安全重試測試,不宣稱 PK/FK/index/identity 搬移或所有版本已驗證。
- 邏輯搬移先取得 byte length 與 SQL 限額投影再讀取 BLOB/文字,避免超過 1 MiB 才拒絕時已配置整個大陣列;單欄與累計預算在 materialize 前檢查,超限保持空目標。
Expand Down
2 changes: 1 addition & 1 deletion docs/LOGICAL_MIGRATION_MATRIX.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
實作適用明確手動給 ID 的業務表:例如 `customers(id INTEGER NOT NULL, label TEXT COLLATE BINARY NOT NULL, note TEXT, payload BLOB, PRIMARY KEY(id)) STRICT, WITHOUT ROWID`,可帶 `customer_label(label)`、`customer_note(note,id)` 普通索引;複合主鍵如 `lines(customer,code)` 同樣保留順序、唯一性及 NOT NULL。所有索引 DDL 都列在確認預覽,建立後正式 catalog 再讀回驗證,缺索引/改欄位順序也會整批回滾。

- SQLite 文字只接受 BINARY(隱含或明確),PostgreSQL 必須是內建 `pg_catalog."C"`;兩端 UTF-8,逐欄嚴格解碼,拒絕 NUL、無效 UTF-8,不偷偷替換字元。保留空字串、NULL、Unicode、大小寫與 byte-order 比較語意;不承諾兩引擎字串函式或 LIKE 行為相同。
- SQLite PK 表要求完整可辨識 grammar、明確 NOT NULL 與 WITHOUT ROWID,避免 INTEGER PRIMARY KEY 的 rowid alias 自動給 ID 行為被偷偷改成 PostgreSQL 無 sequence 的 bigint。單欄整數/文字/二進位與複合 PK 均可;不保證引擎自動產生的 PK constraint/內部索引名稱或實體配置相同。
- SQLite PK 表要求完整可辨識 grammar、明確 NOT NULL 與 WITHOUT ROWID,避免 INTEGER PRIMARY KEY 的 rowid alias 自動給 ID 行為被偷偷改成 PostgreSQL 無 sequence 的 bigint。單欄整數/文字/二進位與複合 PK 均可。PostgreSQL → PostgreSQL 保留 catalog 的主鍵 constraint/背後索引名稱,名稱納入確認 fingerprint 與目標結構驗證。SQLite → PostgreSQL 明確產生 `表名前 58 字元_pkey`,並預檢它與所有表/普通索引/其他主鍵索引的 public 名稱空間碰撞,不讓引擎自動另取名稱。PostgreSQL → SQLite 僅允許上述預設命名慣例;其他名稱無法保留 `ON CONFLICT ON CONSTRAINT` 依賴,預檢拒絕。catalog 無法區分手動指定但恰好符合預設慣例的名稱,跨 SQLite 路徑明確不保證此預設名稱/內部索引名稱與實體配置;需要名稱依賴的資料庫應使用 PostgreSQL 目標。
- 普通非 unique btree 索引保留名稱、欄位與順序,只允許升冪、預設內建 opclass、無 expression/partial/INCLUDE/自訂 storage。SQLite 升冪 NULL 在前,目標 PostgreSQL 明確 `ASC NULLS FIRST`;反向 nullable 欄位索引若為 PostgreSQL 預設 NULLS LAST 則拒絕,NOT NULL 索引沒有這項差異。DESC、unique(PK 除外)、自訂定序/opclass、clustered/replica identity 索引都阻擋。
- SQLite/PostgreSQL schema+索引+資料仍為單一交易,不引入逐表 partial success。每次重新預檢/最終驗證涵蓋 PK、索引、內容;已提交或 ack lost 不可盲重試。
- 有界讀取先在 SQL 投影取 byte length、超限 CASE 回傳 NULL,再檢查預算,才 materialize 欄位。BLOB 不會先 GetValue 完整大陣列;SQLite expression 用已限額的 byte[] 讀取(避開 provider GetBytes 的 metadata 路徑),PostgreSQL 分段讀取。NULL 與空欄位至少計入 8 bytes 預算。這是應用端快照限額,不是對引擎自身 page cache/SQL 執行記憶體的硬上限。
Expand Down
1 change: 1 addition & 0 deletions mySQLPunk.Core/Models/LogicalMigration.cs
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ public sealed record LogicalMigrationIndex(string Name, IReadOnlyList<string> Co
public sealed record LogicalMigrationTable(string Name, IReadOnlyList<LogicalMigrationColumn> Columns, long Rows, string ContentHash)
{
public IReadOnlyList<string> PrimaryKey { get; init; } = Array.Empty<string>();
public string? PrimaryKeyName { get; init; }
public IReadOnlyList<LogicalMigrationIndex> Indexes { get; init; } = Array.Empty<LogicalMigrationIndex>();
}
public sealed record LogicalMigrationPreview(
Expand Down
25 changes: 21 additions & 4 deletions mySQLPunk.Core/Services/LogicalMigrationKeys.cs
Original file line number Diff line number Diff line change
Expand Up @@ -26,12 +26,19 @@ internal static bool PostgreSqlIndexKeyFits(int blockSize, IReadOnlyList<object?
return true;
}

private sealed record Keys(IReadOnlyList<string> PrimaryKey, IReadOnlyList<LogicalMigrationIndex> Indexes);
private static string DefaultPrimaryKeyName(string table) => table[..Math.Min(table.Length, 58)] + "_pkey";

private static LogicalMigrationTable TargetTable(DatabaseProviderKind provider, LogicalMigrationTable table) =>
table with { PrimaryKeyName = table.PrimaryKey.Count == 0 || provider == DatabaseProviderKind.Sqlite
? null : table.PrimaryKeyName ?? DefaultPrimaryKeyName(table.Name) };

private sealed record Keys(IReadOnlyList<string> PrimaryKey, IReadOnlyList<LogicalMigrationIndex> Indexes, string? PrimaryKeyName);

private static async Task<Keys> ReadKeysAsync(DbConnection connection, DbTransaction? transaction,
DatabaseProviderKind provider, string table, List<LogicalMigrationColumn> columns, List<string> blockers, CancellationToken token)
{
var primary = new List<string>();
string? primaryName = null;
var indexes = new List<LogicalMigrationIndex>();
if (provider == DatabaseProviderKind.Sqlite)
{
Expand Down Expand Up @@ -92,14 +99,24 @@ private static async Task<Keys> ReadKeysAsync(DbConnection connection, DbTransac
else
{
await using var command = connection.CreateCommand(); command.Transaction = transaction;
command.CommandText = "SELECT ic.relname, i.indisprimary, a.attname FROM pg_index i JOIN pg_class c ON c.oid=i.indrelid JOIN pg_namespace n ON n.oid=c.relnamespace JOIN pg_class ic ON ic.oid=i.indexrelid CROSS JOIN LATERAL unnest(i.indkey) WITH ORDINALITY k(attnum,position) LEFT JOIN pg_attribute a ON a.attrelid=c.oid AND a.attnum=k.attnum WHERE n.nspname='public' AND c.relname=@table ORDER BY ic.relname COLLATE \"C\", k.position;";
command.CommandText = "SELECT ic.relname, i.indisprimary, a.attname, pc.conname FROM pg_index i JOIN pg_class c ON c.oid=i.indrelid JOIN pg_namespace n ON n.oid=c.relnamespace JOIN pg_class ic ON ic.oid=i.indexrelid LEFT JOIN pg_constraint pc ON pc.conrelid=c.oid AND pc.conindid=i.indexrelid AND pc.contype='p' CROSS JOIN LATERAL unnest(i.indkey) WITH ORDINALITY k(attnum,position) LEFT JOIN pg_attribute a ON a.attrelid=c.oid AND a.attnum=k.attnum WHERE n.nspname='public' AND c.relname=@table ORDER BY ic.relname COLLATE \"C\", k.position;";
var parameter = command.CreateParameter(); parameter.ParameterName = "@table"; parameter.Value = table; command.Parameters.Add(parameter);
await using var reader = await command.ExecuteReaderAsync(token).ConfigureAwait(false);
var indexFields = new SortedDictionary<string, List<string>>(StringComparer.Ordinal);
while (await reader.ReadAsync(token).ConfigureAwait(false))
{
if (reader.IsDBNull(2)) { blockers.Add("不支援 expression 索引:" + reader.GetString(0)); continue; }
if (reader.GetBoolean(1)) primary.Add(reader.GetString(2));
if (reader.GetBoolean(1))
{
primary.Add(reader.GetString(2));
if (reader.IsDBNull(3)) blockers.Add("無法識別主鍵 constraint 名稱:" + table);
else
{
primaryName = reader.GetString(3);
if (!Identifier.IsMatch(primaryName) || primaryName != reader.GetString(0))
blockers.Add("主鍵 constraint/索引名稱超出可忠實保留子集:" + table);
}
}
else
{
var name = reader.GetString(0);
Expand All @@ -112,6 +129,6 @@ private static async Task<Keys> ReadKeysAsync(DbConnection connection, DbTransac
}
if (primary.Any(name => columns.All(column => column.Name != name || column.Nullable)))
blockers.Add("主鍵必須明確 NOT NULL:" + table);
return new(primary.ToArray(), indexes.OrderBy(index => index.Name, StringComparer.Ordinal).ToArray());
return new(primary.ToArray(), indexes.OrderBy(index => index.Name, StringComparer.Ordinal).ToArray(), primaryName);
}
}
15 changes: 12 additions & 3 deletions mySQLPunk.Core/Services/LogicalMigrationService.cs
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,12 @@ private static LogicalMigrationPreview BuildPreview(ConnectionProfile source, Co
.Concat(destination.Blockers.Select(reason => "目標:" + reason)).ToList();
if (target.Provider == DatabaseProviderKind.PostgreSql)
{
var relationNames = origin.Tables.Select(table => table.Name)
.Concat(origin.Tables.SelectMany(table => table.Indexes.Select(index => index.Name)))
.Concat(origin.Tables.Where(table => table.PrimaryKey.Count > 0)
.Select(table => TargetTable(target.Provider, table).PrimaryKeyName!));
if (relationNames.GroupBy(name => name, StringComparer.Ordinal).Any(group => group.Count() > 1))
blockers.Add("表/普通索引/主鍵索引名稱在 PostgreSQL public 名稱空間衝突;不允許引擎自動更名。");
if (destination.BlockSize != 8192) blockers.Add("PostgreSQL 索引預算只驗證過 8192-byte block_size;未知/其他 page 大小不能放行。");
foreach (var table in origin.Tables)
{
Expand All @@ -88,6 +94,9 @@ private static LogicalMigrationPreview BuildPreview(ConnectionProfile source, Co
if (target.Provider == DatabaseProviderKind.Sqlite && (origin.Tables.GroupBy(table => table.Name, StringComparer.OrdinalIgnoreCase).Any(group => group.Count() > 1) ||
origin.Tables.Any(table => table.Columns.GroupBy(column => column.Name, StringComparer.OrdinalIgnoreCase).Any(group => group.Count() > 1))))
blockers.Add("來源名稱只差大小寫;SQLite 無法忠實保存這些不同的表/欄位名稱。");
if (target.Provider == DatabaseProviderKind.Sqlite)
foreach (var table in origin.Tables.Where(table => table.PrimaryKeyName != null && table.PrimaryKeyName != DefaultPrimaryKeyName(table.Name)))
blockers.Add(table.Name + ":SQLite 無法保留 PostgreSQL 明確主鍵 constraint/索引名稱 " + table.PrimaryKeyName + ";請使用 PostgreSQL 目標。");
if (target.Provider == DatabaseProviderKind.Sqlite && origin.Tables.Select(table => table.Name)
.Concat(origin.Tables.SelectMany(table => table.Indexes.Select(index => index.Name)))
.GroupBy(name => name, StringComparer.OrdinalIgnoreCase).Any(group => group.Count() > 1))
Expand Down Expand Up @@ -180,7 +189,7 @@ private static async Task<LogicalMigrationResult> RunCoreAsync(ConnectionProfile
stage = "逐欄型別、NULL、內容與結構驗證";
observer?.Invoke("verify", destination, targetTransaction);
var written = await ReadSnapshotAsync(destination, targetTransaction, target.Provider, true, token).ConfigureAwait(false);
if (written.Blockers.Count != 0 || JsonSerializer.Serialize(written.Tables) != JsonSerializer.Serialize(sourceSnapshot.Tables))
if (written.Blockers.Count != 0 || JsonSerializer.Serialize(written.Tables) != JsonSerializer.Serialize(sourceSnapshot.Tables.Select(table => TargetTable(target.Provider, table))))
throw new InvalidOperationException("目標結構或內容 SHA-256 多重集合驗證不符。");
var sourceFinal = await ReadSnapshotAsync(origin, sourceTransaction, source.Provider, true, token).ConfigureAwait(false);
if (sourceFinal.Blockers.Count != 0 || JsonSerializer.Serialize(sourceFinal.Tables) != JsonSerializer.Serialize(sourceSnapshot.Tables))
Expand Down Expand Up @@ -306,7 +315,7 @@ private static async Task<Snapshot> ReadSnapshotAsync(DbConnection connection, D
rows.Add(row);
}
}
tables.Add(new(definition.Name, columns.AsReadOnly(), rows.Count, ContentHash(rows)) { PrimaryKey = keys.PrimaryKey, Indexes = keys.Indexes });
tables.Add(new(definition.Name, columns.AsReadOnly(), rows.Count, ContentHash(rows)) { PrimaryKey = keys.PrimaryKey, PrimaryKeyName = keys.PrimaryKeyName, Indexes = keys.Indexes });
data.Add(definition.Name, rows);
}
return new(version, identity, tables, data, blockers, blockSize);
Expand Down Expand Up @@ -400,7 +409,7 @@ private static string CreateTableSql(DatabaseProviderKind provider, LogicalMigra
Quote(column.Name) + " " + (column.Kind == "integer" ? provider == DatabaseProviderKind.Sqlite ? "INTEGER" : "bigint"
: column.Kind == "text" ? provider == DatabaseProviderKind.Sqlite ? "TEXT COLLATE BINARY" : "text COLLATE \"C\""
: provider == DatabaseProviderKind.Sqlite ? "BLOB" : "bytea") + (column.Nullable ? "" : " NOT NULL"))) +
(table.PrimaryKey.Count == 0 ? "" : ",PRIMARY KEY (" + string.Join(",", table.PrimaryKey.Select(Quote)) + ")") +
(table.PrimaryKey.Count == 0 ? "" : "," + (provider == DatabaseProviderKind.PostgreSql ? "CONSTRAINT " + Quote(TargetTable(provider, table).PrimaryKeyName!) + " " : "") + "PRIMARY KEY (" + string.Join(",", table.PrimaryKey.Select(Quote)) + ")") +
")" + (provider == DatabaseProviderKind.Sqlite ? " STRICT" + (table.PrimaryKey.Count == 0 ? ";" : ", WITHOUT ROWID;") : ";");

private static string CreateIndexSql(DatabaseProviderKind provider, LogicalMigrationTable table, LogicalMigrationIndex index) =>
Expand Down
Loading
Loading