feat(shadow-diff): shadow-diff harness proving Rust/Zig parity - #49
Merged
Conversation
Porting library SQL to Zig has no Zig-side ground truth, so the port is only defensible if both implementations can be run against the same input and their output compared. MT_SHADOW_DIFF makes library_get_all serve the Rust result as before while also asking the sidecar the same query and comparing the two compact JSON documents byte for byte -- the strictest comparison available, since zig-core emits keys in Track's declaration order on purpose and a byte compare also catches a value written into the wrong key slot. A divergence is logged, never thrown, with the first differing byte offset, context from both sides, and a structural summary; the Rust response stays what the frontend sees. With the flag off the command costs one env::var read -- the clone, the round-trip and the comparison are all behind it. The fixture-driven test drives the real sidecar binary over a socket across 26 query shapes (every sort column, both orders, ignore_words through the strip_sort_prefix UDF each side registers independently, search with every character the encoder has to agree about, and four pagination edges) over tests/fixtures/mt_fixture.db, built from mt_20260127.sql through the real schema path and generated on demand so CI needn't order the tasks. --sabotage on the sidecar forces one field to a constant so the harness is shown to catch a divergence, not merely to compare. It is a flag rather than an edit-and-revert so the demonstration stays repeatable; respond delegates with false, leaving the production path byte-identical. ci:shadow-diff runs the harness in a new non-blocking test.yml job with continue-on-error, off every other job's needs, so the main rust job keeps neither the Zig toolchain nor the ability to fail on Zig-side work. Verified: Rust 892 passed / 0 failed, zig 36/36, and the Playwright non-@tauri suite at 504 passed with MT_SHADOW_DIFF=1 and zero divergences (the 5 failures reproduce identically with these changes stashed). The repo-root mt.db is a pre-migration schema and cannot be an automated fixture -- the sidecar fails on it with "no such column: disc_number" -- so mt_20260127.sql through Database::new is the fixture, recorded as a divergence rather than worked around.
…e gap Re-ran the Playwright suite under the repo's actual default engine (webkit, via the official Playwright container image rather than symlinking mismatched host libjpeg/libjxl sonames) and recorded the result: same pre-existing failures as chromium, one parallel-load-flaky test cleared on isolated rerun, no new diff-attributable failures. Also documents a gap found during that follow-up: the non-@tauri Playwright suite never invokes the real Tauri/Rust command backing MT_SHADOW_DIFF (window.__TAURI__ is absent or hand-mocked in every spec), so AC#3's "zero divergences" claim is vacuously true on both engines. Left as-is per reviewer decision; flagged for a follow-on task.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
MT_SHADOW_DIFF=1-gated dual-call/compare path solibrary_get_allcan run the existing Rust implementation alongside the new Zig sidecar endpoint and byte-compare their canonical JSON output, logging (never throwing on) divergence with enough detail to diagnose it (field, both values, byte offset).zig-core's--sabotageflag plus a Rust-side parity test matrix (26 query shapes against the realmt_20260127.sql-derived fixture, >1,000 rows compared) demonstrating both zero divergences on clean parity and exactly one divergence when sabotaged.shadow-diffCI job (continue-on-error: true, absent from every other job'sneeds) so this never gates the critical path.Review notes
piagent run in an isolated worktree, then human-reviewed: diff scope verified, ACs confirmed genuinely checked (not just claimed), a stray autoformatter diff on two YAML files was found and reverted, and the AC#3 chromium-vs-webkit gap the agent disclosed was independently re-verified.mcr.microsoft.com/playwright:v1.58.0-noblecontainer rather than symlinking mismatched host libjpeg/libjxl sonames — genuinely compatible libs, not a manufactured pass. Same pre-existing failures reproduce as under chromium; one additional failure was confirmed to be parallel-load flakiness, not a regression (this diff touches zero frontend files).@tauriPlaywright suite never actually invokes the real Tauri/Rust command backingMT_SHADOW_DIFF(IPC is mocked or absent in every spec), so AC#3's "zero divergences logged" claim is vacuously true on both engines. AC#1/chore(deps): lock file maintenance #2/docs: add comprehensive project documentation #5's Rust-level parity tests remain the real evidence the harness works; this is flagged for a follow-on task rather than blocking this one.Test plan
cargo nextest run --workspace(892 passed / 0 failed)zig build test(36/36)shadow_diff_parity_test.rs: 26-query-shape matrix, zero divergences; sabotage test asserts exactly one divergencedeno fmt --check,deno lint,cargo fmt --all -- --check,zig fmt --check,actionlintall clean