Skip to content

Reject truncated tensors in the Flash Next pack comparator - #919

Open
cwschroeder wants to merge 1 commit into
ashhart:mainfrom
cwschroeder:fix/flashnext-pack-compare-short-reads
Open

cwschroeder wants to merge 1 commit into
ashhart:mainfrom
cwschroeder:fix/flashnext-pack-compare-short-reads

Conversation

@cwschroeder

@cwschroeder cwschroeder commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

tools/zig/pack_compare.py could report packs match for truncated Flash Next packs: it clipped both tensor reads to their common prefix. Two identically truncated payloads also passed.

Validate each tensor's byte count against its dtype and shape, file boundary and actual read length before comparison. Invalid tensors count as differences; complete tensors still require matching names, dtypes, shapes and bytes. Empty and scalar tensors remain valid.

Validation on upstream 46a08c18608153bfc566703a88acc883bc6140f4, patch 9ef17f27e1c94bdc2ed8c91d6a6affc7ec36786e:

  • Six regressions pass with Python 3.14.7 and NumPy 2.5.2. The unchanged original base produced four failures. The path-scoped CI workflow pins both versions.
  • Full Zig 0.17.0 host, kernel-contract, golden and fuzz checks pass: 214/214 build steps; zero unexpected golden differences. Whole-tree formatting and patch whitespace pass. Lean: 517 inherited findings, zero new.
  • Range-diff confirms the patch is unchanged after the rebase. Astra found no new integration blocker. All five checks for this rebased head pass: pack comparison, Zig format, No GPU, desktop and mobile.

Found while auditing the packing evidence for #706. This is tensor comparison with length and geometry checks. It does not validate all Safetensors invariants or qualify a 4-bit checkpoint/model.

@cwschroeder

Copy link
Copy Markdown
Contributor Author

All five checks pass on 54ab3294, including the six pack-comparison regressions on Linux with Python 3.14.7 and NumPy 2.5.2. Local whole-tree Zig tests also finished: 160/160 steps, 1981 passed and 8 skipped. Formatting, server goldens, fuzz and the lean comparison pass (zero new findings). Main is still c7137bb3.

I marked this comparator-only fix ready for review. The checkpoint and model gates for #706 remain separate.

@cwschroeder
cwschroeder force-pushed the fix/flashnext-pack-compare-short-reads branch from 54ab329 to 9ef17f2 Compare October 16, 2026 09:23
@cwschroeder

Copy link
Copy Markdown
Contributor Author

Rebased onto main 46a08c18 at 9ef17f27; range-diff confirms the comparator patch is unchanged. All five checks pass on this head, including the six Linux regressions with Python 3.14.7 and NumPy 2.5.2. Local whole-tree host/kernel/golden/fuzz validation passes 214/214 build steps, formatting passes, and lean has zero new findings.

Ready for maintainer review. This remains the comparator completeness fix; #706 still needs checkpoint and model qualification.

@ashhart ashhart mentioned this pull request Oct 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant