Repository navigation
QNN Qwen3 calibration fix, and every job on GitHub-hosted runners - #17
Merged
Merged
Conversation
…Qwen3 window calibrates The Qualcomm script calibrates on wikitext through lm_eval with ExecuTorch's EagerEvalWrapper, whose prefix_token_id in 1.5.1 returns bos_id whenever the attribute exists, None included. Qwen3 has no BOS token, so every Qwen3-0.6B and Qwen3-1.7B window failed three minutes in with "'NoneType' object cannot be interpreted as an integer". ExecuTorch main falls back to the end-of-text token, as lm_eval's HFLM does; pipeline/qnn_launch.py sets that property and runs the script unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… on Modal The Namespace profile took six jobs at a time, and each export job held one for its whole run only to wait on a Modal Sandbox, so a QNN dispatch of five models queued behind itself. ubuntu-latest, free for this public repository, has no such cap. It stops a job at 360 minutes, so the Sandboxes now end by 330 and the long ones get more cores instead: 64 for QNN models above 1.3B and for MediaTek at 32k, 32 for MediaTek from 8k to 16k. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every Qwen3 QNN window failed about three minutes in. The failing runs were 37643657045 (Qwen3-0.6B, all three windows) and 37643663364 (Qwen3-1.7B). The error came from inside lm_eval during wikitext calibration:
TypeError: 'NoneType' object cannot be interpreted as an integer.The cause is in ExecuTorch 1.5.1.
EagerEvalWrapper.prefix_token_idreturnsbos_idwhenever the attribute exists, even when it isNone, and Qwen3 has no BOS token. ExecuTorchmainfalls back to the end-of-text token in that case, as lm_eval's own HFLM does.pipeline/qnn_launch.pybackports that one property and then runs the Qualcomm script unchanged, viarunpy.The tests cover the patched property against the real
eager_eval(22 QNN tests pass with the export pins). CI skips that one test, since it doesn't install lm_eval or transformers.Every job now runs on GitHub-hosted runners. The exports already run on Modal, but each export job held a Namespace runner the whole time it waited. The profile took six jobs at a time, so a five-model QNN dispatch queued behind itself.
ubuntu-latestis free for this public repo and has no such cap. It does stop a job at 360 minutes, so:.github/actionlint.yaml, which only declared the Namespace label, is removed.🤖 Generated with Claude Code
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is enabled.