Repository navigation
MediaTek and QNN exports on Modal, sized per window - #16
Merged
Merged
Conversation
…FM2 checkpoints The Namespace profile (15.7 GiB, 4 vCPUs) carried no LFM2 window by the estimate, and a 32k streaming calibration needs cores more than memory. mtk-matrix now gives each window a Sandbox: memory from calibration_bytes (16 GiB floor), cores and minutes from export_mtk.MODAL_SHAPES, and one attention layer per chunk for a hybrid at 16k and up (finding 39). The image carries MediaTek's public Python 3.10 tools; scripts/mtk-setup.sh downloads, checks and deletes the NeuroPilot SDK inside each run, so none of it is stored. LiquidAI/LFM2-350M, -700M and -1.2B describe their layers as full_attn_idxs and block_ff_dim rather than LFM2.5's layer_types and intermediate_size; hub.normalize_config reads both, so sizing, chunking and lfm2.py see the same config. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Namespace profile has 15.7 GiB and no swap, and the one QNN export measured, Qwen3-0.6B, already peaked at 15.6 GB of RSS at 2k and 4k (findings 8 and 10), so nothing larger could build there. qnn-matrix gives each window a Sandbox: 20 bytes a parameter (24 GiB floor), half again by 8k, 16 or 32 cores and up to 23 hours by model size. The request is a minimum Modal lets the run exceed, and each report records the peak to refit it against. The wheel still downloads QAIRT on first import, now inside the Sandbox, so the Actions cache that kept a copy is gone and the SDK is never stored. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…hybrid-only chunking to hybrids From Codex's review of the Modal move: the Sandbox was sized at the default 128-token prompt batch while the export took the prompt_tokens input (73 GiB against 24 GiB for the 1.2B at 32k with 4096); a window past MODAL_SHAPES raised and dropped the whole matrix instead of that one window; and a plain transformer whose config lists its layers as all full_attention was cut one chunk per layer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s own memory peak From Codex's review of the QNN move, true of every export on Modal: the Sandbox did not get GITHUB_SERVER_URL, GITHUB_REPOSITORY or GITHUB_RUN_ID, so run_info was empty and the model cards lost their build links; and MemorySampler's peak_in_use_bytes comes from /proc/meminfo, which in a Sandbox is the host's. It now also records the cgroup's peak where the container has one, as peak_cgroup_bytes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…no longer fetches 1.5.1 emptied backends/qualcomm/__init__.py and moved the download to utils/qnn_sdk_setup.setup_qnn_sdk, called by the paths that start a backend. The probe still imported the package and read the paths it would use, so the first export since the bump (SmolLM2-135M at 2k on Modal) found no libQnnHtp.so there and failed inside the Qualcomm script. The probe now calls setup_qnn_sdk and checks the libraries exist before the script. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…Linux CI runner has Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Moves the last two exports off the 15.7 GiB Namespace runner and onto Modal, the way XNNPACK and Vulkan already run.
mtk-matrixsizes a Sandbox per window. Memory comes fromcalibration_bytes(16 GiB floor), and cores and minutes fromexport_mtk.MODAL_SHAPES. Hybrids at 16k and up get one attention layer per chunk (finding 39).scripts/mtk-setup.shdownloads, verifies and deletes the NeuroPilot SDK inside each run, so none of it is stored.qnn-matrixrequests 20 B per parameter (24 GiB floor, up to 1.5× by 8k), with 16 or 32 cores. This is a minimum that Modal lets the run exceed. QAIRT is downloaded inside each run, and the Actions cache that kept a copy is removed.setup_qnn_sdk()and checks that the libraries exist.hub.normalize_configreadsfull_attn_idxsandblock_ff_dim.prompt_tokensoverride.peak_cgroup_bytes).Tested on this branch, with publishing off:
264 tests pass, plus the 20 QNN tests against executorch 1.5.1.
🤖 Generated with Claude Code
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is enabled.