[WIP] CI: split test into 4 parallel shards by cennn · Pull Request #68 · SandAI-org/MagiCompiler

cennn · 2026-09-01T13:03:11Z

动机

当前 CI 每个 PyTorch 版本的 test job 耗时约 39 分钟（单个 pytest -v tests/），是整个 pipeline 的瓶颈。现有 4 个 runner 但只用了 1 个跑测试，利用率低。且 feature_tests/ 下 23 个测试文件混杂在一起，功能边界不清，定位失败困难。

核心改动

1. 重组 tests/feature_tests/ 为逻辑子目录（纯 git mv，0 行代码改动）

feature_tests/
├── fsdp/          ← FSDP overlap (torchrun subprocess tests)
├── cache/         ← cache invariant / topology / artifact
├── pass/          ← inductor passes, CUTLASS fusion
├── conftest.py    ← 共享 fixtures（不动，子目录自动继承）
└── test_*.py      ← 剩余：CUDA graph / symbolic / profiling / offload

Helper 目录随 test 一起移动（fsdp_overlap_helper/ → fsdp/、cache_reuse_helper/ → cache/），所有 helper 引用都用 Path(__file__).parent / ... 相对路径，移动后自动正确。

2. CI matrix 从 1 → 7 个并行 shard

Shard	路径	测试数	特性
perf	`tests/perf_tests/`	29	GPU timing 敏感，独立 rerun
api	`tests/api_tests/` + `tests/model_tests/`	120	重度 torch.compile
feature-fsdp	`tests/feature_tests/fsdp/`	20	FSDP overlap / torchrun
feature-cache	`tests/feature_tests/cache/`	65	cache + artifact 管理
feature-pass	`tests/feature_tests/pass/`	56	inductor pass + fusion
feature-runtime	`tests/feature_tests/` (remaining) + `tests/torch_native_tests/`	59	CUDA graph / symbolic / profiling
depyf	`tests/magi_depyf/`	219	大部分纯 CPU bytecode roundtrip

用 strategy.matrix 实现，fail-fast: false 保证不互相取消。

3. 自动选卡：每个 shard 启动时 nvidia-smi 查询显存占用最低的 GPU → CUDA_VISIBLE_DEVICES，多 shard 共享 runner 时减少 GPU 争抢。

收益

wall-clock 降低：4 runner 并行 → 测试从 ~39 min 降至瓶颈 shard 的 ~8-12 min
故障隔离：perf timing 不稳只 rerun perf shard；FSDP torchrun 失败只 rerun fsdp shard
可观测性：CI 面板 7 个独立 check 状态，一眼看出哪个子系统出问题
启动开销：每 shard 额外 ~20-30s（pull cached image + start container），7 shard 总额外开销 ~2-3 min，远小于并行节省的 25+ min

文件变更

_ci_pipeline.yml：single test → 7-shard matrix + GPU auto-select
tests/feature_tests/：23 个 test 文件按功能归入 fsdp/ / cache/ / pass/ 子目录，纯 rename 0 代码改动

Split the monolithic test job (~39 min) into 4 independent shards: - test-perf: perf_tests (timing-sensitive, isolated for rerun) - test-api: api_tests + model_tests (heavy torch.compile) - test-feature: feature_tests + torch_native_tests (FSDP/cache/mixed) - test-depyf: magi_depyf (mostly CPU bytecode roundtrips) All 4 shards run in parallel after build, each with 25-min timeout. With 4 runners, wall-clock test time drops from ~39 min to ~15 min.

Replace 4 copy-pasted test jobs with a single matrix job (fail-fast: false). Restore env-check diagnostic output (PT version, CUDA version, GPU name).

…tion Reorganize tests/feature_tests/ into logical subdirectories: - fsdp/: FSDP overlap tests + torchrun subprocess helpers - cache/: cache invariant, topology, artifact, build cleanup - pass/: inductor passes, CUTLASS fusion, conv channels-last All moves use git mv (pure rename, 0 code changes). Helpers use Path(__file__).parent so relative paths remain correct. CI matrix expanded from 4 to 7 shards: perf, api, feature-fsdp, feature-cache, feature-pass, feature-runtime, depyf. Each shard auto-selects the least-loaded GPU via nvidia-smi at startup (CUDA_VISIBLE_DEVICES) to minimise contention when multiple shards land on the same physical runner.

FSDP overlap e2e/reorder and profiling collective tests use torchrun --nproc_per_node=2, requiring 2 visible GPUs. Previous single-GPU selection would cause these tests to skip or fail.

test_register_custom_op.py alone has 88 tests and is the heaviest file in api_tests. Splitting gives better load balance across runners.

…p_helper)

cennn added 2 commits September 1, 2026 21:02

ci: use matrix strategy to deduplicate 4 test shards

2936a9b

Replace 4 copy-pasted test jobs with a single matrix job (fail-fast: false). Restore env-check diagnostic output (PT version, CUDA version, GPU name).