Three compute kernels, each optimized step by step with a different parallel-programming model — every version is benchmarked against the same correctness-checked baseline. Best result: scaled-dot-product attention from 241 s to 3.2 s (75.6×) with MPI + cache blocking + AVX-512 on a 4-node cluster.
| Kernel | Optimization stages | Tech |
|---|---|---|
attention/ |
serial → MPI row partitioning → + cache-blocked matmul → + AVX-512 intrinsics | MPI, _mm512 FMA intrinsics |
cuda-transpose/ |
naive → shared-memory tiling → + bank-conflict avoidance via padding | CUDA, Nsight Compute |
conv2d/ |
serial → OpenMP collapse(2) → + cache tiling |
OpenMP |
Measured on a 4-node RDMA cluster, 16 processes per node; timings recorded in the source header with correctness verified per run:
| Version | Time | Speedup |
|---|---|---|
| serial | 240.99 s | 1× |
| MPI | 6.68 s | 36.1× |
| MPI + blocked matmul | 5.70 s | 42.3× |
| MPI + AVX-512 | 5.15 s | 46.8× |
| MPI + blocked + AVX-512 | 3.19 s | 75.6× |
The report also examines process-count scaling (1→16 processes: 6.3× → 37.7× on the largest input) and why node count does not scale monotonically (communication vs. computation trade-off).
| Version | Time | vs. naive |
|---|---|---|
| naive (global memory) | 627 µs | 1× |
| shared-memory tiles (32×32) | 224 µs | 2.8× |
| + bank-conflict avoidance (padding) | 176 µs | 3.6× |
run_experiments.sh sweeps 12 matrix sizes × 100 repetitions; Nsight Compute counters
(l1tex__data_bank_conflicts_*) confirm the padding removes the 32-way conflicts caused by
the 8-byte complex element stride (full analysis in the report).
3–4× speedup at 4–8 threads on a 4-core CPU; the report analyzes why scaling degrades beyond the physical core count (threading overhead, false sharing) and the effect of cache tiling (measurements in the report).
Each kernel directory has its own Makefile:
# conv2d (gcc + OpenMP)
cd conv2d && make
./conv-openmp MATRIX_FILE KERNEL_FILE ANSWER_FILE
# attention (OpenMPI; AVX-512-capable CPU for the avx512 targets)
cd attention && make
mpirun --hostfile hosts -npernode 16 ./attention-mpi-blocked-avx512 TESTCASE.bin
# cuda-transpose (CUDA toolkit)
cd cuda-transpose && make
./run_experiments.sh # full benchmark sweep
./ncu-profiling.sh # Nsight Compute profilingTest inputs are not committed (course-provided binaries up to 513 MB); input formats are documented in each kernel's source.
Assignment work for High-Performance Big Data and AI Systems at NTU (Spring 2025), taught by Prof. Shih-Hao Hung. The convolution and CUDA-transpose kernels are my individual work. The attention kernel was a four-person group assignment; I was responsible for the AVX-512 and cache-blocking optimizations and the experiments. Cleaned up and documented for release.