MPI_OP2 Communication-Avoiding Optimizations Demo

Implementation of "Communication-Avoiding Optimizations for Large-Scale Unstructured-Mesh Applications with OP2" (ICPP 2023)

Project Overview

This project demonstrates the core concepts from the research paper on communication-avoiding (CA) optimizations for unstructured-mesh applications without implementing the full code generation aspect.

Current Status

✅ Phase 1: Data Loading & Mesh Structures (COMPLETED)

Implemented:

• Core mesh data structures (nodes, edges, cells)
• Connectivity maps (node-to-edge, edge-to-cell, node-to-cell)
• HDF5 mesh loader for OP2 format files
• SU2 mesh format loader (CFD standard)
• Automatic connectivity building
• Mesh statistics and bounds calculation

Test Results:

• Successfully loaded M6 wing mesh (m6wing.h5)
  • 108,396 nodes
  • 710,525 edges
  • Avg 13.11 edges per node
  • Bounds: 20.96m × 10.96m × 19.34m

✅ Phase 2: Mesh Partitioning (COMPLETED)

Implemented:

• METIS-based graph partitioning (node adjacency)
• Partition structures with owned, boundary, and ghost elements
• Neighbor discovery and shared boundary tracking
• Partition summary metrics (load balance, communication pattern)

Test Results:

• 4-way split of 108K-node ONERA M6 mesh
  • Edge-cut: 8,008
  • Load imbalance < 1%
  • ~3.3–4.7% boundary nodes per partition
  • Full neighbor communication graph extracted

✅ Phase 3: Halo Construction (COMPLETED)

Implemented:

• Single-layer halo builder mirroring standard OP2 exchanges
• Multi-layer halo expansion (depth-selectable) via BFS
• Aggregated per-neighbor, per-depth ghost counts
• Demo comparing halo growth on test and real meshes

Test Results:

• Single-layer halos on M6 mesh: 4,741 send / 3,336 receive nodes across ranks
• Depth-2 CA halos: 10,878 ghost nodes (2.3× growth) with neighbor breakdown

✅ Phase 4: Loop-Chain Analysis (COMPLETED)

Implemented:

• Loop-chain metadata structures (sets, datasets, access descriptors)
• Dependency analysis that accumulates read-after-write distances
• Per-loop halo depth tracking and aggregate statistics
• Demo reproducing paper's loop chain and a deeper mapped example

Test Results:

• Example chain (update_kernel → regflug → ilsav → periodicity) ⇒ required halo depth 2
• Mixed edge/cell/node chain ⇒ required halo depth 3 with depth trace 1,3,3

✅ Phase 5: CA Optimizer (COMPLETED)

Implemented:

• Partition-level communication calculator comparing baseline vs CA
• Aggregated metrics (messages, node volume) derived from loop depth
• Demo combining loop-chain analysis, partitioning, and halo builders

Test Results:

• 12×12 grid, 4 partitions: 75% fewer messages, ~50% lower volume
• ONERA M6, 4 partitions: 75% fewer messages, ~43% lower volume

✅ Phase 6: Performance Modeling (COMPLETED)

Implemented:

• perf_model module encoding Equations 2–4 with configurable alpha/β/γ
• Synthetic communication timers inside ca_optimizer_demo
• CLI overrides for latency, bandwidth, payload size, and redundant compute cost
• Reported per-partition and chain-level timing predictions

Sample Results (defaults):

• 12×12 grid, 4 partitions: CA time drops by ~74%, predicted speedup ≈ 3.9×
• ONERA M6, 4 partitions: CA time drops by ~46%, predicted speedup ≈ 1.9×

� Phase 7: Visualization & Validation (IN PROGRESS)

Implemented so far:

• ca_optimizer_demo --csv captures per-partition and chain metrics for plotting
• scripts/plot_ca_metrics.py renders baseline vs CA volume/time charts from CSV output

Upcoming:

• Collect real MPI timings to validate the analytic model
• Generate publication-quality plots and summary tables

�🔄 Next Phases

Phase 7: Visualization & Validation

• Plot model predictions vs synthetic timers
• Integrate real MPI timings when deployment environment is ready
• Produce summary tables/plots for paper reproduction

Building the Project

# Source environment (sets up ParMETIS/METIS paths)
source scripts/setup_env.sh

# Build
./scripts/build.sh

# Run demo
./build/mesh_loader_demo data/m6wing.h5

Project Structure

MPI_OP2/
├── CMakeLists.txt          # Build configuration
├── include/                # Header files
│   ├── mesh.h             # Mesh data structures
│   ├── mesh_loader.h      # HDF5/SU2 loaders
│   ├── partition.h        # METIS partitioning utilities
│   ├── halo.h             # Halo builders (single & multi-layer)
│   ├── loop_chain.h       # Loop-chain descriptors & analysis
│   ├── ca_optimizer.h     # Communication metrics for CA vs baseline
│   └── perf_model.h       # Analytic latency/bandwidth model helpers
├── src/                   # Implementation
│   ├── mesh.c
│   ├── mesh_loader.c
│   ├── partition.c
│   ├── halo.c
│   ├── loop_chain.c
│   ├── ca_optimizer.c
│   └── perf_model.c
├── demo/                  # Demo programs
│   ├── mesh_loader_demo.c
│   ├── partition_demo.c
│   ├── halo_demo.c
│   ├── loop_chain_demo.c
│   └── ca_optimizer_demo.c
├── data/                  # Mesh datasets
│   ├── m6wing.h5         # ONERA M6 wing (OP2 format)
│   ├── m6wing.su2        # ONERA M6 wing (SU2 format)
│   └── ONERAM6_pitching.h5
├── scripts/               # Build scripts
│   ├── setup_env.sh
│   ├── build.sh
│   └── plot_ca_metrics.py # Visualization helper for CSV output
└── paper/                 # Research paper pages

Dependencies

• HDF5: Reading OP2 mesh files
• ParMETIS/METIS: Mesh partitioning (already installed)
• MPI: Distributed memory simulation
• CMake: Build system

Key Concepts from the Paper

1. Loop-Chains

Sequences of consecutive loops without global synchronization. Example from paper:

update_kernel → regflug → ilsav → periodicity

2. Multi-Layered Halo

Extended halo regions to support dependency chains:

• Standard OP2: Single layer halo
• CA version: Multi-depth halo based on loop-chain analysis

3. Communication Reduction

• Paper shows 30-65% runtime reduction for certain loop-chains
• Achieved by grouping messages and reducing MPI call count
• Trade-off: Increased computation for reduced communication

Next Steps

Experiment with the performance model and timers:

# Synthetic grid with custom latency/bandwidth
./build/ca_optimizer_demo 24 8 --latency 2.5 --bandwidth 20 --compute 15 --bytes 96

# Real ONERA M6 mesh
./build/ca_optimizer_demo --real 4

# Capture metrics to CSV and plot
./build/ca_optimizer_demo 24 8 --csv metrics.csv
python scripts/plot_ca_metrics.py metrics.csv metrics

Values are interpreted as microseconds (--latency), GB/s (--bandwidth), nanoseconds per redundant node (--compute), and bytes per communicated node (--bytes). Plots require matplotlib (install with pip install matplotlib).

References

• Paper: Ekanayake et al., "Communication-Avoiding Optimizations for Large-Scale Unstructured-Mesh Applications with OP2", ICPP 2023
• OP2 Library: https://github.com/OP-DSL/OP2-Common