Hangs in some IMB-RMA tests when I_MPI_ROOT is set and when using a non-One API-provided libfabric

[charlesshereda]

The IMB-RMA tests Accumulate, Get_accumulate, Fetch_and_op, and Compare_and_swap all hang for us if we use a libfabric version other than the one provided with OneAPI.

If, however, we unset the env var I_MPI_ROOT, all the tests complete normally.

We discovered this because the script which we source for OneAPI MPI is setting that env var:

source /opt/intel/oneapi/mpi/latest/env/vars.sh release

[JuliaRS]

@charlesshereda which libfabric version and oneAPI did you try to use ?

[charlesshereda]

@JuliaRS Sorry, I never saw the notification for your communication and wasn't actively checking this issue. The libfabric version was a custom build of ours; we're developing the OPX provider so you should be able to see it from the latest release of libfabric (e.g., 1.21).

At the time that we discovered this we were testing with IMPI 2021.9.0 - so whatever oneAPI that includes that IMPI version. We haven't tried with the latest IMPI, which I understand just came out in June.

[JuliaRS]

@charlesshereda how did you source libfabric?
You need export FI_PROVIDER_PATH to folder with libopx-fi.so and LD_LIBRARY_PATH to folder with libfabric.so (library with libfabric core)

[zachdworkin]

Adding more information to this thread. I realize this is the verbs provider but I have also noticed this "hang" for aggregate mode accumulate test but with out the box impi. Here is what my output looks like. It is not actually hanging but running each test size very very slowly (~1-2min per size until timeout). If I run with ompi its ok so it leads me to believe its an impi issue.

Running with impi 2021.14, oneapi 2025.0.0, built-in-libfabric, imb 2021.8
source intel/oneapi/2025.0.0/setvars.sh
source intel/oneapi/2025.0.0/mpi/2021.14/env/vars.sh
export FI_PROVIDER="verbs;ofi_rxm"
mpirun -n 2 -ppn 1 -hosts n1-ib0,n2-ib0 ./IMB-RMA accumulate
#----------------------------------------------------------------

Intel(R) MPI Benchmarks 2021.8, MPI-RMA part

#----------------------------------------------------------------

Date : Fri Feb 21 18:39:03 2025

Machine : x86_64

System : Linux

Release : 5.14.0-427.13.1.el9_4.x86_64

Version : #1 SMP PREEMPT_DYNAMIC Wed Apr 10 10:29:16 EDT 2024

MPI Version : 3.1

MPI Thread Environment:

Calling sequence was:

./IMB-RMA accumulate

Minimum message length in bytes: 0

Maximum message length in bytes: 4194304

MPI_Datatype : MPI_BYTE

MPI_Datatype for reductions : MPI_FLOAT

MPI_Op : MPI_SUM

List of Benchmarks to run:

Accumulate

#---------------------------------------------------

Benchmarking Accumulate

#processes = 2

#---------------------------------------------------

MODE: NON-AGGREGATE

   #bytes #repetitions      t[usec]   Mbytes/sec
        0          100         0.22         0.00
        4          100         2.83         1.41
        8          100         2.82         2.84
       16          100         2.86         5.59
       32          100         6.43         4.98
       64          100         2.77        23.07
      128          100         3.12        41.03
      256          100         3.17        80.73
      512          100         3.40       150.65
     1024          100         3.80       269.28
     2048          100         4.14       494.61
     4096          100         5.29       773.93
     8192          100         7.56      1083.03
    16384          100         9.36      1750.67
    32768          100        12.20      2685.82
    65536          100        17.12      3827.64
   131072          100        28.59      4584.08
   262144          100        48.19      5440.08
   524288           80        87.27      6007.37
  1048576           40       187.74      5585.36
  2097152           20       357.02      5874.02
  4194304           10       700.70      5985.84

#---------------------------------------------------

Benchmarking Accumulate

#processes = 2

#---------------------------------------------------

MODE: AGGREGATE

   #bytes #repetitions      t[usec]   Mbytes/sec
        0         1000         0.05         0.00
        4          112         0.86         4.66
        8          112         0.87         9.25
       16 time-out.; Time limit (secs_per_sample * msg_sizes_list_len) is over; use "-time X" or SECS_PER_SAMPLE=X (IMB_settings.h) to increase time limit.

Running imb 2021.8 + ompi 5.0.6 + libfabric 2.0.0 and the test completes very quickly.

export PATH=/path/to/ompi/5.0.6/bin:$PATH
export PATH=/path/to/libfabric/2.0.0/bin:$PATH
export LD_LIBRARY_PATH=/path/to/ompi/5.0.6/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/path/to/libfabric/2.0.0/lib:$LD_LIBRARY_PATH
export FI_PROVIDER="verbs;ofi_rxm"
mpirun -x LD_LIBRARY_PATH --host "n1-ib0,n2-ib0" -n 2 --mca opal_common_ofi_pr
ovider_include "verbs;ofi_rxm" --mca mtl ofi --mca btl ^tcp,ofi,vader,openib --bind-to core --map-by ppr:1:node --oversubscribe ./IMB-RMA accumulate
#----------------------------------------------------------------

Intel(R) MPI Benchmarks 2021.8, MPI-RMA part

#----------------------------------------------------------------

Date : Fri Feb 21 18:36:48 2025

Machine : x86_64

System : Linux

Release : 5.14.0-427.13.1.el9_4.x86_64

Version : #1 SMP PREEMPT_DYNAMIC Wed Apr 10 10:29:16 EDT 2024

MPI Version : 3.1

MPI Thread Environment:

Calling sequence was:

./IMB-RMA accumulate

Minimum message length in bytes: 0

Maximum message length in bytes: 4194304

MPI_Datatype : MPI_BYTE

MPI_Datatype for reductions : MPI_FLOAT

MPI_Op : MPI_SUM

List of Benchmarks to run:

Accumulate

#---------------------------------------------------

Benchmarking Accumulate

#processes = 2

#---------------------------------------------------

MODE: NON-AGGREGATE

   #bytes #repetitions      t[usec]   Mbytes/sec
        0          100         6.40         0.00
        4          100         8.10         0.49
        8          100         8.16         0.98
       16          100         8.22         1.95
       32          100         8.22         3.89
       64          100         8.31         7.71
      128          100         8.37        15.30
      256          100         8.96        28.57
      512          100        41.95        12.21
     1024          100        38.85        26.36
     2048          100        10.00       204.79
     4096          100        11.15       367.45
     8192          100        13.03       628.57
    16384          100        51.29       319.44
    32768          100        58.45       560.61
    65536          100        71.12       921.45
   131072          100        90.04      1455.70
   262144          100       127.35      2058.53
   524288           80       213.74      2452.87
  1048576           40       368.60      2844.76
  2097152           20       667.84      3140.22
  4194304           10      1217.64      3444.62

#---------------------------------------------------

Benchmarking Accumulate

#processes = 2

#---------------------------------------------------

MODE: AGGREGATE

   #bytes #repetitions      t[usec]   Mbytes/sec
        0         1000         5.68         0.00
        4         1000         7.37         0.54
        8         1000         7.40         1.08
       16         1000         7.44         2.15
       32         1000         7.52         4.26
       64         1000         7.52         8.51
      128         1000         7.61        16.82
      256         1000         8.04        31.83
      512         1000        42.10        12.16
     1024         1000        36.27        28.24
     2048         1000         9.13       224.26
     4096         1000        10.35       395.90
     8192         1000        12.15       674.11
    16384         1000        15.37      1066.03
    32768         1000        57.26       572.22
    65536          640        28.88      2269.29
   131072          320        91.12      1438.45
   262144          160        79.97      3277.94
   524288           80       209.51      2502.42
  1048576           40       366.67      2859.70
  2097152           20       546.80      3835.31
  4194304           10      1216.82      3446.95

All processes entering MPI_Finalize