Skip to content

ParFlowGPU build fails on Stages/2026 NVHPC+OpenMPI #148

Closed

Description

@kvrigor

This fails:

$ ./build_tsmp2.sh parflowgpu --env env/jsc.2026.nvhpc.openmpi --verbose
   ...
-- Creating BLT OpenMP targets...
-- Found OpenMP_C: -mp (found version "5.1")
-- Found OpenMP_CXX: -mp (found version "5.1")
-- Found OpenMP_CUDA: -fopenmp (found version "4.5")
-- Found OpenMP_Fortran: -mp (found version "5.1")
-- Found OpenMP: TRUE (found version "5.1")
-- BLT OpenMP Compile Flags: $<$<NOT:$<COMPILE_LANGUAGE:CUDA>>:-mp>;$<$<COMPILE_LANGUAGE:CUDA>:-Xcompiler=-mp>
-- BLT OpenMP Link Flags:    -mp
   ...
[ 28%] Building CUDA object pfsimulator/amps/CMakeFiles/amps.dir/mpi1/amps_gpupacking.cu
nvcc -forward-unknown-to-host-compiler -DCAMP_HAVE_CUDA -DFMT_HEADER_ONLY=1
  --options-file CMakeFiles/amps.dir/includes_CUDA.rsp
  --expt-extended-lambda
  -I/e/software/default/stages/2026/software/OpenMPI/5.0.8-nvidia-compilers-25.9-CUDA-13/include
  -O3 -DNDEBUG "--generate-code=arch=compute_90,code=[compute_90,sm_90]"
  -Xcompiler=-mp -MD -MT
  pfsimulator/amps/CMakeFiles/amps.dir/mpi1/amps_gpupacking.cu.o
  -MF CMakeFiles/amps.dir/mpi1/amps_gpupacking.cu.o.d
  -x cu -c TSMP2/models/parflow/pfsimulator/amps/mpi1/amps_gpupacking.cu
  -o CMakeFiles/amps.dir/mpi1/amps_gpupacking.cu.o

gcc: error: unrecognized command-line option ‘-mp’; did you mean ‘-MP’?

On the other hand jsc.2026.gnu.openmpi passes. This environment correctly detects the right OpenMP linker flags:

$ ./build_tsmp2.sh parflowgpu --env env/jsc.2026.gnu.openmpi
    ...
-- Creating BLT OpenMP targets...
-- Found OpenMP_C: -fopenmp (found version "4.5")
-- Found OpenMP_CXX: -fopenmp (found version "4.5")
-- Found OpenMP_CUDA: -fopenmp (found version "4.5")
-- Found OpenMP_Fortran: -fopenmp (found version "4.5")
-- Found OpenMP: TRUE (found version "4.5")
-- BLT OpenMP Compile Flags: $<$<NOT:$<COMPILE_LANGUAGE:CUDA>>:-mp>;$<$<COMPILE_LANGUAGE:CUDA>:-Xcompiler=-mp>
-- BLT OpenMP Link Flags:    -fopenmp
    ...
[100%] Completed 'ParFlow'
[100%] Built target ParFlow

Stages/2026 NVHPC invokes nvcc -Xcompiler=-mp which causes the failure. The correct call should be nvcc -Xcompiler=-fopenmp. The auto-detected OpenMP linker flags (e.g. -fopenmp/-mp) seem to come from CMake's FindOpenMP function:

function(_OPENMP_FLAG_CANDIDATES LANG)
  if(NOT OpenMP_${LANG}_FLAG)
    unset(OpenMP_FLAG_CANDIDATES)

    set(OMP_FLAG_GNU "-fopenmp")
    set(OMP_FLAG_LCC "-fopenmp")
    set(OMP_FLAG_Clang "-fopenmp=libomp" "-fopenmp=libiomp5" "-fopenmp" "-Xclang -fopenmp")
    ...
    set(OMP_FLAG_PathScale "-openmp")
    set(OMP_FLAG_NAG "-openmp")
    set(OMP_FLAG_Absoft "-openmp")
    set(OMP_FLAG_NVHPC "-mp")
    set(OMP_FLAG_PGI "-mp")
    set(OMP_FLAG_Flang "-fopenmp")

The ParFlow CMake build should be modified to precisely control its OpenMP auto-detection behavior. In particular,

  • find_package(OpenMP) should choose -fopenmp instead of -mp when under NVHPC toolchain; or
  • -mp should be directly passed to mpicc instead of being passed to nvcc -XCompiler=.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions