[GSoC] Feature: Jacobi preconditioner on GPU and CUDA Unified preconditiong matrix

[ddg93]

Proposed Changes

This draft PR introduces CUDA Unified memory allocation and management for the CSysVector class and the preconditioning matrix inside the Jacobi preconditioner. Also, this draft PR extends the section of GPU execution inside the FGMRES solver. Custom CUDA kernels are implemented for the preconditioning matrix, the multi dot product and the linear combination (the first operations of the Modified Gram-Schmidt orthogonalization).

The solver logic is not modified, and the GPU path is hidden inside the specific methods. A custom data() method recovers the CSysVector Unified pointer inside the CUDA logic, cleaning the logic from the double host/device pointers. All memory explicit memory copies are also removed from the CUDA logic, but explicit device synchronizations are introduced around MPI calls and at the end of the CUDA section.

Performance evaluation (on-going)

for the considered test-case (rae2822), the CPU execution expresses an Avg. s/iter: 0.198681 while the GPU execution expresses an Avg. s/iter: 0.256129 on my local Quadro P620 GPU.
I intend to perform more tests on more advanced hardware, considering different test-cases and evaluating against the CPU version and a GPU version with separated device and host pointers and memory copies.

Asynchronous pre-fetching:

The Jacobi preconditioner calculations are performed on GPU through a new custom CUDA kernel through the preconditioner abstraction.
The preconditioning matrix is selected to test CUDA Unified Memory asynchronous prefetching to the GPU.
For simplicity, the double CPU/GPU pointer is still maintained in the current logic, although the device pointer reduces to an alias for the Unified Memory pointer when this kind of allocation is adopted.

This strategy introduces a simple context to test the CUDA Unified Memory usage and study the possibility of overlapping memory transfers and calculations without the need to introduce CUDA streams.

Concretely, this PR:

• introduces new CUDA Unified Memory allocation methods and asynchronous prefetching;
• introduces the GPU logic for the Jacobi preconditioner;
• introduces the GPU logic for the multi dot product and the linear combination (Modified Gram-Schmidt orthogonalization);

This work is part of my ongoing contribution during the Google Summer of Code 2026 program.

Validation

Validated locally with:

• serial CUDA build compilation
• serial CPU build compilation
• CPU/GPU numerical comparison on 1 representative case (rae2822) tested with LINEAR_SOLVER_PREC=JACOBI.

Nsys profiling was performed to confirm the asynchronous prefetching of the CUDA Unified Memory preconditioning matrix on my local GPU. Partial prefetching is observed, although page faults were reported during the preconditioner CUDA kernel, indicating that the calculations were slowed down by the prefetching matrix still being transferred to the GPU. This overlap is expected to largely improve on more modern hardware; tests are ongoing in the cloud.

Related Work

The Jacobi preconditioner kernels come from the PR #2825.

Observed Issues

Some issues were observed during this first period of GSoC:

• PR Add initial end-to-end CUDA FGMRES solver path #2825 compiles with CUDA 13.3 but CUSPARSE calls raise an unknown operation at the first matrix-vector product.
• the build.meson file has a hard-coded CUDA arch.
• the HAVE_MPI flag is not being passed to the nvcc compiler in the develop branch. This raises a linking error if MPI operations are included within the *.cu files. That is not the case in the master branch.

Next steps

I propose to continue working on the following steps:

• perform further testing on the asynchronous prefetching of the preconditioning matrix;
• extend the CUDA Unified Memory allocation to the CSysVector class
• evaluate if it might be of interest to extend the CUDA Unified Memory allocation to the CSysMatrix class;
• I observe that CSysMatrixVectorProduct object in the FGMRES solver is the Jacobian matrix. If the Jacobian preconditioner is applied, the preconditioning matrix is also calculated from the Jacobian matrix. It would then be convenient to issue the transfer of the Jacobian matrix to the GPU inside the building of the preconditioner and perform the preconditioning matrix calculations on the GPU (sparse matrix inversion based on the current LU decomposition). Then, the preconditioning matrix would already be on the GPU, reducing by one the number of transfers from CPU to GPU. This is not the current case as both the Jacobian matrix and the preconditioning matrix are being transferred to device.

PR Checklist

Put an X by all that apply. You can fill this out after submitting the PR. If you have any questions, don't hesitate to ask! We want to help. These are a guide for you to know what the reviewers will be looking for in your contribution.

• I am submitting my contribution to the develop branch.
• My contribution generates no new compiler warnings (try with --warnlevel=3 when using meson).
• My contribution is commented and consistent with SU2 style (https://su2code.github.io/docs_v7/Style-Guide/).
• I used the pre-commit hook to prevent dirty commits and used pre-commit run --all to format old commits.
• I have added a test case that demonstrates my contribution, if necessary.
• I have updated appropriate documentation (Tutorials, Docs Page, config_template.cpp), if necessary.

[bigfooted]

That's great!
Can you run pre-commit to fix the code style/formatting?

[pcarruscag]

I think Areen looked into unified memory a bit.
Do you have some performance measurements? I imagine we save on the data transfer but suffer a little on the kernel performance?
Does unified memory introduce a significant restriction in terms of the required compute capability?

[ddg93]

That's great! Can you run pre-commit to fix the code style/formatting?

Thanks for the heads-up. Updated that commit and the following ones.

[ddg93]

I think Areen looked into unified memory a bit. Do you have some performance measurements? I imagine we save on the data transfer but suffer a little on the kernel performance? Does unified memory introduce a significant restriction in terms of the required compute capability?

For the moment, I've only one performance measurement on my local GPU Quadro P620:

• GPU version based on Unified memory: Avg. s/iter: 0.256129
• CPU version: Avg. s/iter: 0.198681
  This result is clearly not sufficient but shows a decent performance of the GPU version based on Unified memory on a small test-case and on a very modest GPU.
  I intend to measure performance for a GPU version without Unified memory, and also consider larger tests and more advanced hardware (through cloud services).

Preliminary profiling shows indeed that the transfer of the preconditioning matrix is asynchronously launched but then the preconditioning kernel takes longer due to page fault. I intend to better profile what is going on with the CSysVector Unified memory implementation.

Unified memory is available for the compute capability 6.0 or higher. Some of its advanced features seem to depend on the system.

Is there some report by Areen? It could be interesting for me to read it.

[pcarruscag]

Thanks I was wondering about unified vs explicit transfers not CPU vs GPU.
Not sure if Areen registered those tests somewhere.
Based on what nvidia says about unified memory, and the our goal with the linear solver, we should transfer explicitly and avoid complexity of keeping both strategies