2C (GKS) VXCx/VXCy unit assertions exceed 1e-10 tolerance on A100 / CUDA 13

[wavefunction91]

Summary

The two-component (GKS / non-collinear) VXCx and VXCy checks in tests/xc_integrator.cxx fail on NVIDIA A100 (sm_80) + CUDA 13, exceeding the hardcoded 1e-10 per-nbf norm tolerance. EXC, scalar VXC, and VXCz all pass. The failure is deterministic and shows up on both single-GPU and 4-rank MPI runs (so it is not a race / oversubscription artifact).

This looks like hardware/toolchain-dependent floating-point roundoff (the reference data appears to predate this HW/CUDA), amplified because the x/y magnetization components are near-zero and the most roundoff-sensitive. Filing so we can revisit tolerance / reference portability for the 2C path.

Failing assertions

tests/xc_integrator.cxx (GKS branch):

• L252 CHECK( VXCy_diff_nrm / basis.nbf() < 1e-10 );

• L253 CHECK( VXCx_diff_nrm / basis.nbf() < 1e-10 );

• L264 CHECK( VXCy1_diff_nrm / basis.nbf() < 1e-10 ); (state-propagation re-eval)

• L265 CHECK( VXCx1_diff_nrm / basis.nbf() < 1e-10 );

• 1-GPU: 33/34 cases pass, 4 / 6,804,067 assertions fail (the four above).

• 4-GPU (mpirun -np 4): same checks fail (~6 assertions/rank); no segfault, all 4 GPUs utilized.

• Exact diff_nrm/nbf magnitudes: TODO (present in the run logs; to be appended).

Environment

• Azure Standard_NC96ads_A100_v4, 4x A100 80GB PCIe, driver 580.126.20
• CUDA 13.0.88, Ubuntu 22.04 (Azure HPC image), GCC 11.4, OpenMPI, system OpenBLAS
• GauXC master @ 15d57f4, CMAKE_CUDA_ARCHITECTURES=80

Reproducer

git clone --recursive https://github.com/wavefunction91/GauXC.git
cmake -S GauXC -B build -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_POLICY_VERSION_MINIMUM=3.5 \
  -DGAUXC_ENABLE_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=80 \
  -DGAUXC_ENABLE_MPI=ON -DGAUXC_ENABLE_OPENMP=ON \
  -DGAUXC_ENABLE_MAGMA=OFF -DGAUXC_ENABLE_CUTLASS=OFF \
  -DGAUXC_ENABLE_TESTS=ON
cmake --build build -j
# OPENBLAS_NUM_THREADS=1 avoids a nested-thread OpenBLAS segfault on many-core hosts
CUDA_VISIBLE_DEVICES=0 OMP_NUM_THREADS=$(nproc) OPENBLAS_NUM_THREADS=1 ./build/tests/gauxc_test

Notes: -DCMAKE_POLICY_VERSION_MINIMUM=3.5 is required for the bundled libxc under CMake 4.x.

Possible follow-ups

• Capture actual VXCx/VXCy_diff_nrm / nbf values to distinguish "tolerance too tight" vs real drift.
• Consider a relative / hardware-portable tolerance for the near-zero 2C components.
• Regenerate the 2C reference data on current HW/CUDA.
• Check whether a non-collinear kernel has arch-dependent reduction ordering.