From a832076d0b9cdb411d1654a46d92577b20f2b277 Mon Sep 17 00:00:00 2001
From: William Dawson <william.dawson@riken.jp>
Date: Wed, 5 Aug 2026 22:48:43 +0900
Subject: [PATCH 1/5] Add new application: petsc-gmres [code:petsc-gmres]

PETSc KSP (GMRES + GAMG) benchmark: loads the audikw_1 sparse SPD matrix
(SuiteSparse GHS_psdef group, 943,695 x 943,695, 77,651,847 nnz) via
MatLoad against a pre-converted PETSc binary file, solves Ax=b for a
known x, reports relative L2 error and solve wall-time.

PETSc is fetched via bk_fetch_source, pinned to v3.25.2. The app's own
source is vendored directly in src/ rather than fetched from a separate
repo, since its upstream development happens in a private RIKEN-RCCS
repository (not fetchable by bk_fetch_source's plain git clone).

RIKYU build/run recipe validated end to end on real hardware: PETSc
builds and configures cleanly against nvhpc-hpcx, the app compiles and
links, and a 4-rank run against the staged matrix produces a correct
result (relative L2 0.03154, matching the expected ~0.03-0.04 band).
Fugaku and RC_DGXSP cases are included in build.sh/run.sh based on
previously-validated recipes on those systems, but their list.csv rows
are enable=no pending data staging (see README.md's "Staging the data"
section -- Fugaku's group-storage volumes are currently at quota).

-pc_gamg_square_graph 0 is required at any rank count above 1 on every
system, to avoid a crash in GAMG's aggressive-coarsening graph-squaring
step on this matrix's connectivity (a cuSPARSE resource error on GPU, a
multi-GB single-allocation OOM on CPU).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---
 programs/petsc-gmres/README.md           |  74 ++++++++++++++++
 programs/petsc-gmres/build.sh            | 102 +++++++++++++++++++++++
 programs/petsc-gmres/list.csv            |   4 +
 programs/petsc-gmres/run.sh              |  99 ++++++++++++++++++++++
 programs/petsc-gmres/src/GMRES-PETSc.cpp | 102 +++++++++++++++++++++++
 5 files changed, 381 insertions(+)
 create mode 100644 programs/petsc-gmres/README.md
 create mode 100644 programs/petsc-gmres/build.sh
 create mode 100644 programs/petsc-gmres/list.csv
 create mode 100644 programs/petsc-gmres/run.sh
 create mode 100644 programs/petsc-gmres/src/GMRES-PETSc.cpp

diff --git a/programs/petsc-gmres/README.md b/programs/petsc-gmres/README.md
new file mode 100644
index 0000000..88fc508
--- /dev/null
+++ b/programs/petsc-gmres/README.md
@@ -0,0 +1,74 @@
+# petsc-gmres
+
+A PETSc KSP (GMRES + GAMG algebraic multigrid preconditioner) benchmark:
+load the `audikw_1` sparse SPD matrix ([SuiteSparse Matrix
+Collection](https://sparse.tamu.edu/GHS_psdef/audikw_1), `GHS_psdef`
+group — a real structural-engineering FEM problem, 943,695 rows,
+77,651,847 nonzeros), solve `Ax = b` for a known `x`, report the relative
+L2 error and solve wall-time (`FOM: ranks=<N> solve_time_s=<t>`).
+
+Source (`src/GMRES-PETSc.cpp`) is vendored directly in this directory
+rather than fetched from a separate repo at build time, since its
+upstream development happens in a private RIKEN-RCCS repository. PETSc
+itself is fetched normally via `bk_fetch_source` from its official repo,
+pinned to `v3.25.2`.
+
+Confirmed working on Rikyu (GB200 NVL4), Fugaku (A64FX), and R-CCS
+Cloud's DGX Spark (`RC_DGXSP`, GB10 Blackwell) — see this app's `build.sh`
+for the per-system recipe. Full strong-scaling results, root-cause
+diagnoses for a couple of real bugs found getting each port working
+(a matrix-distribution bug causing OOM, a GAMG coarsening crash on this
+matrix's connectivity, an MPI-transport gotcha on one system), and the
+underlying build recipes are documented in more depth in that same
+internal repository — not linked here since it isn't publicly readable,
+but available to RIKEN-RCCS members on request.
+
+## Correctness
+
+`relative L2 norm of the error` should land near 0.03–0.04 at any rank
+count (GMRES's default relative-residual tolerance, not a tight solve —
+this is a benchmark, not a production accuracy target). A result outside
+that band signals a real bug, not benchmark noise.
+
+## `-pc_gamg_square_graph 0`
+
+Required on every system, at any rank count above 1 (single-rank runs
+happen not to trigger it, but don't rely on that). Without it, GAMG's
+default aggressive-coarsening graph-squaring step blows up on
+`audikw_1`'s connectivity — a `CUSPARSE_STATUS_INSUFFICIENT_RESOURCES`
+crash on GPU, or a genuine multi-GB single-allocation PETSc "Out of
+memory" abort on CPU. Both `build.sh`'s configure line and `run.sh`'s
+launch command already account for everything needed except this flag,
+which is passed explicitly in `run.sh`.
+
+## Staging the data
+
+The matrix is pre-converted once (offline, not part of `build.sh`/`run.sh`)
+from MatrixMarket format to PETSc's binary format via a small one-time
+conversion tool (also part of the private upstream repo, not shipped
+here — it's not needed at benchmark build/run time, only to produce the
+staged file below once). This avoids every rank re-parsing a
+multi-hundred-MB text file at load time, which was the actual root cause
+of the original reported "runs out of memory for no reason" bug this
+benchmark exists to catch a regression of.
+
+Pre-staged locations (same convention as this repo's `ffb` and
+`LQCD_dw_solver`, which pre-stage their own — much larger — source
+archives at a fixed per-system path rather than fetching them at
+build/run time):
+
+| system | path |
+|---|---|
+| RIKYU | `/data1/rkp00015/benchkit-data/petsc-gmres/audikw_1.petscbin` |
+| Fugaku | *(not yet staged — group-storage quota exhausted on the volumes covered by this repo's `FJ` queue.csv `GFSCACHE` declaration; `Fugaku` row is `enable=no` in `list.csv` until this is resolved)* |
+| RC_DGXSP | *(not yet staged; `RC_DGXSP` row is `enable=no` in `list.csv` until this is resolved)* |
+
+To re-stage on a system with an existing PETSc install: download
+`audikw_1.mtx` from the SuiteSparse Matrix Collection link above, then use
+PETSc's own `MatLoad`/`MatView` binary-viewer round trip (or the private
+repo's `mtx2petsc` tool, if you have access) to write it to PETSc binary
+format. Verify the result: 943,695 × 943,695, nnz = 77,651,847 — the
+standard SuiteSparse download is MatrixMarket `symmetric` format (one
+triangle + diagonal only), so a correct converter must mirror off-diagonal
+entries; a naive read of the file as-is will silently produce a wrong,
+singular matrix with no error.
diff --git a/programs/petsc-gmres/build.sh b/programs/petsc-gmres/build.sh
new file mode 100644
index 0000000..e9121f2
--- /dev/null
+++ b/programs/petsc-gmres/build.sh
@@ -0,0 +1,102 @@
+#!/bin/bash
+set -euo pipefail
+
+system="$1"
+
+PETSC_REPO="https://gitlab.com/petsc/petsc.git"
+PETSC_TAG="v3.25.2"
+PETSC_DIR="${PWD}/petsc"
+ARTIFACT_DIR="${PWD}/artifacts"
+
+source scripts/bk_functions.sh
+
+mkdir -p "${ARTIFACT_DIR}"
+bk_fetch_source "${PETSC_REPO}" "petsc" "${PETSC_TAG}"
+
+case "$system" in
+  RIKYU)
+    module load nvhpc-hpcx/26.3
+    export PETSC_ARCH=arch-rikyu
+    NVPL=/shared/software/hpc_sdk/Linux_aarch64/26.3/math_libs/nvpl/lib
+    (
+      cd "${PETSC_DIR}"
+      # --with-fc=0: nvhpc-hpcx's Fortran wrapper doesn't support an F2018
+      # pointer-initialization feature PETSc's Fortran bindings need; this
+      # app is C++ only anyway. --with-cuda=0: this is the CPU-only build
+      # (see petsc-benchmarking in RIKEN-RCCS/block1-eeas for why a
+      # +cuda-enabled build's PetscInitialize scales ~linearly with rank
+      # count from CUDA-library dynamic-linking contention, unrelated to
+      # solve performance). Serial NVPL BLAS/LAPACK: this app runs flat
+      # MPI, no OpenMP threading.
+      ./configure \
+        --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 \
+        --with-debugging=0 --with-cuda=0 \
+        --with-blaslapack-lib="-L${NVPL} -Wl,-rpath,${NVPL} -lnvpl_lapack_lp64_seq -lnvpl_blas_lp64_seq" \
+        COPTFLAGS='-O3' CXXOPTFLAGS='-O3'
+      make PETSC_DIR="${PETSC_DIR}" PETSC_ARCH="${PETSC_ARCH}" -j8 all
+    )
+    mpicxx -O3 -I"${PETSC_DIR}/include" -I"${PETSC_DIR}/${PETSC_ARCH}/include" \
+      src/GMRES-PETSc.cpp -o "${ARTIFACT_DIR}/GMRES-PETSc" \
+      -Xlinker -rpath="${PETSC_DIR}/${PETSC_ARCH}/lib" \
+      -L"${PETSC_DIR}/${PETSC_ARCH}/lib" -lpetsc
+    ;;
+  Fugaku)
+    # See petsc-build in RIKEN-RCCS/block1-eeas: LLVM cross-compiler
+    # (mpiclang++) beats the Fujitsu compiler here, and Fugaku's own
+    # Spack-provided PETSc hits the same GAMG square-graph crash as a bare
+    # SIGKILL instead of a catchable PETSc error, so we build our own.
+    module load lang/tcsds-1.2.43
+    module load LLVM/llvmorg-22.1.0
+    export PETSC_ARCH=arch-fugaku-llvm
+    TCSDS=/opt/FJSVxtclanga/tcsds-latest/lib64
+    ELF=/opt/FJSVxos/devkit/aarch64/rfs/usr/lib64/libelf.so
+    (
+      cd "${PETSC_DIR}"
+      # Login nodes are x86, compute nodes are A64FX/aarch64 -- a genuine
+      # cross-compile, hence --with-batch. BLAS/LAPACK is Fujitsu's own
+      # fjlapacksve (serial), not OpenBLAS -- OpenBLAS pulls in an
+      # unresolvable chain of Fortran-runtime symbols on this toolchain
+      # (RIKEN's own docs: "use of libraries provided with the Fujitsu
+      # compiler with the other compiler environments is not supported",
+      # but they document this exact non-Fujitsu-compiler linkage recipe).
+      ./configure \
+        --with-cc=mpiclang --with-cxx=mpiclang++ --with-fc=0 \
+        --with-debugging=0 --with-batch \
+        --with-blaslapack-lib="${TCSDS}/libfjlapacksve.so ${TCSDS}/libfj90i.so ${TCSDS}/libfj90f.so ${TCSDS}/libfjsrcinfo.so ${TCSDS}/libfjcrt.so ${ELF}" \
+        --PETSC_ARCH="${PETSC_ARCH}"
+      make PETSC_DIR="${PETSC_DIR}" PETSC_ARCH="${PETSC_ARCH}" all
+    )
+    mpiclang++ -O3 -I"${PETSC_DIR}/include" -I"${PETSC_DIR}/${PETSC_ARCH}/include" \
+      src/GMRES-PETSc.cpp -o "${ARTIFACT_DIR}/GMRES-PETSc" \
+      "$(grep '^PETSC_WITH_EXTERNAL_LIB' "${PETSC_DIR}/${PETSC_ARCH}/lib/petsc/conf/petscvariables" | cut -d= -f2-)"
+    ;;
+  RC_DGXSP)
+    # GPU build -- audikw_1 is solved on-GPU here (1 rank/GPU), matching
+    # the cross-machine reproduction in RIKEN-RCCS/block1-eeas's
+    # petsc-build. -pc_gamg_square_graph 0 (set in run.sh) avoids a
+    # cuSPARSE crash on this matrix's connectivity during GAMG's
+    # aggressive-coarsening graph-squaring step.
+    source /etc/profile.d/modules.sh
+    module load system/ng-dgx nvhpc-hpcx/26.3
+    export PETSC_ARCH=arch-dgxsp-cuda
+    CUDA_ARCH=$(nvidia-smi --query-gpu=compute_cap --format=csv,noheader | head -1 | tr -d '.')
+    MATHLIBS=$(dirname "$(command -v nvcc)")/../../math_libs/*/lib64
+    (
+      cd "${PETSC_DIR}"
+      ./configure \
+        --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 \
+        --with-debugging=0 --with-cuda=1 --with-cuda-arch="${CUDA_ARCH}" \
+        LDFLAGS="-L${MATHLIBS} -Wl,-rpath,${MATHLIBS}" \
+        COPTFLAGS='-O3' CXXOPTFLAGS='-O3'
+      make PETSC_DIR="${PETSC_DIR}" PETSC_ARCH="${PETSC_ARCH}" -j8 all
+    )
+    mpicxx -O3 -I"${PETSC_DIR}/include" -I"${PETSC_DIR}/${PETSC_ARCH}/include" \
+      src/GMRES-PETSc.cpp -o "${ARTIFACT_DIR}/GMRES-PETSc" \
+      -Xlinker -rpath="${PETSC_DIR}/${PETSC_ARCH}/lib" \
+      -L"${PETSC_DIR}/${PETSC_ARCH}/lib" -lpetsc
+    ;;
+  *)
+    echo "Unknown system: $system" >&2
+    exit 1
+    ;;
+esac
diff --git a/programs/petsc-gmres/list.csv b/programs/petsc-gmres/list.csv
new file mode 100644
index 0000000..b78cd19
--- /dev/null
+++ b/programs/petsc-gmres/list.csv
@@ -0,0 +1,4 @@
+system,enable,nodes,numproc_node,nthreads,elapse
+RIKYU,yes,1,4,1,0:15:00
+Fugaku,no,1,48,1,0:15:00
+RC_DGXSP,no,1,1,1,0:15:00
diff --git a/programs/petsc-gmres/run.sh b/programs/petsc-gmres/run.sh
new file mode 100644
index 0000000..9ed504f
--- /dev/null
+++ b/programs/petsc-gmres/run.sh
@@ -0,0 +1,99 @@
+#!/bin/bash
+set -euo pipefail
+
+system="$1"
+nodes="$2"
+numproc_node="$3"
+nthreads="$4"
+n_ranks=$((nodes * numproc_node))
+
+source scripts/bk_functions.sh
+
+ARTIFACT="${PWD}/artifacts/GMRES-PETSc"
+RESULTS_DIR="${PWD}/results"
+mkdir -p "${RESULTS_DIR}"
+: > "${RESULTS_DIR}/result"
+
+if [[ ! -x "${ARTIFACT}" ]]; then
+  echo "Required artifact not found or not executable: ${ARTIFACT}" >&2
+  exit 1
+fi
+
+export OMP_NUM_THREADS="${nthreads}"
+
+# The benchmark matrix (audikw_1, SuiteSparse GHS_psdef group, 943,695 x
+# 943,695, 77,651,847 nnz, converted once to PETSc binary format) is
+# pre-staged at a fixed path per system rather than fetched at build/run
+# time -- same convention as this repo's ffb and LQCD_dw_solver, which
+# pre-stage their (much larger) source archives the same way. See
+# README.md for exactly how each copy was produced and how to re-stage it.
+logfile="solve.log"
+touch .run_marker
+
+case "${system}" in
+  RIKYU)
+    DATA=/data1/rkp00015/benchkit-data/petsc-gmres/audikw_1.petscbin
+    module load nvhpc-hpcx/26.3
+    mpirun -np "${n_ranks}" -N "${numproc_node}" --bind-to core --map-by core \
+      "${ARTIFACT}" -f "${DATA}" -pc_type gamg -pc_gamg_square_graph 0 \
+      > "${logfile}" 2>&1 || true
+    ;;
+  Fugaku)
+    # TODO: stage audikw_1.petscbin on Fugaku group storage and fill in
+    # DATA (see README.md's "Staging the data" section for the blocker
+    # hit doing this -- group quota exhausted on the volumes covered by
+    # this repo's own FJ queue.csv GFSCACHE declaration).
+    DATA=/vol0002/data/ra000009/benchkit-data/petsc-gmres/audikw_1.petscbin
+    module load lang/tcsds-1.2.43
+    module load LLVM/llvmorg-22.1.0
+    mpiexec -n "${n_ranks}" \
+      "${ARTIFACT}" -f "${DATA}" -pc_type gamg -pc_gamg_square_graph 0 \
+      > "${logfile}" 2>&1 || true
+    # Fugaku's PJM mpiexec writes each rank's real stdout/stderr under
+    # ./output.$PJM_JOBID/, ignoring plain shell redirection for the
+    # application's own output -- fall back to searching for it if the
+    # marker wasn't captured above (same pattern as this repo's sbd).
+    if ! grep -q "^FOM: ranks=" "${logfile}" 2>/dev/null; then
+      found=$(find . -maxdepth 5 -type f -newer .run_marker -name 'stdout*' 2>/dev/null | sort | head -n 1)
+      [[ -n "${found}" ]] && logfile="${found}"
+    fi
+    ;;
+  RC_DGXSP)
+    # TODO: stage audikw_1.petscbin on R-CCS Cloud storage and fill in
+    # DATA. GPU run (1 rank/GPU) -- see build.sh.
+    DATA=/lustre/share/benchkit-data/petsc-gmres/audikw_1.petscbin
+    source /etc/profile.d/modules.sh
+    module load system/ng-dgx nvhpc-hpcx/26.3
+    mpirun -np "${n_ranks}" \
+      "${ARTIFACT}" -f "${DATA}" -pc_type gamg -pc_gamg_square_graph 0 \
+      -mat_type aijcusparse \
+      > "${logfile}" 2>&1 || true
+    ;;
+  *)
+    echo "Unknown system: ${system}" >&2
+    exit 1
+    ;;
+esac
+
+if [[ ! -f "${DATA}" ]]; then
+  echo "Benchmark matrix not found at ${DATA} -- see README.md for how to stage it" >&2
+  exit 1
+fi
+
+if ! grep -q "^FOM: ranks=" "${logfile}" 2>/dev/null; then
+  echo "petsc-gmres success marker not found" >&2
+  echo "---- ${logfile} tail ----" >&2
+  tail -n 80 "${logfile}" >&2 || true
+  exit 1
+fi
+
+solve_time=$(grep "^FOM: ranks=" "${logfile}" | sed -E 's/.*solve_time_s=([0-9.]+).*/\1/')
+
+bk_emit_result \
+  --fom "${solve_time}" \
+  --fom-unit s \
+  --fom-version solve_time \
+  --exp audikw_1 \
+  --nodes "${nodes}" \
+  --numproc-node "${numproc_node}" \
+  --nthreads "${nthreads}" >> "${RESULTS_DIR}/result"
diff --git a/programs/petsc-gmres/src/GMRES-PETSc.cpp b/programs/petsc-gmres/src/GMRES-PETSc.cpp
new file mode 100644
index 0000000..23ff9a4
--- /dev/null
+++ b/programs/petsc-gmres/src/GMRES-PETSc.cpp
@@ -0,0 +1,102 @@
+// CSR data transfered to PETSc  Copyright,  Atsushi Suzuki 30 Jul.2025
+//
+// A PETSc KSP (GMRES + GAMG algebraic multigrid preconditioner) benchmark:
+// load a large sparse SPD matrix, solve Ax = b for a known x, report the
+// relative L2 error and solve wall-time.
+//
+// Loads a pre-converted PETSc binary matrix via MatLoad() against
+// PETSC_COMM_WORLD, so rows are properly partitioned across MPI ranks
+// (memory per rank scales as O(1/ranks), as a real distributed benchmark
+// should). The binary is produced once, offline, from the MatrixMarket
+// form of the benchmark matrix (audikw_1, SuiteSparse Matrix Collection,
+// GHS_psdef group) -- see this app's README.md for where each target
+// system's copy lives and how it was produced.
+static char help[] = "load a PETSc binary matrix and call the KSP solver\n";
+
+#include <petscksp.h>
+
+int main(int argc, char **args)
+{
+  Vec         x, b, u;        /* approx solution, RHS, exact solution */
+  Mat         A;              /* linear system matrix */
+  KSP         ksp;            /* linear solver context */
+  PC          pc;             /* PC context */
+  PetscMPIInt size, rank;
+  PetscBool   flg;
+  PetscScalar one = 1.0;
+  PetscReal   e, e0;
+
+  char        fname[1024];
+  PetscViewer viewer;
+
+  PetscFunctionBeginUser;
+  PetscCall(PetscInitialize(&argc, &args, (char *)0, help));
+  PetscCallMPI(MPI_Comm_size(PETSC_COMM_WORLD, &size));
+  PetscCallMPI(MPI_Comm_rank(PETSC_COMM_WORLD, &rank));
+  PetscCall(PetscOptionsGetString(NULL, NULL, "-f", fname, sizeof(fname), &flg));
+  if (!flg) {
+    PetscCall(PetscPrintf(PETSC_COMM_WORLD,
+			  "usage: %s -f <matrix.petscbin> [-ksp_monitor] [-pc_type gamg] [-log_view]\n",
+			  args[0]));
+    PetscCall(PetscFinalize());
+    exit(-1);
+  }
+
+  PetscCall(MatCreate(PETSC_COMM_WORLD, &A));
+  PetscCall(MatSetType(A, MATAIJ));
+  PetscCall(MatSetFromOptions(A));
+  PetscCall(PetscViewerBinaryOpen(PETSC_COMM_WORLD, fname, FILE_MODE_READ, &viewer));
+  PetscCall(MatLoad(A, viewer));
+  PetscCall(PetscViewerDestroy(&viewer));
+
+  //     Create and set vectors
+  PetscCall(MatCreateVecs(A, &u, &b));  // create vectors right u and left b
+  PetscCall(VecDuplicate(u, &x));       // copy vector x from u
+  PetscCall(VecSet(u, one));            // all entries of u are one
+  PetscCall(MatMult(A, u, b));          // RHS from b = A * u
+
+  //     Create linear solver context
+  PetscCall(KSPCreate(PETSC_COMM_WORLD, &ksp));
+  PetscCall(KSPSetOperators(ksp, A, A));
+
+  //     Set defulat preconditioner as diagonal preconditioning
+  PetscCall(KSPGetPC(ksp, &pc));
+  PetscCall(PCSetType(pc, PCJACOBI));
+  //     receive options for KSP solver from command line
+  PetscCall(KSPSetFromOptions(ksp));
+
+  // Figure of merit: wall-clock time of the actual solve (KSPSetUp, which
+  // builds the GAMG hierarchy, runs lazily inside this KSPSolve call, so
+  // setup + iterate are both included -- this is "how long it took to go
+  // from an assembled matrix to a converged solution", independent of
+  // process-launch/link startup cost, which is not part of the solve).
+  PetscLogDouble solve_t0, solve_t1;
+  PetscCall(PetscBarrier((PetscObject)ksp));
+  PetscCall(PetscTime(&solve_t0));
+  PetscCall(KSPSolve(ksp, b, x));
+  PetscCall(PetscBarrier((PetscObject)ksp));
+  PetscCall(PetscTime(&solve_t1));
+  {
+    PetscReal local_dt = (PetscReal)(solve_t1 - solve_t0), max_dt;
+    PetscCallMPI(MPI_Allreduce(&local_dt, &max_dt, 1, MPIU_REAL, MPI_MAX, PETSC_COMM_WORLD));
+    PetscCall(PetscPrintf(PETSC_COMM_WORLD, "FOM: ranks=%d solve_time_s=%.6f\n", (int)size, (double)max_dt));
+  }
+
+  PetscCall(VecNorm(u, NORM_2, &e0));
+  PetscCall(VecAXPY(x, -1.0, u));
+  PetscCall(VecNorm(x, NORM_2, &e));
+  PetscCall(PetscPrintf(PETSC_COMM_WORLD,
+			"relative L2 norm of the error: %g\n", (double)(e/e0)));
+
+  flg = PETSC_FALSE;
+  PetscCall(PetscOptionsGetBool(NULL, NULL, "-print_error", &flg, NULL));
+  if (flg) PetscCall(PetscPrintf(PETSC_COMM_WORLD, "Infinity norm of the error: %g\n", (double)e));
+
+  PetscCall(KSPDestroy(&ksp));
+  PetscCall(VecDestroy(&u));
+  PetscCall(VecDestroy(&x));
+  PetscCall(VecDestroy(&b));
+  PetscCall(MatDestroy(&A));
+  PetscCall(PetscFinalize());
+  return 0;
+}

From c9418b1233a06e663dcf5039de58622acda443e8 Mon Sep 17 00:00:00 2001
From: William Dawson <william.dawson@riken.jp>
Date: Wed, 5 Aug 2026 23:42:57 +0900
Subject: [PATCH 2/5] Enable RC_DGXSP for petsc-gmres [code:petsc-gmres]

Validated end to end on real ng-dgx-m2 hardware (same GB10 Blackwell
hardware as the ng-dgx-m1 this app's list.csv actually targets, just
idle at the time): PETSc CUDA build (CUDA_ARCH auto-detected as 121,
matching GB10's sm_121), matrix conversion (nnz=77,651,847, correct),
and a 1-rank GPU run all succeeded -- solve_time_s=7.57,
relative L2=0.0443346 (correct, matches the known-good cross-machine
baseline).

Stages audikw_1.petscbin at a fixed path under $HOME (this system has
no separate group-storage tier, unlike RIKYU/Fugaku), flips RC_DGXSP to
enable=yes in list.csv, and drops the now-stale nvhpc-hpcx/26.3 version
pin in favor of the unversioned module name actually available here.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---
 programs/petsc-gmres/README.md | 2 +-
 programs/petsc-gmres/build.sh  | 2 +-
 programs/petsc-gmres/list.csv  | 2 +-
 programs/petsc-gmres/run.sh    | 9 +++++----
 4 files changed, 8 insertions(+), 7 deletions(-)

diff --git a/programs/petsc-gmres/README.md b/programs/petsc-gmres/README.md
index 88fc508..deff6c8 100644
--- a/programs/petsc-gmres/README.md
+++ b/programs/petsc-gmres/README.md
@@ -61,7 +61,7 @@ build/run time):
 |---|---|
 | RIKYU | `/data1/rkp00015/benchkit-data/petsc-gmres/audikw_1.petscbin` |
 | Fugaku | *(not yet staged — group-storage quota exhausted on the volumes covered by this repo's `FJ` queue.csv `GFSCACHE` declaration; `Fugaku` row is `enable=no` in `list.csv` until this is resolved)* |
-| RC_DGXSP | *(not yet staged; `RC_DGXSP` row is `enable=no` in `list.csv` until this is resolved)* |
+| RC_DGXSP | `/home/users/william.dawson/benchkit-data/petsc-gmres/audikw_1.petscbin` (this system has no separate group-storage tier, so — unlike RIKYU/Fugaku — this lives under `$HOME`) |
 
 To re-stage on a system with an existing PETSc install: download
 `audikw_1.mtx` from the SuiteSparse Matrix Collection link above, then use
diff --git a/programs/petsc-gmres/build.sh b/programs/petsc-gmres/build.sh
index e9121f2..bc4b358 100644
--- a/programs/petsc-gmres/build.sh
+++ b/programs/petsc-gmres/build.sh
@@ -77,7 +77,7 @@ case "$system" in
     # cuSPARSE crash on this matrix's connectivity during GAMG's
     # aggressive-coarsening graph-squaring step.
     source /etc/profile.d/modules.sh
-    module load system/ng-dgx nvhpc-hpcx/26.3
+    module load system/ng-dgx nvhpc-hpcx
     export PETSC_ARCH=arch-dgxsp-cuda
     CUDA_ARCH=$(nvidia-smi --query-gpu=compute_cap --format=csv,noheader | head -1 | tr -d '.')
     MATHLIBS=$(dirname "$(command -v nvcc)")/../../math_libs/*/lib64
diff --git a/programs/petsc-gmres/list.csv b/programs/petsc-gmres/list.csv
index b78cd19..4fb47bb 100644
--- a/programs/petsc-gmres/list.csv
+++ b/programs/petsc-gmres/list.csv
@@ -1,4 +1,4 @@
 system,enable,nodes,numproc_node,nthreads,elapse
 RIKYU,yes,1,4,1,0:15:00
 Fugaku,no,1,48,1,0:15:00
-RC_DGXSP,no,1,1,1,0:15:00
+RC_DGXSP,yes,1,1,1,0:15:00
diff --git a/programs/petsc-gmres/run.sh b/programs/petsc-gmres/run.sh
index 9ed504f..5783e6a 100644
--- a/programs/petsc-gmres/run.sh
+++ b/programs/petsc-gmres/run.sh
@@ -59,11 +59,12 @@ case "${system}" in
     fi
     ;;
   RC_DGXSP)
-    # TODO: stage audikw_1.petscbin on R-CCS Cloud storage and fill in
-    # DATA. GPU run (1 rank/GPU) -- see build.sh.
-    DATA=/lustre/share/benchkit-data/petsc-gmres/audikw_1.petscbin
+    # GPU run (1 rank/GPU) -- see build.sh. This system has no separate
+    # group-storage tier (see README.md), so the data lives under $HOME
+    # like everything else here.
+    DATA=/home/users/william.dawson/benchkit-data/petsc-gmres/audikw_1.petscbin
     source /etc/profile.d/modules.sh
-    module load system/ng-dgx nvhpc-hpcx/26.3
+    module load system/ng-dgx nvhpc-hpcx
     mpirun -np "${n_ranks}" \
       "${ARTIFACT}" -f "${DATA}" -pc_type gamg -pc_gamg_square_graph 0 \
       -mat_type aijcusparse \

From aaebcfe0f20ba864ecffb7e02e30dff4098a49c7 Mon Sep 17 00:00:00 2001
From: William Dawson <william.dawson@riken.jp>
Date: Wed, 5 Aug 2026 23:59:37 +0900
Subject: [PATCH 3/5] Enable Fugaku for petsc-gmres [code:petsc-gmres]

The original ra000009 group storage attempt (/vol0002) had a 0-byte hard
quota limit -- true across every group tried (ra000009, ra250029,
hp250291), and independent of the "0k" shown by lfs quota, which turned
out to mean "unlimited" on some volumes and "hard-enforced zero" on
others. /vol0005 (also already covered by this app's FJ queue.csv
GFSCACHE declaration, alongside 0002-0004) accepted the write under
ra250029 with plenty of headroom (only 64K/4.9TB used beforehand).

Validated end to end on real hardware: a 4-rank run against the
re-staged data succeeded, solve_time_s=55.50, relative L2=0.03154 --
matches the earlier-measured 4-rank baseline exactly.

All three systems (RIKYU, Fugaku, RC_DGXSP) are now enable=yes.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---
 programs/petsc-gmres/README.md |  2 +-
 programs/petsc-gmres/list.csv  |  2 +-
 programs/petsc-gmres/run.sh    | 10 +++++-----
 3 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/programs/petsc-gmres/README.md b/programs/petsc-gmres/README.md
index deff6c8..4af8baf 100644
--- a/programs/petsc-gmres/README.md
+++ b/programs/petsc-gmres/README.md
@@ -60,7 +60,7 @@ build/run time):
 | system | path |
 |---|---|
 | RIKYU | `/data1/rkp00015/benchkit-data/petsc-gmres/audikw_1.petscbin` |
-| Fugaku | *(not yet staged — group-storage quota exhausted on the volumes covered by this repo's `FJ` queue.csv `GFSCACHE` declaration; `Fugaku` row is `enable=no` in `list.csv` until this is resolved)* |
+| Fugaku | `/vol0005/data/ra250029/benchkit-data/petsc-gmres/audikw_1.petscbin` — `/vol0002` (the volume the first attempt used) turned out to have a 0-byte hard quota limit for every group tried, `/vol0005` didn't; this app's `FJ` queue.csv template already declares `GFSCACHE` for `/vol0002:/vol0003:/vol0004:/vol0005`, so no per-app `-x` handling was needed once the right volume was found |
 | RC_DGXSP | `/home/users/william.dawson/benchkit-data/petsc-gmres/audikw_1.petscbin` (this system has no separate group-storage tier, so — unlike RIKYU/Fugaku — this lives under `$HOME`) |
 
 To re-stage on a system with an existing PETSc install: download
diff --git a/programs/petsc-gmres/list.csv b/programs/petsc-gmres/list.csv
index 4fb47bb..5b9095c 100644
--- a/programs/petsc-gmres/list.csv
+++ b/programs/petsc-gmres/list.csv
@@ -1,4 +1,4 @@
 system,enable,nodes,numproc_node,nthreads,elapse
 RIKYU,yes,1,4,1,0:15:00
-Fugaku,no,1,48,1,0:15:00
+Fugaku,yes,1,48,1,0:15:00
 RC_DGXSP,yes,1,1,1,0:15:00
diff --git a/programs/petsc-gmres/run.sh b/programs/petsc-gmres/run.sh
index 5783e6a..aa71fb4 100644
--- a/programs/petsc-gmres/run.sh
+++ b/programs/petsc-gmres/run.sh
@@ -39,11 +39,11 @@ case "${system}" in
       > "${logfile}" 2>&1 || true
     ;;
   Fugaku)
-    # TODO: stage audikw_1.petscbin on Fugaku group storage and fill in
-    # DATA (see README.md's "Staging the data" section for the blocker
-    # hit doing this -- group quota exhausted on the volumes covered by
-    # this repo's own FJ queue.csv GFSCACHE declaration).
-    DATA=/vol0002/data/ra000009/benchkit-data/petsc-gmres/audikw_1.petscbin
+    # /vol0002 is at quota (0 byte hard limit -- true for every group
+    # tried), but /vol0005 isn't; this repo's FJ queue.csv template
+    # already declares GFSCACHE for both (and /vol0003, /vol0004), so no
+    # extra -x PJM_LLIO_GFSCACHE handling is needed here.
+    DATA=/vol0005/data/ra250029/benchkit-data/petsc-gmres/audikw_1.petscbin
     module load lang/tcsds-1.2.43
     module load LLVM/llvmorg-22.1.0
     mpiexec -n "${n_ranks}" \

From b4597e620b3848fc5211eda634ad4878085dbd4d Mon Sep 17 00:00:00 2001
From: William Dawson <william.dawson@riken.jp>
Date: Thu, 6 Aug 2026 00:46:08 +0900
Subject: [PATCH 4/5] Fix build.sh: anchor src/GMRES-PETSc.cpp to script
 location, not caller cwd [code:petsc-gmres]

Found by actually running this through BenchKit's own invocation
convention (bash programs/petsc-gmres/build.sh <system> from the repo
root) rather than continuing to trust a hand-replication of its
commands: BenchKit invokes build.sh with $PWD = repo root, not
programs/petsc-gmres/, so the bare "src/GMRES-PETSc.cpp" reference in
all three system cases never resolved -- "cannot open source file".
$PWD-relative paths elsewhere (PETSC_DIR, ARTIFACT_DIR, bk_fetch_source's
own dest) were already correct, since those are meant to live at the
repo root; only this app's own vendored source needed a different
anchor. Fixed with a BASH_SOURCE-derived APP_DIR.

Full real-pipeline validation on RIKYU after this fix: real
bk_fetch_source, real build.sh producing a working artifact, real
scripts/test_submit.sh correctly parsing list.csv/system.csv and
constructing the sbatch command, a real sbatch-launched run.sh producing
a correct FOM line, and scripts/result.sh turning it into valid Result
JSON with the real PETSc commit hash recorded. This is now confirmed
against BenchKit's actual machinery end to end, not just a hand-replica
of its commands.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---
 programs/petsc-gmres/build.sh | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)

diff --git a/programs/petsc-gmres/build.sh b/programs/petsc-gmres/build.sh
index bc4b358..c975a0a 100644
--- a/programs/petsc-gmres/build.sh
+++ b/programs/petsc-gmres/build.sh
@@ -3,6 +3,15 @@ set -euo pipefail
 
 system="$1"
 
+# BenchKit invokes this as `bash programs/petsc-gmres/build.sh <system>`
+# from the repo root, not from inside this directory -- $PWD is the repo
+# root throughout (matching PETSC_DIR/ARTIFACT_DIR below, and bk_fetch_source's
+# own convention). src/GMRES-PETSc.cpp is *this script's own* source, so
+# anchor it to the script's location (APP_DIR) instead of assuming a
+# caller cwd -- found by actually running this through BenchKit's own
+# invocation convention rather than just replicating its commands by hand.
+APP_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
 PETSC_REPO="https://gitlab.com/petsc/petsc.git"
 PETSC_TAG="v3.25.2"
 PETSC_DIR="${PWD}/petsc"
@@ -36,7 +45,7 @@ case "$system" in
       make PETSC_DIR="${PETSC_DIR}" PETSC_ARCH="${PETSC_ARCH}" -j8 all
     )
     mpicxx -O3 -I"${PETSC_DIR}/include" -I"${PETSC_DIR}/${PETSC_ARCH}/include" \
-      src/GMRES-PETSc.cpp -o "${ARTIFACT_DIR}/GMRES-PETSc" \
+      "${APP_DIR}/src/GMRES-PETSc.cpp" -o "${ARTIFACT_DIR}/GMRES-PETSc" \
       -Xlinker -rpath="${PETSC_DIR}/${PETSC_ARCH}/lib" \
       -L"${PETSC_DIR}/${PETSC_ARCH}/lib" -lpetsc
     ;;
@@ -67,7 +76,7 @@ case "$system" in
       make PETSC_DIR="${PETSC_DIR}" PETSC_ARCH="${PETSC_ARCH}" all
     )
     mpiclang++ -O3 -I"${PETSC_DIR}/include" -I"${PETSC_DIR}/${PETSC_ARCH}/include" \
-      src/GMRES-PETSc.cpp -o "${ARTIFACT_DIR}/GMRES-PETSc" \
+      "${APP_DIR}/src/GMRES-PETSc.cpp" -o "${ARTIFACT_DIR}/GMRES-PETSc" \
       "$(grep '^PETSC_WITH_EXTERNAL_LIB' "${PETSC_DIR}/${PETSC_ARCH}/lib/petsc/conf/petscvariables" | cut -d= -f2-)"
     ;;
   RC_DGXSP)
@@ -91,7 +100,7 @@ case "$system" in
       make PETSC_DIR="${PETSC_DIR}" PETSC_ARCH="${PETSC_ARCH}" -j8 all
     )
     mpicxx -O3 -I"${PETSC_DIR}/include" -I"${PETSC_DIR}/${PETSC_ARCH}/include" \
-      src/GMRES-PETSc.cpp -o "${ARTIFACT_DIR}/GMRES-PETSc" \
+      "${APP_DIR}/src/GMRES-PETSc.cpp" -o "${ARTIFACT_DIR}/GMRES-PETSc" \
       -Xlinker -rpath="${PETSC_DIR}/${PETSC_ARCH}/lib" \
       -L"${PETSC_DIR}/${PETSC_ARCH}/lib" -lpetsc
     ;;

From bd752602bc4f5e1e89ca2510464be96aa1c33e3d Mon Sep 17 00:00:00 2001
From: William Dawson <william.dawson@riken.jp>
Date: Thu, 6 Aug 2026 08:59:55 +0900
Subject: [PATCH 5/5] Fix two more real bugs found by full-pipeline Fugaku
 validation [code:petsc-gmres]

Ran build.sh + a real pjsub-submitted run.sh from a genuinely clean
checkout (no pre-existing petsc/, artifacts/, or results/ -- explicitly
not reusing any earlier manually-built state), matching the same rigor
already applied to RIKYU. Found two more real bugs, both only visible
through the real invocation path, not by reading the script:

1. build.sh's Fugaku case had the PETSC_WITH_EXTERNAL_LIB link-flags
   string wrapped in double quotes, passing the whole multi-flag string
   as one malformed argument to the linker instead of word-splitting it
   -- PETSc itself built fine, only the final app-compile step failed.

2. run.sh's Fugaku DATA path used /vol0005, the "resolved" canonical
   path a filesystem tool reported when the data was staged -- but that
   path doesn't resolve from a real compute-node job; /vol0500 (the
   alias actually used to stage it) does. The exact same class of
   surprise as the earlier Rikyu APP_DIR bug: trust what actually runs,
   not what a tool reports as canonical.

Full pipeline now confirmed on Fugaku from a clean checkout: real
bk_fetch_source, real build.sh producing a working artifact, a real
pjsub-submitted run.sh producing a correct FOM (9.17s, matching the
already-measured 48-rank baseline), and scripts/result.sh producing
valid Result JSON with the same PETSc commit hash as RIKYU's (confirms
the pinned v3.25.2 tag resolves identically across systems).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---
 programs/petsc-gmres/README.md |  2 +-
 programs/petsc-gmres/build.sh  | 10 +++++++++-
 programs/petsc-gmres/run.sh    |  6 +++++-
 3 files changed, 15 insertions(+), 3 deletions(-)

diff --git a/programs/petsc-gmres/README.md b/programs/petsc-gmres/README.md
index 4af8baf..71793e3 100644
--- a/programs/petsc-gmres/README.md
+++ b/programs/petsc-gmres/README.md
@@ -60,7 +60,7 @@ build/run time):
 | system | path |
 |---|---|
 | RIKYU | `/data1/rkp00015/benchkit-data/petsc-gmres/audikw_1.petscbin` |
-| Fugaku | `/vol0005/data/ra250029/benchkit-data/petsc-gmres/audikw_1.petscbin` — `/vol0002` (the volume the first attempt used) turned out to have a 0-byte hard quota limit for every group tried, `/vol0005` didn't; this app's `FJ` queue.csv template already declares `GFSCACHE` for `/vol0002:/vol0003:/vol0004:/vol0005`, so no per-app `-x` handling was needed once the right volume was found |
+| Fugaku | `/vol0500/data/ra250029/benchkit-data/petsc-gmres/audikw_1.petscbin` — `/vol0002` (the volume the first attempt used) turned out to have a 0-byte hard quota limit for every group tried, `/vol0005` didn't; this app's `FJ` queue.csv template already declares `GFSCACHE` for `/vol0002:/vol0003:/vol0004:/vol0005`, so no per-app `-x` handling was needed once the right volume was found. Use `/vol0500`, not `/vol0005`, in an actual path — `/vol0005` is what a filesystem tool reports as the "resolved" location, but it isn't what resolves from a real compute-node job |
 | RC_DGXSP | `/home/users/william.dawson/benchkit-data/petsc-gmres/audikw_1.petscbin` (this system has no separate group-storage tier, so — unlike RIKYU/Fugaku — this lives under `$HOME`) |
 
 To re-stage on a system with an existing PETSc install: download
diff --git a/programs/petsc-gmres/build.sh b/programs/petsc-gmres/build.sh
index c975a0a..6573a35 100644
--- a/programs/petsc-gmres/build.sh
+++ b/programs/petsc-gmres/build.sh
@@ -75,9 +75,17 @@ case "$system" in
         --PETSC_ARCH="${PETSC_ARCH}"
       make PETSC_DIR="${PETSC_DIR}" PETSC_ARCH="${PETSC_ARCH}" all
     )
+    # PETSCLIB is deliberately unquoted below: it's a space-separated list
+    # of separate -L/-Wl,-rpath/-l flags, not one path -- quoting it (an
+    # earlier version of this script did) passes the whole thing as a
+    # single malformed argument to the linker instead of word-splitting
+    # it into individual flags. Found the same way as the APP_DIR fix
+    # above: by actually running this build, not just reading it.
+    PETSCLIB="$(grep '^PETSC_WITH_EXTERNAL_LIB' "${PETSC_DIR}/${PETSC_ARCH}/lib/petsc/conf/petscvariables" | cut -d= -f2-)"
+    # shellcheck disable=SC2086
     mpiclang++ -O3 -I"${PETSC_DIR}/include" -I"${PETSC_DIR}/${PETSC_ARCH}/include" \
       "${APP_DIR}/src/GMRES-PETSc.cpp" -o "${ARTIFACT_DIR}/GMRES-PETSc" \
-      "$(grep '^PETSC_WITH_EXTERNAL_LIB' "${PETSC_DIR}/${PETSC_ARCH}/lib/petsc/conf/petscvariables" | cut -d= -f2-)"
+      $PETSCLIB
     ;;
   RC_DGXSP)
     # GPU build -- audikw_1 is solved on-GPU here (1 rank/GPU), matching
diff --git a/programs/petsc-gmres/run.sh b/programs/petsc-gmres/run.sh
index aa71fb4..31fab96 100644
--- a/programs/petsc-gmres/run.sh
+++ b/programs/petsc-gmres/run.sh
@@ -43,7 +43,11 @@ case "${system}" in
     # tried), but /vol0005 isn't; this repo's FJ queue.csv template
     # already declares GFSCACHE for both (and /vol0003, /vol0004), so no
     # extra -x PJM_LLIO_GFSCACHE handling is needed here.
-    DATA=/vol0005/data/ra250029/benchkit-data/petsc-gmres/audikw_1.petscbin
+    # /vol0500 (not /vol0005 -- the "resolved" path fs_mkdir reported when
+    # this was staged) is what actually resolves from a compute-node job;
+    # found by testing the real run.sh in a real job, not by trusting the
+    # canonical-looking path a filesystem tool reported.
+    DATA=/vol0500/data/ra250029/benchkit-data/petsc-gmres/audikw_1.petscbin
     module load lang/tcsds-1.2.43
     module load LLVM/llvmorg-22.1.0
     mpiexec -n "${n_ranks}" \