Add new application: petsc-gmres - #87
Draft
william-dawson wants to merge 5 commits into
Draft
Conversation
PETSc KSP (GMRES + GAMG) benchmark: loads the audikw_1 sparse SPD matrix (SuiteSparse GHS_psdef group, 943,695 x 943,695, 77,651,847 nnz) via MatLoad against a pre-converted PETSc binary file, solves Ax=b for a known x, reports relative L2 error and solve wall-time. PETSc is fetched via bk_fetch_source, pinned to v3.25.2. The app's own source is vendored directly in src/ rather than fetched from a separate repo, since its upstream development happens in a private RIKEN-RCCS repository (not fetchable by bk_fetch_source's plain git clone). RIKYU build/run recipe validated end to end on real hardware: PETSc builds and configures cleanly against nvhpc-hpcx, the app compiles and links, and a 4-rank run against the staged matrix produces a correct result (relative L2 0.03154, matching the expected ~0.03-0.04 band). Fugaku and RC_DGXSP cases are included in build.sh/run.sh based on previously-validated recipes on those systems, but their list.csv rows are enable=no pending data staging (see README.md's "Staging the data" section -- Fugaku's group-storage volumes are currently at quota). -pc_gamg_square_graph 0 is required at any rank count above 1 on every system, to avoid a crash in GAMG's aggressive-coarsening graph-squaring step on this matrix's connectivity (a cuSPARSE resource error on GPU, a multi-GB single-allocation OOM on CPU). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Validated end to end on real ng-dgx-m2 hardware (same GB10 Blackwell hardware as the ng-dgx-m1 this app's list.csv actually targets, just idle at the time): PETSc CUDA build (CUDA_ARCH auto-detected as 121, matching GB10's sm_121), matrix conversion (nnz=77,651,847, correct), and a 1-rank GPU run all succeeded -- solve_time_s=7.57, relative L2=0.0443346 (correct, matches the known-good cross-machine baseline). Stages audikw_1.petscbin at a fixed path under $HOME (this system has no separate group-storage tier, unlike RIKYU/Fugaku), flips RC_DGXSP to enable=yes in list.csv, and drops the now-stale nvhpc-hpcx/26.3 version pin in favor of the unversioned module name actually available here. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The original ra000009 group storage attempt (/vol0002) had a 0-byte hard quota limit -- true across every group tried (ra000009, ra250029, hp250291), and independent of the "0k" shown by lfs quota, which turned out to mean "unlimited" on some volumes and "hard-enforced zero" on others. /vol0005 (also already covered by this app's FJ queue.csv GFSCACHE declaration, alongside 0002-0004) accepted the write under ra250029 with plenty of headroom (only 64K/4.9TB used beforehand). Validated end to end on real hardware: a 4-rank run against the re-staged data succeeded, solve_time_s=55.50, relative L2=0.03154 -- matches the earlier-measured 4-rank baseline exactly. All three systems (RIKYU, Fugaku, RC_DGXSP) are now enable=yes. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…er cwd [code:petsc-gmres] Found by actually running this through BenchKit's own invocation convention (bash programs/petsc-gmres/build.sh <system> from the repo root) rather than continuing to trust a hand-replication of its commands: BenchKit invokes build.sh with $PWD = repo root, not programs/petsc-gmres/, so the bare "src/GMRES-PETSc.cpp" reference in all three system cases never resolved -- "cannot open source file". $PWD-relative paths elsewhere (PETSC_DIR, ARTIFACT_DIR, bk_fetch_source's own dest) were already correct, since those are meant to live at the repo root; only this app's own vendored source needed a different anchor. Fixed with a BASH_SOURCE-derived APP_DIR. Full real-pipeline validation on RIKYU after this fix: real bk_fetch_source, real build.sh producing a working artifact, real scripts/test_submit.sh correctly parsing list.csv/system.csv and constructing the sbatch command, a real sbatch-launched run.sh producing a correct FOM line, and scripts/result.sh turning it into valid Result JSON with the real PETSc commit hash recorded. This is now confirmed against BenchKit's actual machinery end to end, not just a hand-replica of its commands. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…:petsc-gmres] Ran build.sh + a real pjsub-submitted run.sh from a genuinely clean checkout (no pre-existing petsc/, artifacts/, or results/ -- explicitly not reusing any earlier manually-built state), matching the same rigor already applied to RIKYU. Found two more real bugs, both only visible through the real invocation path, not by reading the script: 1. build.sh's Fugaku case had the PETSC_WITH_EXTERNAL_LIB link-flags string wrapped in double quotes, passing the whole multi-flag string as one malformed argument to the linker instead of word-splitting it -- PETSc itself built fine, only the final app-compile step failed. 2. run.sh's Fugaku DATA path used /vol0005, the "resolved" canonical path a filesystem tool reported when the data was staged -- but that path doesn't resolve from a real compute-node job; /vol0500 (the alias actually used to stage it) does. The exact same class of surprise as the earlier Rikyu APP_DIR bug: trust what actually runs, not what a tool reports as canonical. Full pipeline now confirmed on Fugaku from a clean checkout: real bk_fetch_source, real build.sh producing a working artifact, a real pjsub-submitted run.sh producing a correct FOM (9.17s, matching the already-measured 48-rank baseline), and scripts/result.sh producing valid Result JSON with the same PETSc commit hash as RIKYU's (confirms the pinned v3.25.2 tag resolves identically across systems). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A PETSc KSP (GMRES + GAMG) benchmark, verified on Rikyu, Fugaku, and R-CCS Cloud's DGX Spark.
[code:petsc-gmres]We have not verified scalability with number of GPUs — the problem size (
audikw_1, 943,695 rows) is likely too small to amortize GAMG's setup-communication cost across multiple GPUs.Performance data
Rikyu (GB200 NVL4), 1-node CPU strong scaling
Rikyu, 2-node CPU strong scaling
Rikyu, GPU (single node)
1 GPU beats multi-GPU — this is the scalability gap noted above, not yet resolved with a larger matrix.
Fugaku (A64FX), 1-node strong scaling (48 ranks/node)
Fugaku, multi-node strong scaling (fully packed, 48 ranks/node)
DGX Spark (GB10, R-CCS Cloud), single GPU
Correctness (relative L2 error, ~0.03–0.04) matched exactly across all of the above.