Skip to content

Faster neighbour sums on CPU and GPU, multi-GPU support for the openaccPTM plugin. - #1635

Open
carlocamilloni wants to merge 14 commits into
masterfrom
openaccPTM-multigpu
Open

carlocamilloni wants to merge 14 commits into
masterfrom
openaccPTM-multigpu

Conversation

@carlocamilloni

@carlocamilloni carlocamilloni commented Nov 15, 2026 •

Copy link
Copy Markdown
Member

Summary

Faster neighbour sums on CPU and GPU, and multi-GPU support for the openaccPTM plugin.

  • COORDINATIONNUMBER and Q1/Q3/Q4/Q6 (and what is built on them: LOCAL_Q*, LOCAL_AVERAGE, …) sum over the neighbours of each atom in one pass over the linked cells. They no longer build the contact matrix or the matrices of the spherical harmonics. The same kernels run on the CPU (OpenMP) and on the GPU (openaccPTM, USEGPU).
  • openaccPTM plugin: MPI support (MULTIGPU), the GPU chosen by node-local rank, and one-pass force gathering by default.
  • One site against many atoms (e.g. an ion against all the waters): handled without cells in the fused actions.

Values are unchanged. Forces change only in the last digits (relative 1e-14), and with SPECIES they no longer depend on the number of OpenMP threads.

Overall, on a Leonardo Booster node (32-core Xeon 8358, A100), 20000 atoms:

ms/step before (matrix chain) this PR
COORDINATIONNUMBER, 32 CPU cores 26.7 3.96
COORDINATIONNUMBER, 1 A100 77.9 3.57
Q6, 32 CPU cores 441 15.7
Q6, 1 A100 759 12.6

See "Performance" below for the baselines and for a same-build comparison.

What changes

1. Fused neighbour sums (core, src/symfunc/NeighborSums.{h,cpp})

  • New actions: COORDINATIONNUMBER_SUMS, Q1/Q3/Q4/Q6_SUMS, Qn_DENOM_SUMS, Qn_NORM. They are created by the shortcuts as suffixed variants (not meant to be written by hand).
  • What the shortcuts now do. Without MASK, COORDINATIONNUMBER and Q* compute the sums per atom directly from the linked cells:
    • cells of half the cutoff, as in LinkCellsView;
    • for Q*: the coordination numbers (q6_denom) in the same pass as the sums of the spherical harmonics;
    • forces contracted pair by pair (no per-pair arrays of derivatives);
    • the norms (q6_norm) computed by one action instead of a COMBINE of 2(2l+1) vectors.
  • Forces:
    • with SPECIES, each atom gathers its force from its own neighbours: no atomics, and the forces do not depend on the number of threads;
    • with SPECIESA/SPECIESB, the column forces go to per-thread buffers that are reduced in a fixed order.
  • Labels kept. cn, q6_sp.* (exactly the 2(2l+1) components), q6_denom and q6_norm keep their labels and shapes.
    • The matrix actions (cn_mat, q6_mat, q6_sh, …) are still created and are computed only if another action uses them.
    • q6_norm2 (the COMBINE) is still created, dormant.
  • Few rows (e.g. SPECIESA=1 SPECIESB=<all waters>). Up to max(4, threads) rows on the CPU, or 512 rows on the GPU:
    • no cells are built; the columns are split into blocks between the threads (GPU threads, or MPI ranks);
    • partial sums are reduced in a fixed order, so the result does not depend on the threads.
  • MPI: rows split between the ranks (columns with few rows); SERIAL is accepted.
  • Fallback: without D_MAX, or with a box smaller than 3 cutoffs, all pairs are computed with Pbc::distance.

2. openaccPTM plugin

  • Fused actions on the GPU. With USEGPU, COORDINATIONNUMBER and Q* use the same row kernels on the GPU (*_SUMSACC):
    • one GPU thread per row with the cells, or column blocks with few rows;
    • the cells are built on the host and uploaded;
    • no OpenMP region in the code the plugin instantiates. Two OpenMP runtimes, nvomp from nvc++ and libgomp from the core, made Q6 about 7× slower; this is now avoided.
  • MPI:
    • by default only the first rank computes the GPU actions on its GPU and the result is shared (the previous behaviour);
    • new MULTIGPU splits the tasks between the ranks, each on its own GPU;
    • SERIAL is accepted by all the shortcuts that take USEGPU (DISTANCES, Q*, LOCAL_Q*, LOCAL_AVERAGE, TETRAHEDRAL-like);
    • the two-pass gathering works with MPI.
  • GPU choice. When the MD code does not assign a GPU and ACC_DEVICE_NUM is not set, a rank uses GPU (node-local rank mod number of GPUs). The node-local rank is read from the OpenMPI, MVAPICH, Intel MPI, PMI, PALS, Flux or SLURM variables. The choice is written in the log.
  • Force gathering of matrices on the GPU is one pass by default. AVOID_THREAD_GATHER gives the previous two-pass, low-memory version.
  • COORDINATIONNUMBER / COORDINATION_MOMENTS accept USEGPU.
  • Q with USEGPU:* the norm is computed on the CPU (it was slower on the GPU).
  • Bug fixed: with VMEAN/VSUM the GPU flags were passed to a COMBINE of scalars, which rejects them.

3. Docs

  • COORDINATIONNUMBER and CONTACT_MATRIX: notes on the fused path and the few-rows case.
  • Plugin README: MPI, the GPU choice, the fused actions, timings.
  • CHANGES/v2.11.md: new items for all of the above.

How to get the previous behaviour

change previous behaviour
Fused COORDINATIONNUMBER / Q* Used automatically with MASK and for COORDINATION_MOMENTS. To force it elsewhere, write the chain out: CONTACT_MATRIX + MATRIX_VECTOR_PRODUCT for the coordination numbers; for Q*, the CONTACT_MATRIX COMPONENTS + SPHERICAL_HARMONIC + MATRIX_VECTOR_PRODUCT chain of the expanded shortcut (e.g. plumed driver --parse-only, or the manual). There is no keyword to switch off the fused path.
Q* norm (Qn_NORM) and denominator in the same pass Only through the chain written out, as above.
USEGPU without the plugin loaded Unchanged: the matrix chain with USEGPU, as before.
GPU actions with MPI Unchanged by default (first rank computes). MULTIGPU is opt-in.
GPU choice per local rank Set ACC_DEVICE_NUM (or let the MD code assign the GPU) to choose the GPU explicitly.
One-pass force gathering on the GPU AVOID_THREAD_GATHER gives the two-pass, low-memory version (the old GPU default; about two times slower, uses less memory).
Few rows without cells (fused actions) No switch. The threshold is internal (fewRows(): CPU max(4, threads), GPU 512). Results agree with the cells path to 1e-13.

Output that changes:

  • Plugin references:
    • rt-symfunc-nbonds(-one)-q6: the virial, which is now exactly the CPU one;
    • the two-pass tests now ask for AVOID_THREAD_GATHER explicitly.
  • symfunc/rt-parse-only: the expansion text of the Q* shortcuts (q1_norm is now Q1_NORM).

No other reference changed.

Performance

How to read the numbers.

  • Time per step (ms) of plumed benchmark, forward plus backward, averaged without the slowest step.
  • Random atoms at 74/nm³; RATIONAL R_0=0.3, D_MAX=0.6.
  • Leonardo Booster node: 32-core Xeon 8358, A100, gcc 12.2 core build, nvc++ plugin.

Same build, fused vs matrix chain (2-core VM, -O2; the chain written out by hand; includes the cells of half the cutoff already in master)

input threads matrix chain fused, first version + Q* denominator and norm in the same pass
COORDINATIONNUMBER 2000 1 / 2 12.2 / 8.05 7.80 / 4.21 7.20 / 3.76
COORDINATIONNUMBER 20000 1 / 2 159 / 114 95.3 / 49.2 71.7 / 37.8
Q6 2000 1 / 2 122 / 92.3 50.1 / 29.3 25.1 / 13.1
Q6 20000 1 / 2 2032 / 1487 639 / 367 285 / 146
COORDINATIONNUMBER 64 1 / 2 0.29 / 0.19 0.23 / 0.15 —

The columns come from successive sessions on the same VM, which is noisy (±10%). The last column is the code before the linked-cells and few-rows changes, which are measured on Leonardo below.

Leonardo (32 CPU cores / 1 A100 with 8 host threads)

The "now" columns include the faster linked cells of linkcells-threaded (already in master): they took COORDINATIONNUMBER 20000 on 32 cores from 4.70 to 3.96 ms.

input CPU before CPU now GPU before GPU now
COORDINATIONNUMBER 20000 26.7 3.96 (6.7×) 77.9 3.57 (22×)
COORDINATIONNUMBER 100000 164 22.4 (7.3×) 576 16.1 (36×)
Q6 20000 441 15.7 (28×) 759 12.6 (60×)
Q6 100000 2281 90.4 (25×) 4047 57.7 (70×)
  • The "before" baselines:
    • CPU: the matrix chain of the branch base (round 3, before master gained the cells of half the cutoff, which by themselves made COORDINATIONNUMBER about 1.4× faster);
    • GPU: the per-action offload of the matrix chain (round 1).
    • Against today's master the CPU gain is smaller than the column above, roughly by that 1.4× for COORDINATIONNUMBER. The same-build table above is the fair comparison.
  • Before this PR the GPU was slower than the CPU (0.3× for COORDINATIONNUMBER, 0.45× for Q6). Now one A100 beats 32 cores in all four cases, by 1.1–1.6×.
  • MULTIGPU: COORDINATIONNUMBER with 100000 atoms, 4 GPUs, 11.9 ms vs 20.5 on one (measured with an earlier kernel). No gain at 20000 atoms, so it stays opt-in.

One site against many atoms (one atom or a few against 100000 water oxygens)

input before now
COORDINATIONNUMBER SPECIESA=1 (2-core VM, 1 / 2 threads) 7.38 / 5.43 2.71 / 1.99
COORDINATIONNUMBER, 16 rows (Leonardo, 32 cores / A100) 4.03 / 4.76 2.47 / 2.06
COORDINATIONNUMBER, 256 rows (Leonardo, 32 cores / A100) 5.98 / 4.89 4.81 / 3.51
  • For a single scalar, COORDINATION with NLIST is still the fastest (0.59 ms with NL_STRIDE=10). The docs say so.

Linked cells alone (Leonardo, µs per build, 20000 / 100000 atoms; from linkcells-threaded, already in master)

threads before now
1 860 / 4678 398 / 2320
8 297 / 1519 115 / 610
32 299 / 1251 89 / 391

The old code stopped improving at 4 threads, because placing the atoms in the cells was serial. With the new cells, COORDINATIONNUMBER with 20000 atoms on 32 cores goes from 4.70 to 3.96 ms per step.

GPU force gathering (the matrix chain on the GPU, COORDINATIONNUMBER 20000 written out)

  • Two passes: 75.8 ms per step. One pass (the new default): 51.2 ms.

Tests

  • New core regtests: symfunc/rt-neighbor-sums, -mpi, -small, -parts, -few-rows, -few-rows-mpi.
    • The references were computed with the matrix chain written out (chain.dat in each directory) and agree with the fused output at the regtest precision.
    • Cases covered: triclinic boxes, overlapping SPECIESA/B, no D_MAX, small box, 2 threads, 3 MPI ranks, forces on parts of the Q* outputs.
  • New plugin regtests (targetGPU): rt-GPU-mpi-nogather, rt-GPU-mpi-serial, rt-GPU-mpi-shortcuts, rt-GPU-neighbor-sums.
  • Checks run:
    • all core regtests with PLUMED_NUM_THREADS=2, both with gcc+OpenMP+MPI and with clang without OpenMP;
    • plugin regtests with nvc++ on Leonardo GPUs;
    • fused vs chain at full precision: values to 1e-13 relative, forces to 5e-13 of the largest force;
    • GitHub CI, including linux-nvhpc (the nvc++ build of the plugin) and CodeQL.

@codecov

codecov Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 88.99%. Comparing base (422109d) to head (77435aa).

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1635      +/-   ##
==========================================
+ Coverage   88.96%   88.99%   +0.02%     
==========================================
  Files         652      655       +3     
  Lines       81934    82471     +537     
==========================================
+ Hits        72890    73391     +501     
- Misses       9044     9080      +36     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@carlocamilloni carlocamilloni changed the title openaccPTM: split the tasks of the GPU actions between the MPI processes openaccPTM: make it faster than CPU for matrices Nov 16, 2026
@carlocamilloni
carlocamilloni changed the base branch from master to shortcuts-first-pass November 16, 2026 12:51
@carlocamilloni
carlocamilloni changed the base branch from shortcuts-first-pass to master November 16, 2026 15:16
@carlocamilloni
carlocamilloni force-pushed the openaccPTM-multigpu branch 2 times, most recently from 6223e6b to 185eb76 Compare November 17, 2026 09:57
@carlocamilloni carlocamilloni changed the title openaccPTM: make it faster than CPU for matrices Faster neighbour sums on CPU and GPU, multi-GPU support for the openaccPTM plugin, and faster linked cells. Nov 17, 2026
@gtribello

Copy link
Copy Markdown
Member

Hi @carlocamilloni and @GiovanniBussi

I am a bit concerned about all the changes that you are making so rapidly using this AI stuff. I am grateful that you are working to make the code faster but I am also frankly a bit frustrated that you haven't had a conversation with me about the prompts that you are giving the AI engines. In other words, I am frustrated that you aren't discussing what you are trying to achieve with me before you set the agents off to make the changes to the code.

The things that you are changing are pieces of codes that have taken me hours to write and I am worried that they will become unrecognisable to me and that only the AI will be able to adjust them in the future. Furthermore, it is frankly tiring to be asked to review a bunch of AI slop text every couple of days with little to no context about what you were trying to achieve. In fact I don't even know what you are looking for me to do when you ask me for a review. On Monday you sent me some code to review, I had a brief look at it, thought to myself, "I will reply later." However, by the time I set out to respond the code had already been merged so any thoughts I had on it were moot.

As far as I can tell, one of the big things in this change is that you add new actions so you can avoid calculating the contact matrices and then multiplying them by vectors as I did. In other words, you take these two actions:

c1: CONTACT_MATRIX GROUP=@natoms SWITCH={RATIONAL R_0=0.1}
ones: ONES SIZE=@natoms
v: MATRIX_VECTOR_PRODUCT ARG=c1,ones 

I know that replacing three actions with one action will likely make the code faster. However, my objective with all the changes that I made to this code when I worked on v2.10 was to make it easier to understand how the calculations are performed. When you introduce combined actions back in to make the code faster you are removing the work that was done to improve comprehensibility.

I believe that comprehensibility is more important than performance, especially given AI. If a person knows exactly what they are trying to do then they will be able to optimse the code to that particular purpose. This code, optimised to the particular application the person is working on, will always be faster than the sort of general purpose code that appears in PLUMED. In addition, AI makes it easy to write this type of highly specialised code. If AI makes writing this specialised and optimal code easy what is the purpose of the slow unoptimised code in PLUMED?

To be clear, I am not saying that I am not willing to change the code that I have written. I am also not saying that you have to agree with my view that comprehensibility is more important than performance. What I would like to do is slow down and have some agreed vision for what we are trying to achieve here. For example, I have always wondered whether, now that we have disentangled what CVs such as Q6 and so on are computing, the code wouldn't be faster (and easier to maintain) if we just used a library like torch to do these calculations for us. In other words, I think there may be merit in deleting much of the code that I have written and using a library instead.

I would also note that when Claude is adjusting the documentation it isn't using $$ around the equations. There is thus a danger that the manual ends up with a mixture of LaTeX equations and raw LaTeX.

Professional science has, at times, felt to me like a bit of a meanigless pissing up the wall contest. I have had experiences of meaningful human connection when doing science. I worry, however, that this AI stuff is enhancing everyone's worse instincts and that these moments of meaningful human connection will consequently become even more rare. That is not to say AI shouldn't be used. It is to say that you have to use it in a way that is sympathetic to the people that you are notionally working with. After all, when you move fast and break things the things that get broken are always people.

@GiovanniBussi

Copy link
Copy Markdown
Member

I agree we should definitely slow down large changes.

For me it took quite some time to understand the changes proposed by me via AI

My experience so far is that it works very well for localized changes, where it can do systematic improvements that make the code more manageable for the compiler and hence faster with very little maintenaibility cost

For larger architectural changes, my experience is that there are an infinite number of ways to do things and it takes a lot of human time and iterations to find the best compromise in terms of performance and readability of the code.

I think we need a better plan

@carlocamilloni

carlocamilloni commented Nov 18, 2026 •

Copy link
Copy Markdown
Member Author

@gtribello @GiovanniBussi I agree and disagree :-) I agree that we are changing a lot and very rapidly and i agree that we can talk about these large changes before merging them. Most other changes were not changing the logic of the code itself.

Here I pushed the code rewriting to keep the original flexibility but just leave it optional, indeed the code has all the previous capabilites and also still explains the actual internal logic of the actions. To get to this implementation i first began asking to get as much performance as possible without fusing actions but the final analysis was that it was not possible, in particular on GPU, because all the timings pointed to data transfer and memory management, with essentially no cost on the calculation itself (the current main already includes some optimisations) and that this was particularly relevant on the GPU, the analysis suggested a potential to get a x10-x100 speed up by fusing and so i tried. Also note that the way is implemented does not change the logic of the plugin as well, where each action is first implemented on CPU and then only specialised for GPU. Consider that to get with this PR I also run myself a lot of simulations on leonardo to get timings at multple stages.

i disagree on other points, I do not learn how MD simulations work by reading GROMACS source code, i use gromacs becasuse it works and it is efficient and I may learn by chaning parameters not the code. I do not think that people use plumed source files to learn the inner working of enhanced sampling or order parameters but they can learn playing with actions. I agree that the ability to implement complex CVs as combination of simpler one is very useful and the manual and the log can still explain what is the set of operations performed and one can still do them in the expanded manner. From this point of view your original idea is still present. But at the same time plumed to me is the same as gromacs but for enhanced sampling and CVs, a state-of-the-art production tool. From this point of view I hope you agree that a GPU implementation that is twice as slow than the CPU is not very useful and that an onverall implementation that can be 10-100 times faster that the previous is preferable. The numbers for secondary structure CVs are even more impressive that this one (they are not yet updated because i still need one more round of validation).

documentation latex format can be easily fixed. AI learns by doing, most good practices are already saved, others are added overtime.

@gtribello

Copy link
Copy Markdown
Member

@carlocamilloni

I tried to be calm and reasonable this morning but your last message has pissed me off. Please don't straw man what I am saying. I obviously do not think that people learn how these CVs work by reading the PLUMED source code. Similarly, I took pains in my last message to say that I appreciated the work that you are doing to speed up the code. I am not so vain that I think we should have an artificially slow version of the CVs implemented.

What I do think is that we do not need a faster implementation of Q6 by next week. I do not think that this is so urgent that there isn't time to discuss with me what you are trying to achieve during one of our regular Monday meetings. Heck if you can't wait that long at least do me the curtesy of writing me an email to explain before you start. That is not hard - this is basic collaboration stuff.

Heck, if we aren't in such a rush about getting things done and if take the time to have a conversation we might realise that there is something in what we are doing that is worth putting in an article.

@carlocamilloni

Copy link
Copy Markdown
Member Author

@gtribello i am sorry that my message pissed you off, that was not my intention at all. As you wrote for your side my intention was the same, was that of clarifying the rational of the work with the goal of improving performance while preserving your plan. We can talk on Monday that is easier!

carlocamilloni and others added 13 commits November 18, 2026 20:20
With MPI each process computes a contiguous block of the active tasks on its
own GPU and the value stashes and the forces are summed, as in the CPU task
manager; with SERIAL (or one process) the first process computes everything
and broadcasts the results, as before. Only the part of the value stash
touched by the block is cleared, uploaded and downloaded.

- The low memory force gathering (AVOID_THREAD_GATHER, the default on the GPU)
  works with MPI: in the second pass only the first process keeps the forces
  of the first one; a process without tasks in the first pass uploads the
  positions in the second.
- GPU choice: without an MD code and with several GPUs on the node (e.g.
  mpirun -np 4 plumed driver), process i of the node uses GPU i % nGPUs
  (local rank from the launcher environment), unless ACC_DEVICE_NUM is set.
- Shortcuts: DISTANCES, Q1-Q6, LOCAL_Q*, LOCAL_AVERAGE, TETRAHEDRAL-like
  shortcuts accept SERIAL and pass it with USEGPU; COORDINATIONNUMBER and
  COORDINATION_MOMENTS accept USEGPU; Q* and LOCAL_AVERAGE put the contact
  matrix on the GPU too (as LOCAL_Q* already did).
- Regtests rt-GPU-mpi-nogather (4 processes, a process without rows),
  rt-GPU-mpi-serial and rt-GPU-mpi-shortcuts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016vwxtUCrVzSZSsREbiW4jL
…over the neighbours without matrices

From the Leonardo benchmarks (A100): splitting the tasks between the GPUs was
slower for every input tested, the two-pass force gathering doubled the
backward pass, and the matrices copied between CPU and GPU dominate
COORDINATIONNUMBER and Q6.

- With MPI the GPU actions are computed by the first process again; MULTIGPU
  (also accepted by the shortcuts that take USEGPU) splits their tasks.
- Forces of the matrices: one pass (atomics) is the default on the GPU too;
  AVOID_THREAD_GATHER asks for the low memory version. DO_THREAD_GATHER is
  kept for compatibility.
- New plugin actions COORDINATIONNUMBER_SUMSACC and Q1-Q6_SUMSACC (and
  Qn_DENOM_SUMSACC, src/NeighborSums.cpp): sums over the neighbours of each
  atom on the GPU in one pass, cells in scaled coordinates built on the CPU
  (any box), minimum image with Pbc, forces with atomics. Used by
  COORDINATIONNUMBER and Q1-Q6 with USEGPU (not with MASK, nor for
  COORDINATION_MOMENTS). Same results as the CPU (double precision).
- SphericalHarmonic: setup() for a given l, functions in the header inline.
- Regtest rt-GPU-neighbor-sums (triclinic box, 2 and 3 cells, no D_MAX,
  SPECIESA/B, MULTIGPU on 3 processes); the explicit Q6 chain test now
  runs SPHERICAL_HARMONIC on the GPU; references of two tests now exact.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016vwxtUCrVzSZSsREbiW4jL
The neighbour loop was a macro whose argument contained OpenACC pragmas,
which are not processed inside macro arguments: nvc++ (linux-nvhpc CI)
saw atomics on invalid expressions and the loop as a routine. The loop
is now written out in the two kernels; the action data is a local copy
on the device (as in the task manager); pair() is an acc routine seq;
epsilon is passed as a firstprivate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016vwxtUCrVzSZSsREbiW4jL
The COMBINE of the 2(2l+1) vectors was slower on the GPU than on the
CPU (Leonardo, q6-20000: 6+3 ms vs 4.5+1.2 ms), since its inputs and
outputs go to the GPU and back at every step. SERIAL is still passed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016vwxtUCrVzSZSsREbiW4jL
With SPECIESA/SPECIESB the forces on the columns are accumulated in one
buffer per thread. The buffers were sized with OpenMP::getNumThreads(),
which without OpenMP still returns PLUMED_NUM_THREADS, while only thread 0
runs: the reduction read the empty buffer of thread 1 and crashed
(symfunc/rt-neighbor-sums, -small, -mpi and rt-q6-subset on the Windows and
the Linux no-OpenMP builds of the CI, which use PLUMED_NUM_THREADS=2).
The threads that run now mark their buffer, and only those are summed;
this also covers an OpenMP team smaller than requested.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016vwxtTud9yy24ZL9dp5N2v
…v variable

The sums are copied to the values with transferStashToValues and the forces
on the values with transferForcesToStash (the sums already have the layout
of the stash): they copy in memory order and with OpenMP. The previous
serial loops over the components called Value::set/getForce per element,
15% of Q6 on 20000 atoms with 32 threads. The sums are zeroed only when
this process does not compute all the rows.

The openaccPTM plugin no longer reads PLUMED_NEIGHBORSUMS_SUBDIVISION: two
cells per cutoff is 25-32% faster than one on the A100 too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016vwxtTud9yy24ZL9dp5N2v
… several threads per row, few rows without cells

- The pair functors add the values of a pair to the sums (addValues) and contract the
  forces pair by pair (contract), without storing the derivatives of all the components:
  for the spherical harmonics the components +m and -m are done together.
- Qn_SUMS also computes the coordination numbers in the same pass over the neighbours;
  Qn_DENOM_SUMS SUMS=label gives them as a vector and passes back the forces on them
  (it applies before the sums). This replaces the second pass (the ContactPairs
  Qn_DENOM_SUMS and the plugin's Qn_DENOM_SUMSACC). Qn_NORM computes the norms of the
  sums in one action; q6_norm2 (COMBINE) is still created and computed only if used.
- The search around each row can be split in parts: groups of the rows of cells around
  it (forEachCandidate/forEachNeighbor with slice, nslices), or blocks of columns without
  cells. The GPU actions use several GPU threads per row (about 131k threads, at most 25
  per row) and sum the parts of each row in order with a second kernel.
- Without GROUP and with few rows (at most 8) the cells are not built: the columns are
  split in blocks shared by the threads (and by the MPI processes), with the minimum
  images computed in blocks (Pbc::apply) on the CPU. The results do not depend on the
  number of threads.
- TEMPORARY, for the Leonardo measurements: PLUMED_NEIGHBORSUMS_SLICES (GPU parts per
  row) and PLUMED_NEIGHBORSUMS_FEWROWS (the few-rows threshold).
- New regtests rt-neighbor-sums-few-rows (-mpi) and rt-neighbor-sums-parts; the
  references agree with the contact matrix written out (chain.dat). rt-parse-only:
  q1_norm is now Q1_NORM (text only).

2-core VM, -O2, ms/step forward+backward, 1/2 threads, before -> after:
Q6 2000 atoms 47.9/24.9 -> 25.1/13.1; Q6 20000 528/270 -> 285/146;
COORDINATIONNUMBER 2000 and 20000 3-5% faster; one atom against 100000 waters
(SPECIESA/SPECIESB) 7.4/5.4 -> 2.7/2.0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016vwxtTq4ZtJbDFt8nrMV3A
…ne site against many atoms

- COORDINATION (and the other users of NeighborList) with NLISTCELLS builds the list with the
  plain search when one of the two groups has at most 8 atoms: sorting all the atoms into
  cells costs more than one distance per pair (equal cost at about 8 atoms). One atom
  against 100000 water oxygens, list updated at every step: 7.9 -> 3.4 ms/step.
- Docs: COORDINATION ("One atom against many atoms"), COORDINATIONNUMBER and CONTACT_MATRIX
  say which input to use for one site against many atoms. CHANGES/v2.11.md updated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016vwxtTq4ZtJbDFt8nrMV3A
…lways use the core’s OpenMP. The plugin now runs no OpenMP code of its own. I’ve noted this as a rule in the progress doc.

GPU: one thread per row when the cells are used. With few rows, the columns are split into blocks between the GPU threads. The splitting of a row’s cells is gone from LinkCellsView.h, which is back to master’s version.
Few-rows cutoffs: on the CPU, 2 × the number of threads (at least 4), as you chose; on the GPU, 512.
Both temporary environment variables are gone, and the plugin README is updated.
… plugin

On one A100 (8 CPU threads) / 32 CPU cores, ms per step: COORDINATIONNUMBER 3.7 / 4.7 (20000 atoms),
17 / 25 (100000); Q6 13 / 17, 59 / 94 (Leonardo, round 8).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016vwxtTq4ZtJbDFt8nrMV3A
…ings

With the cheaper cells (Leonardo, 32 threads, 100000 columns) all the
pairs cost as much as the cells at about 44 rows: 64 rows took 4.03 ms
without cells against ~3.5 with them. The threshold is now the number
of threads (at least 4; unchanged up to 4 threads). CHANGES and the
plugin README have the round-9 timings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016vwxtTq4ZtJbDFt8nrMV3A
@carlocamilloni carlocamilloni changed the title Faster neighbour sums on CPU and GPU, multi-GPU support for the openaccPTM plugin, and faster linked cells. Faster neighbour sums on CPU and GPU, multi-GPU support for the openaccPTM plugin. Nov 18, 2026
runAllTasksACC and applyForcesWithACC took the action data (input_type)
by value, a deep copy on the host, and moved it to the device with
enter/exit data at every call, twice per step. For large action data
this dominated the time of the action: e.g. the 370k segments of the
secondary structure variables of 144 peptides are 45 MB of indices,
and SECONDARY_STRUCTURE_DRMSD_ACC took 14 ms forward and 10 ms
backward per step with 5000 active segments (4 and 1 ms on the CPU).

The task manager now keeps a copy of the action data on the device of
the action and passes the kernels a pointer to it (present(ad[0:1])).
The copy is made and uploaded again only when the data may have
changed: ParallelTaskManager counts the calls of setActionInput and of
the non-const getActionInput (getActionInputGeneration), which is how
actions change their input (e.g. the neighbour lists of the contact
matrix, the regions of the volumes, the bookkeeping of the matrix
products), so those actions still upload it at every step, as before.
The copy is removed from the device when the task manager is
destroyed.

Tested with the plugin built with g++ and the host stub of the OpenACC
runtime (51 targetGPU regtests pass; the stub ignores the data
directives, so the residency itself is only tested with nvc++).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016vwxtV46v5FLckcMEcJrbt

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants