Skip to content

Secondary structure variables: single actions on CPU and GPU, new defaults for the sheets; openaccPTM improvements - #1641

Closed
carlocamilloni wants to merge 27 commits into
openaccPTM-multigpufrom
acc-resident-actiondata
Closed

carlocamilloni wants to merge 27 commits into
openaccPTM-multigpufrom
acc-resident-actiondata

Conversation

@carlocamilloni

@carlocamilloni carlocamilloni commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Performance with respect to master

Leonardo Booster node (32-core Intel Xeon + one A100), nvc++ 24.5, plumed benchmark with a RESTRAINT on the variables (so the forces
are computed), average time per step in ms without the slowest step. GPU runs use 8 CPU threads. Systems: the amyloid of
regtest/secondarystructure/rt33 (18 peptides of 8 residues, 2520 atoms: amy1) and the same replicated 2x2x2 (144 peptides, 20160 atoms: amy2).

  • alpha: ALPHARMSD (54 / 432 segments);
  • ss: ALPHARMSD + ANTIBETARMSD + PARABETARMSD with STRANDS_CUTOFF=1, DRMSD (about 5500 / 370800 pairs of strands for each sheet);
  • nocut: ANTIBETARMSD with all the pairs (master: no STRANDS_CUTOFF; this PR: STRANDS_CUTOFF=0).
case master CPU 8 thr PR CPU 8 thr master CPU 32 thr PR CPU 32 thr master GPU PR GPU
alpha-amy1 0.09 0.06 0.09 0.06 0.75 0.29
alpha-amy2 0.61 0.43 0.95 0.51 1.05 0.50
ss-amy1 2.31 0.84 1.98 0.48 4.58 1.13
ss-amy2 117.6 12.5 100.0 6.1 152 1.80
nocut-amy1 3.20 0.31 1.05 0.17 1.26 0.34
nocut-amy2 213.9 4.9 63.4 2.4 48.2 0.58
  • ss-amy2: 16 times faster on 32 cores, 9 times on 8 cores, 85 times on the GPU. On master the GPU was slower than the CPU for all the
    cases but nocut-amy2; now it is 3.4 times faster than 32 cores (7 times faster than 8 cores) on the large system.
  • On master the time of ss-amy2 on the CPU was dominated by the vectors of STRANDS_CUTOFF over the 370800 pairs (about 85 ms) and did not
    scale with the threads.
  • For a single protein (ss-amy1, alpha) the CPU remains faster than the GPU: the GPU time is the cost of starting the kernels and of the
    transfers (about 0.3 ms per step for ALPHARMSD).
  • The nocut rows compare the same calculation of all the pairs, but this PR uses the function with D_MAX (needed for the skipping).
  • The master columns were measured with master plus a change that also passed USEGPU to the DISTANCE and LESS_THAN of the shortcuts
    (not part of this PR; it gave no measurable difference on the GPU and does not touch the CPU path).
  • One CPU core (cloud VM): ALPHARMSD 0.32 -> 0.18 ms (54 segments) and 2.35 -> 1.42 ms (432 segments); ss-amy2 298 -> 35 ms.

Summary

[ALPHARMSD], [ANTIBETARMSD] and [PARABETARMSD] now compute the number of segments that are similar to a helix or a sheet with
a single action, on the CPU and (with USEGPU) on the GPU, instead of the chain of vector actions created by the shortcuts.
For large systems this is up to 16 times faster on 32 CPU cores and about 85 times faster on the GPU than master.
ANTIBETARMSD and PARABETARMSD also get new defaults: STRANDS_CUTOFF=1 (nm), and a D_MAX for the default switching function
(also for ALPHARMSD). These defaults change the results of inputs that do not set these keywords; how to get the previous
behaviour is explained below.

The branch also contains the openaccPTM work that is not yet on master (MPI split of the GPU tasks, sums over the neighbours
without matrices, faster linked cells, action data kept on the GPU): see "Also in this branch".

Secondary structure

Single actions

  • New actions ALPHARMSD_SCALAR, ANTIBETARMSD_SCALAR, PARABETARMSD_SCALAR (src/secondarystructure/SecondaryStructureCount.h/.cpp).
    They build the segments from the lengths of the chains (for the sheets, as pairs of strands, with the same rules as the shortcuts),
    compute the distance of each segment from the references, the lowest one (PARABETARMSD) and the switching function, and add the
    derivatives to the atoms in the same pass: there is no backward pass and no vector with an element per segment.
    With two references the derivatives are computed only for the lowest one, and only when the switching function is not constant.
  • The shortcuts still create the vectors of the expanded input with the same labels (label_rmsd, label_lt, label_cut, ...).
    They are computed only if another action uses them (e.g. a PRINT of the vector with LESS_THAN, or LOWEST); the value of the shortcut,
    or its lessthan component, comes from the single action. The results are the same as the sum of the vectors (checked in the
    regtests rt-alpha-scalar and rt-sheet-scalar, which print both).
  • With USEGPU and TYPE=DRMSD, the same calculation runs on the GPU (*_SCALAR_ACC in the openaccPTM plugin), with the same
    per-segment code (shared functions in SecondaryStructureCount.h). With USEGPU and the RMSD types the vectors are used, as before.
  • OpenMP with per-thread buffers cleared and summed in parallel; MPI splits the segments between the ranks.

New defaults (results change)

  • ANTIBETARMSD and PARABETARMSD use STRANDS_CUTOFF=1 (nm) by default: the segments whose strands (CA atoms of the central residues)
    are further apart do not contribute. The value is converted to the units of UNITS.
  • The default switching function (given by R_0, NN, MM, D_0) of the three variables has a D_MAX, where it is 1e-5
    (1.42 nm with the defaults), and is stretched to be zero there, as SwitchingFunction::set(nn,mm,r0,d0) does for the other actions
    that take these keywords. Each segment within the cutoff changes by at most 1e-5.
  • How large the change is: on the amyloid of rt33 (18 peptides) the number of antiparallel segments changes by about 1.2 (out of
    5-58) because of the cutoff; with 144 peptides by about 10 (out of 40-460). ALPHARMSD changes by about 1e-4.
  • The value of D_MAX is in the expanded input (SWITCH of the _lt and _SCALAR actions) and in the log of the _SCALAR action.

How to get the previous behaviour

v2.10 / master input equivalent input with this PR
b: ANTIBETARMSD RESIDUES=all (all pairs, function without D_MAX), using b b: ANTIBETARMSD RESIDUES=all STRANDS_CUTOFF=0 LESS_THAN={RATIONAL R_0=0.08 NN=8 MM=12}, using b.lessthan
b: ANTIBETARMSD RESIDUES=all STRANDS_CUTOFF=1 R_0=0.1 b: ANTIBETARMSD RESIDUES=all LESS_THAN={RATIONAL R_0=0.1 NN=8 MM=12}, using b.lessthan
a: ALPHARMSD RESIDUES=all R_0=0.1 a: ALPHARMSD RESIDUES=all LESS_THAN={RATIONAL R_0=0.1 NN=8 MM=12}, using a.lessthan
  • STRANDS_CUTOFF=0 uses all the pairs of strands.
  • Giving the switching function with LESS_THAN (with the R_0, D_0, NN, MM used before) gives the function without D_MAX;
    with LESS_THAN the number of segments is the lessthan component (the value of the action is then the vector of the distances).
    R_0 in LESS_THAN is in the length units of UNITS (0.8 with UNITS LENGTH=A).
  • This is listed in CHANGES ("Changes that can change your results") and in the migration guide (developer-doc/Migration211.md).
  • Checked against the build before the change of defaults with explicit STRANDS_CUTOFF=1 and LESS_THAN={... D_MAX=...}:
    identical values and forces, for the sheets and the helix.

Exact skipping of the distant pairs

When the switching function is zero beyond a finite distance (the new default, or D_MAX in LESS_THAN), the single action also
skips the pairs of strands that are so far apart that their segment certainly has a distance from every reference larger than
D_MAX. The bound uses the distance between the central CA atoms and the radii of the two strands (DRMSD: the distances between
the strands in the reference; RMSD: the radii of gyration). The result does not change (equal, to the last digit, to the sum of
the vectors, which compute every pair), and STRANDS_CUTOFF=0 becomes cheap: ANTIBETARMSD on 144 peptides (370800 pairs) takes
2.4 ms per step on 32 cores (63 ms on master, with every pair computed).

Documentation

  • New sections "Strands cutoff" and "How the number of segments is computed" in ANTIBETARMSD and PARABETARMSD, and the latter in ALPHARMSD.
  • The stretching and D_MAX are explained in the three shortcuts and in the LESS_THAN and R_0 keywords; the LESS_THAN examples give D_MAX.
  • The plugin README has a "Secondary structure" section with the GPU timings.

Validation

  • Single actions equal to the sum of the vectors of the expanded input (values to 1e-10, forces within 1e-9), for DRMSD,
    OPTIMAL and OPTIMAL-FAST, with and without NOPBC, STYLE all/inter/intra, LESS_THAN, one and two references; --debug-forces;
    MPI (rt33-mpi, and 2 ranks on the GPU); 1 to 4 threads, also with fewer OpenMP threads than requested and without OpenMP (CI).
  • GPU: plugin regtests on Leonardo (nvc++), including the new rt-GPU-sheet-scalar (references from the CPU); compute-sanitizer
    memcheck and racecheck clean.
  • New regtests: secondarystructure/rt-alpha-scalar, secondarystructure/rt-sheet-scalar, targetGPU/rt-GPU-sheet-scalar.
    The references of the regtests that use the defaults were updated; only the columns of the three variables change.

Also in this branch (openaccPTM and neighbour sums, not yet on master)

  • openaccPTM: the tasks of the GPU actions are split between the MPI processes with MULTIGPU, each on its own GPU (the first process
    computes everything by default); one-pass force gathering by default on the GPU (AVOID_THREAD_GATHER for the low-memory version);
    the data of the actions is kept on the GPU between steps and copied again only when it may have changed (for the secondary structure
    this alone took ss-amy2 on the GPU from 152 to 118 ms).
  • COORDINATIONNUMBER and Q1-Q6 (CPU and GPU) compute the sums over the neighbours in one pass over the linked cells, without the contact
    matrix (and the matrices of the spherical harmonics); linked cells built about 2 times faster and in parallel. Details and timings in
    CHANGES and in the README of the plugin.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_016vwxtV46v5FLckcMEcJrbt

carlocamilloni and others added 19 commits November 17, 2026 10:56
With MPI each process computes a contiguous block of the active tasks on its
own GPU and the value stashes and the forces are summed, as in the CPU task
manager; with SERIAL (or one process) the first process computes everything
and broadcasts the results, as before. Only the part of the value stash
touched by the block is cleared, uploaded and downloaded.

- The low memory force gathering (AVOID_THREAD_GATHER, the default on the GPU)
  works with MPI: in the second pass only the first process keeps the forces
  of the first one; a process without tasks in the first pass uploads the
  positions in the second.
- GPU choice: without an MD code and with several GPUs on the node (e.g.
  mpirun -np 4 plumed driver), process i of the node uses GPU i % nGPUs
  (local rank from the launcher environment), unless ACC_DEVICE_NUM is set.
- Shortcuts: DISTANCES, Q1-Q6, LOCAL_Q*, LOCAL_AVERAGE, TETRAHEDRAL-like
  shortcuts accept SERIAL and pass it with USEGPU; COORDINATIONNUMBER and
  COORDINATION_MOMENTS accept USEGPU; Q* and LOCAL_AVERAGE put the contact
  matrix on the GPU too (as LOCAL_Q* already did).
- Regtests rt-GPU-mpi-nogather (4 processes, a process without rows),
  rt-GPU-mpi-serial and rt-GPU-mpi-shortcuts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016vwxtUCrVzSZSsREbiW4jL
…over the neighbours without matrices

From the Leonardo benchmarks (A100): splitting the tasks between the GPUs was
slower for every input tested, the two-pass force gathering doubled the
backward pass, and the matrices copied between CPU and GPU dominate
COORDINATIONNUMBER and Q6.

- With MPI the GPU actions are computed by the first process again; MULTIGPU
  (also accepted by the shortcuts that take USEGPU) splits their tasks.
- Forces of the matrices: one pass (atomics) is the default on the GPU too;
  AVOID_THREAD_GATHER asks for the low memory version. DO_THREAD_GATHER is
  kept for compatibility.
- New plugin actions COORDINATIONNUMBER_SUMSACC and Q1-Q6_SUMSACC (and
  Qn_DENOM_SUMSACC, src/NeighborSums.cpp): sums over the neighbours of each
  atom on the GPU in one pass, cells in scaled coordinates built on the CPU
  (any box), minimum image with Pbc, forces with atomics. Used by
  COORDINATIONNUMBER and Q1-Q6 with USEGPU (not with MASK, nor for
  COORDINATION_MOMENTS). Same results as the CPU (double precision).
- SphericalHarmonic: setup() for a given l, functions in the header inline.
- Regtest rt-GPU-neighbor-sums (triclinic box, 2 and 3 cells, no D_MAX,
  SPECIESA/B, MULTIGPU on 3 processes); the explicit Q6 chain test now
  runs SPHERICAL_HARMONIC on the GPU; references of two tests now exact.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016vwxtUCrVzSZSsREbiW4jL
The neighbour loop was a macro whose argument contained OpenACC pragmas,
which are not processed inside macro arguments: nvc++ (linux-nvhpc CI)
saw atomics on invalid expressions and the loop as a routine. The loop
is now written out in the two kernels; the action data is a local copy
on the device (as in the task manager); pair() is an acc routine seq;
epsilon is passed as a firstprivate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016vwxtUCrVzSZSsREbiW4jL
The COMBINE of the 2(2l+1) vectors was slower on the GPU than on the
CPU (Leonardo, q6-20000: 6+3 ms vs 4.5+1.2 ms), since its inputs and
outputs go to the GPU and back at every step. SERIAL is still passed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016vwxtUCrVzSZSsREbiW4jL
With SPECIESA/SPECIESB the forces on the columns are accumulated in one
buffer per thread. The buffers were sized with OpenMP::getNumThreads(),
which without OpenMP still returns PLUMED_NUM_THREADS, while only thread 0
runs: the reduction read the empty buffer of thread 1 and crashed
(symfunc/rt-neighbor-sums, -small, -mpi and rt-q6-subset on the Windows and
the Linux no-OpenMP builds of the CI, which use PLUMED_NUM_THREADS=2).
The threads that run now mark their buffer, and only those are summed;
this also covers an OpenMP team smaller than requested.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016vwxtTud9yy24ZL9dp5N2v
…v variable

The sums are copied to the values with transferStashToValues and the forces
on the values with transferForcesToStash (the sums already have the layout
of the stash): they copy in memory order and with OpenMP. The previous
serial loops over the components called Value::set/getForce per element,
15% of Q6 on 20000 atoms with 32 threads. The sums are zeroed only when
this process does not compute all the rows.

The openaccPTM plugin no longer reads PLUMED_NEIGHBORSUMS_SUBDIVISION: two
cells per cutoff is 25-32% faster than one on the A100 too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016vwxtTud9yy24ZL9dp5N2v
… several threads per row, few rows without cells

- The pair functors add the values of a pair to the sums (addValues) and contract the
  forces pair by pair (contract), without storing the derivatives of all the components:
  for the spherical harmonics the components +m and -m are done together.
- Qn_SUMS also computes the coordination numbers in the same pass over the neighbours;
  Qn_DENOM_SUMS SUMS=label gives them as a vector and passes back the forces on them
  (it applies before the sums). This replaces the second pass (the ContactPairs
  Qn_DENOM_SUMS and the plugin's Qn_DENOM_SUMSACC). Qn_NORM computes the norms of the
  sums in one action; q6_norm2 (COMBINE) is still created and computed only if used.
- The search around each row can be split in parts: groups of the rows of cells around
  it (forEachCandidate/forEachNeighbor with slice, nslices), or blocks of columns without
  cells. The GPU actions use several GPU threads per row (about 131k threads, at most 25
  per row) and sum the parts of each row in order with a second kernel.
- Without GROUP and with few rows (at most 8) the cells are not built: the columns are
  split in blocks shared by the threads (and by the MPI processes), with the minimum
  images computed in blocks (Pbc::apply) on the CPU. The results do not depend on the
  number of threads.
- TEMPORARY, for the Leonardo measurements: PLUMED_NEIGHBORSUMS_SLICES (GPU parts per
  row) and PLUMED_NEIGHBORSUMS_FEWROWS (the few-rows threshold).
- New regtests rt-neighbor-sums-few-rows (-mpi) and rt-neighbor-sums-parts; the
  references agree with the contact matrix written out (chain.dat). rt-parse-only:
  q1_norm is now Q1_NORM (text only).

2-core VM, -O2, ms/step forward+backward, 1/2 threads, before -> after:
Q6 2000 atoms 47.9/24.9 -> 25.1/13.1; Q6 20000 528/270 -> 285/146;
COORDINATIONNUMBER 2000 and 20000 3-5% faster; one atom against 100000 waters
(SPECIESA/SPECIESB) 7.4/5.4 -> 2.7/2.0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016vwxtTq4ZtJbDFt8nrMV3A
…ne site against many atoms

- COORDINATION (and the other users of NeighborList) with NLISTCELLS builds the list with the
  plain search when one of the two groups has at most 8 atoms: sorting all the atoms into
  cells costs more than one distance per pair (equal cost at about 8 atoms). One atom
  against 100000 water oxygens, list updated at every step: 7.9 -> 3.4 ms/step.
- Docs: COORDINATION ("One atom against many atoms"), COORDINATIONNUMBER and CONTACT_MATRIX
  say which input to use for one site against many atoms. CHANGES/v2.11.md updated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016vwxtTq4ZtJbDFt8nrMV3A
…lways use the core’s OpenMP. The plugin now runs no OpenMP code of its own. I’ve noted this as a rule in the progress doc.

GPU: one thread per row when the cells are used. With few rows, the columns are split into blocks between the GPU threads. The splitting of a row’s cells is gone from LinkCellsView.h, which is back to master’s version.
Few-rows cutoffs: on the CPU, 2 × the number of threads (at least 4), as you chose; on the GPU, 512.
Both temporary environment variables are gone, and the plugin README is updated.
… plugin

On one A100 (8 CPU threads) / 32 CPU cores, ms per step: COORDINATIONNUMBER 3.7 / 4.7 (20000 atoms),
17 / 25 (100000); Q6 13 / 17, 59 / 94 (Leonardo, round 8).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016vwxtTq4ZtJbDFt8nrMV3A
The cell of each atom and its wrapped position come from one change of
coordinates (written out), with the same operations as before; the
counting and the placing of the atoms in the cells, which were serial,
are split among the threads by ranges of cells, so the cells and the
order of their atoms are identical whatever the number of threads.
About 2.5 times faster on one thread (20000 atoms: 1.0 -> 0.4 ms).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016vwxtTq4ZtJbDFt8nrMV3A
…ings

With the cheaper cells (Leonardo, 32 threads, 100000 columns) all the
pairs cost as much as the cells at about 44 rows: 64 rows took 4.03 ms
without cells against ~3.5 with them. The threshold is now the number
of threads (at least 4; unchanged up to 4 threads). CHANGES and the
plugin README have the round-9 timings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016vwxtTq4ZtJbDFt8nrMV3A
runAllTasksACC and applyForcesWithACC took the action data (input_type)
by value, a deep copy on the host, and moved it to the device with
enter/exit data at every call, twice per step. For large action data
this dominated the time of the action: e.g. the 370k segments of the
secondary structure variables of 144 peptides are 45 MB of indices,
and SECONDARY_STRUCTURE_DRMSD_ACC took 14 ms forward and 10 ms
backward per step with 5000 active segments (4 and 1 ms on the CPU).

The task manager now keeps a copy of the action data on the device of
the action and passes the kernels a pointer to it (present(ad[0:1])).
The copy is made and uploaded again only when the data may have
changed: ParallelTaskManager counts the calls of setActionInput and of
the non-const getActionInput (getActionInputGeneration), which is how
actions change their input (e.g. the neighbour lists of the contact
matrix, the regions of the volumes, the bookkeeping of the matrix
products), so those actions still upload it at every step, as before.
The copy is removed from the device when the task manager is
destroyed.

Tested with the plugin built with g++ and the host stub of the OpenACC
runtime (51 targetGPU regtests pass; the stub ignores the data
directives, so the residency itself is only tested with nvc++).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016vwxtV46v5FLckcMEcJrbt
The number of segments that are similar to a sheet was the end of a
chain of actions with one element per pair of strands: DISTANCE and a
step CUSTOM for STRANDS_CUTOFF, SECONDARY_STRUCTURE_(D)RMSD with the
mask, (LOWEST,) LESS_THAN, a product CUSTOM and SUM. With many chains
almost all the time went into these vectors: with 144 peptides there
are 370k pairs of strands, and on Leonardo the step took 100 ms on 32
cores, about 90 of them in the per-element work of the chain, which
does not scale with the threads.

New action ANTIBETARMSD_SCALAR / PARABETARMSD_SCALAR (SheetCount.cpp,
created by the shortcuts, documented in their pages):
- the pairs of strands are built from the lengths of the chains
  (SecondaryStructureBase::getSheetPairs, now also used by the
  shortcuts to write the SEGMENT keywords, in the same order);
- the pairs further apart than STRANDS_CUTOFF cost one distance;
- each remaining segment is made whole and aligned as in
  SECONDARY_STRUCTURE_(D)RMSD, its distance from the references is
  computed with the same code (lowest one for PARABETARMSD), and the
  derivatives of the switching function are added to the atoms and to
  the virial in the same pass (OpenMP threads with their own buffers,
  MPI ranks share the pairs).
The shortcut still creates the chain, with the same labels, but the
value (or the lessthan component with LESS_THAN) comes from the new
action: the vectors are computed only if something else uses them
(e.g. PRINT ARG=*). With USEGPU the chain gives the result as before.
SecondaryStructureRMSDInput is now in SecondaryStructureRMSD.h, so
that the new action can use it; readDistanceAction also returns TYPE
and USEGPU.

Results: identical values (to 1e-10) and forces/virial (3e-10, order
of the sums) to the chain on the rt33 amyloid with STRANDS_CUTOFF and
without, LESS_THAN, STYLE inter/intra, NOPBC, DRMSD, OPTIMAL and
OPTIMAL-FAST (new regtest rt-sheet-scalar, references from the
chain); driver --debug-forces including the virial; 1 and 2 OpenMP
threads; rt33-mpi (2 thread-MPI ranks). All the secondarystructure
regtests and the 51 targetGPU regtests of the plugin (g++ + stub)
pass with unchanged references.

One core, ms per step (all three shortcuts, STRANDS_CUTOFF=1):
2520 atoms 9.8 -> 4.7, 20160 atoms (370k pairs) 298 -> 41;
ANTIBETARMSD without cutoff, 2520 atoms: 27 -> 16 (each segment is
computed once, not again in the backward pass).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016vwxtV46v5FLckcMEcJrbt
With USEGPU and TYPE=DRMSD the shortcuts use ANTIBETARMSD_SCALAR_ACC and
PARABETARMSD_SCALAR_ACC (openaccPTM plugin), the GPU version of the single
action that computes the number of segments: one kernel lists the pairs of
strands within STRANDS_CUTOFF, one thread per listed segment computes its
distance, the switching function and the derivatives (added to the atoms
with atomics, consecutive threads taking segments far apart in the list),
and a last kernel sums the values and the virial. There is no backward pass,
and the pairs, the references and the switching function are copied once.

The calculation of a segment (positions made whole and aligned, distance
from the references, switching function) is in SheetCount.h and is shared
by the CPU and GPU versions. With two references the distances are computed
without derivatives, and the derivatives only for the lowest one, and only
when the switching function is not constant.

New plugin regtest rt-GPU-sheet-scalar (references from the CPU).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016vwxtV46v5FLckcMEcJrbt
Neighbouring tasks often add forces to the same atoms (segments that share
a strand, rows of a matrix with the same columns), and the threads of a warp
are serialized when their atomics go to the same address. In the force
kernel thread w now computes the task in position (w%nint)*chunk+w/nint of
the list, i.e. consecutive threads take tasks nint positions apart. The work
buffers are indexed by the thread, so the writes to the buffer of the shared
entries stay coalesced; the last threads may have no task.

nint is 32 by default and can be set with PLUMED_ACC_TASK_INTERLEAVE (1 gives
the previous order), which also applies to ANTIBETARMSD_SCALAR_ACC and
PARABETARMSD_SCALAR_ACC. The variable is meant for benchmarking: the default
is to be confirmed (or the change dropped) on the GPU.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016vwxtV46v5FLckcMEcJrbt
@codecov

codecov Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.32836% with 19 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.00%. Comparing base (d5ed95e) to head (47abad9).

Files with missing lines Patch % Lines
src/secondarystructure/SecondaryStructureBase.h 84.21% 9 Missing ⚠️
src/secondarystructure/SecondaryStructureCount.cpp 95.20% 7 Missing ⚠️
src/secondarystructure/AlphaRMSD.cpp 92.30% 1 Missing ⚠️
src/secondarystructure/AntibetaRMSD.cpp 96.29% 1 Missing ⚠️
src/secondarystructure/ParabetaRMSD.cpp 96.29% 1 Missing ⚠️
Additional details and impacted files
@@                   Coverage Diff                   @@
##           openaccPTM-multigpu    #1641      +/-   ##
=======================================================
+ Coverage                88.98%   89.00%   +0.02%     
=======================================================
  Files                      654      657       +3     
  Lines                    82343    82544     +201     
=======================================================
+ Hits                     73271    73467     +196     
- Misses                    9072     9077       +5     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…loop

The derivatives of the nt buffers are summed after the loop, but each buffer
was cleared by its own thread: when fewer than nt threads ran, the sum read
buffers that were never sized (or kept the values of an earlier step).
This happens without OpenMP, where nt still comes from PLUMED_NUM_THREADS
(segmentation fault in the secondarystructure regtests of the clang build
without OpenMP), and when OpenMP gives fewer threads than requested
(reproduced with PLUMED_NUM_THREADS=4 OMP_THREAD_LIMIT=1).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016vwxtV46v5FLckcMEcJrbt
nvc++ 24.5 stops with "Data clause required with default(none)" for the
arrays declared after the continue that skipped the threads with no task
(slotIndex, slotForce in the force kernel of the task manager), and warns
(W-1070) for myout. The work buffers (myout) and the accumulators are now
declared first in the body of the loop, before its inner loops, and the
threads with no task (or no segment) skip the work with a flag.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016vwxtV46v5FLckcMEcJrbt
…he sheets

On Leonardo (A100) the interleaved order (d7ee50f) made the sheet kernel
7% faster only without STRANDS_CUTOFF on 370k segments, and the force
kernel of the task manager 5-10% slower for ALPHARMSD; the other cases did
not change. The force kernel is back to the code before d7ee50f (one
thread per task, no continue), the sheet kernel has one thread per segment,
and PLUMED_ACC_TASK_INTERLEAVE is gone.

GPU timings of ANTIBETARMSD_SCALAR_ACC and PARABETARMSD_SCALAR_ACC in the
plugin README and in CHANGES: 144 peptides, STRANDS_CUTOFF=1, the three
variables 2.35 ms per step on one A100 (118 ms with the vectors, 7.8 ms on
32 CPU cores).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016vwxtV46v5FLckcMEcJrbt
ALPHARMSD computes the number of segments that are similar to an alpha
helix with ALPHARMSD_SCALAR (ALPHARMSD_SCALAR_ACC in the openaccPTM plugin
with USEGPU and TYPE=DRMSD), as ANTIBETARMSD and PARABETARMSD do: the
segments are built from the lengths of the chains and the derivatives are
added to the atoms while the segments are computed, so there is no backward
pass. The vectors of the expanded input are computed only when used.
About 1.7 times faster on one CPU core (0.32 -> 0.18 ms per step for 54
segments, 2.35 -> 1.42 ms for 432).

SheetCount becomes SecondaryStructureCount (SecondaryStructureCount.h and
.cpp, also in the plugin), with a helix mode: a segment is two halves of 15
atoms and for a helix the segment is made whole as a single chain.

New regtest rt-alpha-scalar: DRMSD, OPTIMAL with LESS_THAN and NOPBC, each
value printed next to the sum of the vector of the expanded input (equal).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016vwxtV46v5FLckcMEcJrbt
…ip the pairs whose switching function is zero

Behaviour change: STRANDS_CUTOFF is 1 nm by default (converted to the units
of UNITS); STRANDS_CUTOFF=0 uses all the pairs of strands as before. With
R_0, NN, MM and D_0 (no LESS_THAN) the switching function gets the D_MAX
that SwitchingFunction::set(nn,mm,r0,d0) uses for the other actions (where
the function is 1e-5) and is stretched to zero there; LESS_THAN={RATIONAL
R_0=0.08 NN=8 MM=12} gives the function of v2.10. Checked against the build
before this commit with explicit STRANDS_CUTOFF=1 and LESS_THAN={... D_MAX=}:
identical values and forces.

When the switching function is zero beyond a finite distance, the single
action (CPU and GPU) also skips the pairs of strands whose central CA atoms
are further apart than a distance computed from the references plus the
radii of the two strands: a rigorous lower bound of the distance of their
segment from every reference is then above D_MAX (DRMSD: the cross-strand
distances; RMSD: the radii of gyration). The result does not change (equal
to the sum of the vectors of the expanded input, which compute every pair),
and STRANDS_CUTOFF=0 is much cheaper: ANTIBETARMSD on 144 peptides (370800
pairs) 930 -> 160 ms per step on one core.

Regtests: references updated for the new defaults (only the sheet columns
change); rt-sheet-scalar and rt-GPU-sheet-scalar have pairs with
STRANDS_CUTOFF=0, rt-sheet-scalar prints them next to the sums of the
vectors. CHANGES (results that change, collective variables) and the
migration guide.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016vwxtV46v5FLckcMEcJrbt
ALPHARMSD gets the same default as the sheets: with R_0, NN, MM and D_0
(no LESS_THAN) the switching function has the D_MAX where it is 1e-5 and is
stretched to zero there, as in the other actions that take these keywords.
readShortcutWords always does it (the flag is gone). Checked against the
build before this commit with LESS_THAN={... D_MAX=}: identical values and
forces. The number of helical segments changes by at most 1e-5 per segment
(references of 8 regtests and 2 GPU regtests updated, alpha columns only).

Documentation: the LESS_THAN examples of the three shortcuts give D_MAX,
the LESS_THAN and R_0 keywords explain it, the formula paragraph of
ALPHARMSD explains the stretching, STRANDS_CUTOFF=1 is dropped from the
sheet examples (it is the default), and the expanded inputs in
SECONDARY_STRUCTURE_(D)RMSD show the D_MAX. The single action writes D_MAX
in the log (it is not in the description of the switching function).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016vwxtV46v5FLckcMEcJrbt
…parallel

The derivative buffers of the nt threads (3 natoms+9 each) were cleared
and summed by one thread. On Leonardo with 32 threads this made ALPHARMSD
on 432 segments slower than with 8 (0.81 vs 0.37 ms). The buffers are now
cleared with a loop over the nt buffers and summed with a loop over the
derivatives, both shared between the threads in the parallel region (still
correct when fewer than nt threads run, and with the same order of the sums,
so the results do not change), and the radii of the strands are computed in
the same region.

Timings of the final benchmark (Leonardo, A100 and 8/32 cores) in the
plugin README and in CHANGES.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016vwxtV46v5FLckcMEcJrbt
… sum of the buffers

Leonardo, 32 cores: the three variables on 144 peptides 7.5 -> 6.1 ms per
step, ALPHARMSD on 432 segments 0.91 -> 0.51 ms (0.43 ms on 8 cores).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016vwxtV46v5FLckcMEcJrbt
@carlocamilloni carlocamilloni changed the title Secondary structure CVs on the GPU (openaccPTM) and fused CPU path: status, benchmark kit and results (2026-11-17) Secondary structure variables: single actions on CPU and GPU, new defaults for the sheets; openaccPTM improvements Nov 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants