Repository navigation
Secondary structure variables: single actions on CPU and GPU, new defaults for the sheets; openaccPTM improvements - #1641
Closed
carlocamilloni wants to merge 27 commits into
Closed
carlocamilloni wants to merge 27 commits into
carlocamilloni wants to merge 27 commits into
Conversation
With MPI each process computes a contiguous block of the active tasks on its own GPU and the value stashes and the forces are summed, as in the CPU task manager; with SERIAL (or one process) the first process computes everything and broadcasts the results, as before. Only the part of the value stash touched by the block is cleared, uploaded and downloaded. - The low memory force gathering (AVOID_THREAD_GATHER, the default on the GPU) works with MPI: in the second pass only the first process keeps the forces of the first one; a process without tasks in the first pass uploads the positions in the second. - GPU choice: without an MD code and with several GPUs on the node (e.g. mpirun -np 4 plumed driver), process i of the node uses GPU i % nGPUs (local rank from the launcher environment), unless ACC_DEVICE_NUM is set. - Shortcuts: DISTANCES, Q1-Q6, LOCAL_Q*, LOCAL_AVERAGE, TETRAHEDRAL-like shortcuts accept SERIAL and pass it with USEGPU; COORDINATIONNUMBER and COORDINATION_MOMENTS accept USEGPU; Q* and LOCAL_AVERAGE put the contact matrix on the GPU too (as LOCAL_Q* already did). - Regtests rt-GPU-mpi-nogather (4 processes, a process without rows), rt-GPU-mpi-serial and rt-GPU-mpi-shortcuts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016vwxtUCrVzSZSsREbiW4jL
…over the neighbours without matrices From the Leonardo benchmarks (A100): splitting the tasks between the GPUs was slower for every input tested, the two-pass force gathering doubled the backward pass, and the matrices copied between CPU and GPU dominate COORDINATIONNUMBER and Q6. - With MPI the GPU actions are computed by the first process again; MULTIGPU (also accepted by the shortcuts that take USEGPU) splits their tasks. - Forces of the matrices: one pass (atomics) is the default on the GPU too; AVOID_THREAD_GATHER asks for the low memory version. DO_THREAD_GATHER is kept for compatibility. - New plugin actions COORDINATIONNUMBER_SUMSACC and Q1-Q6_SUMSACC (and Qn_DENOM_SUMSACC, src/NeighborSums.cpp): sums over the neighbours of each atom on the GPU in one pass, cells in scaled coordinates built on the CPU (any box), minimum image with Pbc, forces with atomics. Used by COORDINATIONNUMBER and Q1-Q6 with USEGPU (not with MASK, nor for COORDINATION_MOMENTS). Same results as the CPU (double precision). - SphericalHarmonic: setup() for a given l, functions in the header inline. - Regtest rt-GPU-neighbor-sums (triclinic box, 2 and 3 cells, no D_MAX, SPECIESA/B, MULTIGPU on 3 processes); the explicit Q6 chain test now runs SPHERICAL_HARMONIC on the GPU; references of two tests now exact. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016vwxtUCrVzSZSsREbiW4jL
The neighbour loop was a macro whose argument contained OpenACC pragmas, which are not processed inside macro arguments: nvc++ (linux-nvhpc CI) saw atomics on invalid expressions and the loop as a routine. The loop is now written out in the two kernels; the action data is a local copy on the device (as in the task manager); pair() is an acc routine seq; epsilon is passed as a firstprivate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016vwxtUCrVzSZSsREbiW4jL
The COMBINE of the 2(2l+1) vectors was slower on the GPU than on the CPU (Leonardo, q6-20000: 6+3 ms vs 4.5+1.2 ms), since its inputs and outputs go to the GPU and back at every step. SERIAL is still passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016vwxtUCrVzSZSsREbiW4jL
With SPECIESA/SPECIESB the forces on the columns are accumulated in one buffer per thread. The buffers were sized with OpenMP::getNumThreads(), which without OpenMP still returns PLUMED_NUM_THREADS, while only thread 0 runs: the reduction read the empty buffer of thread 1 and crashed (symfunc/rt-neighbor-sums, -small, -mpi and rt-q6-subset on the Windows and the Linux no-OpenMP builds of the CI, which use PLUMED_NUM_THREADS=2). The threads that run now mark their buffer, and only those are summed; this also covers an OpenMP team smaller than requested. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016vwxtTud9yy24ZL9dp5N2v
…v variable The sums are copied to the values with transferStashToValues and the forces on the values with transferForcesToStash (the sums already have the layout of the stash): they copy in memory order and with OpenMP. The previous serial loops over the components called Value::set/getForce per element, 15% of Q6 on 20000 atoms with 32 threads. The sums are zeroed only when this process does not compute all the rows. The openaccPTM plugin no longer reads PLUMED_NEIGHBORSUMS_SUBDIVISION: two cells per cutoff is 25-32% faster than one on the A100 too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016vwxtTud9yy24ZL9dp5N2v
… several threads per row, few rows without cells - The pair functors add the values of a pair to the sums (addValues) and contract the forces pair by pair (contract), without storing the derivatives of all the components: for the spherical harmonics the components +m and -m are done together. - Qn_SUMS also computes the coordination numbers in the same pass over the neighbours; Qn_DENOM_SUMS SUMS=label gives them as a vector and passes back the forces on them (it applies before the sums). This replaces the second pass (the ContactPairs Qn_DENOM_SUMS and the plugin's Qn_DENOM_SUMSACC). Qn_NORM computes the norms of the sums in one action; q6_norm2 (COMBINE) is still created and computed only if used. - The search around each row can be split in parts: groups of the rows of cells around it (forEachCandidate/forEachNeighbor with slice, nslices), or blocks of columns without cells. The GPU actions use several GPU threads per row (about 131k threads, at most 25 per row) and sum the parts of each row in order with a second kernel. - Without GROUP and with few rows (at most 8) the cells are not built: the columns are split in blocks shared by the threads (and by the MPI processes), with the minimum images computed in blocks (Pbc::apply) on the CPU. The results do not depend on the number of threads. - TEMPORARY, for the Leonardo measurements: PLUMED_NEIGHBORSUMS_SLICES (GPU parts per row) and PLUMED_NEIGHBORSUMS_FEWROWS (the few-rows threshold). - New regtests rt-neighbor-sums-few-rows (-mpi) and rt-neighbor-sums-parts; the references agree with the contact matrix written out (chain.dat). rt-parse-only: q1_norm is now Q1_NORM (text only). 2-core VM, -O2, ms/step forward+backward, 1/2 threads, before -> after: Q6 2000 atoms 47.9/24.9 -> 25.1/13.1; Q6 20000 528/270 -> 285/146; COORDINATIONNUMBER 2000 and 20000 3-5% faster; one atom against 100000 waters (SPECIESA/SPECIESB) 7.4/5.4 -> 2.7/2.0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016vwxtTq4ZtJbDFt8nrMV3A
…ne site against many atoms
- COORDINATION (and the other users of NeighborList) with NLISTCELLS builds the list with the
plain search when one of the two groups has at most 8 atoms: sorting all the atoms into
cells costs more than one distance per pair (equal cost at about 8 atoms). One atom
against 100000 water oxygens, list updated at every step: 7.9 -> 3.4 ms/step.
- Docs: COORDINATION ("One atom against many atoms"), COORDINATIONNUMBER and CONTACT_MATRIX
say which input to use for one site against many atoms. CHANGES/v2.11.md updated.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016vwxtTq4ZtJbDFt8nrMV3A
…lways use the core’s OpenMP. The plugin now runs no OpenMP code of its own. I’ve noted this as a rule in the progress doc. GPU: one thread per row when the cells are used. With few rows, the columns are split into blocks between the GPU threads. The splitting of a row’s cells is gone from LinkCellsView.h, which is back to master’s version. Few-rows cutoffs: on the CPU, 2 × the number of threads (at least 4), as you chose; on the GPU, 512. Both temporary environment variables are gone, and the plugin README is updated.
… plugin On one A100 (8 CPU threads) / 32 CPU cores, ms per step: COORDINATIONNUMBER 3.7 / 4.7 (20000 atoms), 17 / 25 (100000); Q6 13 / 17, 59 / 94 (Leonardo, round 8). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016vwxtTq4ZtJbDFt8nrMV3A
The cell of each atom and its wrapped position come from one change of coordinates (written out), with the same operations as before; the counting and the placing of the atoms in the cells, which were serial, are split among the threads by ranges of cells, so the cells and the order of their atoms are identical whatever the number of threads. About 2.5 times faster on one thread (20000 atoms: 1.0 -> 0.4 ms). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016vwxtTq4ZtJbDFt8nrMV3A
…ings With the cheaper cells (Leonardo, 32 threads, 100000 columns) all the pairs cost as much as the cells at about 44 rows: 64 rows took 4.03 ms without cells against ~3.5 with them. The threshold is now the number of threads (at least 4; unchanged up to 4 threads). CHANGES and the plugin README have the round-9 timings. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016vwxtTq4ZtJbDFt8nrMV3A
runAllTasksACC and applyForcesWithACC took the action data (input_type) by value, a deep copy on the host, and moved it to the device with enter/exit data at every call, twice per step. For large action data this dominated the time of the action: e.g. the 370k segments of the secondary structure variables of 144 peptides are 45 MB of indices, and SECONDARY_STRUCTURE_DRMSD_ACC took 14 ms forward and 10 ms backward per step with 5000 active segments (4 and 1 ms on the CPU). The task manager now keeps a copy of the action data on the device of the action and passes the kernels a pointer to it (present(ad[0:1])). The copy is made and uploaded again only when the data may have changed: ParallelTaskManager counts the calls of setActionInput and of the non-const getActionInput (getActionInputGeneration), which is how actions change their input (e.g. the neighbour lists of the contact matrix, the regions of the volumes, the bookkeeping of the matrix products), so those actions still upload it at every step, as before. The copy is removed from the device when the task manager is destroyed. Tested with the plugin built with g++ and the host stub of the OpenACC runtime (51 targetGPU regtests pass; the stub ignores the data directives, so the residency itself is only tested with nvc++). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016vwxtV46v5FLckcMEcJrbt
The number of segments that are similar to a sheet was the end of a chain of actions with one element per pair of strands: DISTANCE and a step CUSTOM for STRANDS_CUTOFF, SECONDARY_STRUCTURE_(D)RMSD with the mask, (LOWEST,) LESS_THAN, a product CUSTOM and SUM. With many chains almost all the time went into these vectors: with 144 peptides there are 370k pairs of strands, and on Leonardo the step took 100 ms on 32 cores, about 90 of them in the per-element work of the chain, which does not scale with the threads. New action ANTIBETARMSD_SCALAR / PARABETARMSD_SCALAR (SheetCount.cpp, created by the shortcuts, documented in their pages): - the pairs of strands are built from the lengths of the chains (SecondaryStructureBase::getSheetPairs, now also used by the shortcuts to write the SEGMENT keywords, in the same order); - the pairs further apart than STRANDS_CUTOFF cost one distance; - each remaining segment is made whole and aligned as in SECONDARY_STRUCTURE_(D)RMSD, its distance from the references is computed with the same code (lowest one for PARABETARMSD), and the derivatives of the switching function are added to the atoms and to the virial in the same pass (OpenMP threads with their own buffers, MPI ranks share the pairs). The shortcut still creates the chain, with the same labels, but the value (or the lessthan component with LESS_THAN) comes from the new action: the vectors are computed only if something else uses them (e.g. PRINT ARG=*). With USEGPU the chain gives the result as before. SecondaryStructureRMSDInput is now in SecondaryStructureRMSD.h, so that the new action can use it; readDistanceAction also returns TYPE and USEGPU. Results: identical values (to 1e-10) and forces/virial (3e-10, order of the sums) to the chain on the rt33 amyloid with STRANDS_CUTOFF and without, LESS_THAN, STYLE inter/intra, NOPBC, DRMSD, OPTIMAL and OPTIMAL-FAST (new regtest rt-sheet-scalar, references from the chain); driver --debug-forces including the virial; 1 and 2 OpenMP threads; rt33-mpi (2 thread-MPI ranks). All the secondarystructure regtests and the 51 targetGPU regtests of the plugin (g++ + stub) pass with unchanged references. One core, ms per step (all three shortcuts, STRANDS_CUTOFF=1): 2520 atoms 9.8 -> 4.7, 20160 atoms (370k pairs) 298 -> 41; ANTIBETARMSD without cutoff, 2520 atoms: 27 -> 16 (each segment is computed once, not again in the backward pass). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016vwxtV46v5FLckcMEcJrbt
With USEGPU and TYPE=DRMSD the shortcuts use ANTIBETARMSD_SCALAR_ACC and PARABETARMSD_SCALAR_ACC (openaccPTM plugin), the GPU version of the single action that computes the number of segments: one kernel lists the pairs of strands within STRANDS_CUTOFF, one thread per listed segment computes its distance, the switching function and the derivatives (added to the atoms with atomics, consecutive threads taking segments far apart in the list), and a last kernel sums the values and the virial. There is no backward pass, and the pairs, the references and the switching function are copied once. The calculation of a segment (positions made whole and aligned, distance from the references, switching function) is in SheetCount.h and is shared by the CPU and GPU versions. With two references the distances are computed without derivatives, and the derivatives only for the lowest one, and only when the switching function is not constant. New plugin regtest rt-GPU-sheet-scalar (references from the CPU). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016vwxtV46v5FLckcMEcJrbt
Neighbouring tasks often add forces to the same atoms (segments that share a strand, rows of a matrix with the same columns), and the threads of a warp are serialized when their atomics go to the same address. In the force kernel thread w now computes the task in position (w%nint)*chunk+w/nint of the list, i.e. consecutive threads take tasks nint positions apart. The work buffers are indexed by the thread, so the writes to the buffer of the shared entries stay coalesced; the last threads may have no task. nint is 32 by default and can be set with PLUMED_ACC_TASK_INTERLEAVE (1 gives the previous order), which also applies to ANTIBETARMSD_SCALAR_ACC and PARABETARMSD_SCALAR_ACC. The variable is meant for benchmarking: the default is to be confirmed (or the change dropped) on the GPU. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016vwxtV46v5FLckcMEcJrbt
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## openaccPTM-multigpu #1641 +/- ##
=======================================================
+ Coverage 88.98% 89.00% +0.02%
=======================================================
Files 654 657 +3
Lines 82343 82544 +201
=======================================================
+ Hits 73271 73467 +196
- Misses 9072 9077 +5 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…loop The derivatives of the nt buffers are summed after the loop, but each buffer was cleared by its own thread: when fewer than nt threads ran, the sum read buffers that were never sized (or kept the values of an earlier step). This happens without OpenMP, where nt still comes from PLUMED_NUM_THREADS (segmentation fault in the secondarystructure regtests of the clang build without OpenMP), and when OpenMP gives fewer threads than requested (reproduced with PLUMED_NUM_THREADS=4 OMP_THREAD_LIMIT=1). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016vwxtV46v5FLckcMEcJrbt
nvc++ 24.5 stops with "Data clause required with default(none)" for the arrays declared after the continue that skipped the threads with no task (slotIndex, slotForce in the force kernel of the task manager), and warns (W-1070) for myout. The work buffers (myout) and the accumulators are now declared first in the body of the loop, before its inner loops, and the threads with no task (or no segment) skip the work with a flag. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016vwxtV46v5FLckcMEcJrbt
…he sheets On Leonardo (A100) the interleaved order (d7ee50f) made the sheet kernel 7% faster only without STRANDS_CUTOFF on 370k segments, and the force kernel of the task manager 5-10% slower for ALPHARMSD; the other cases did not change. The force kernel is back to the code before d7ee50f (one thread per task, no continue), the sheet kernel has one thread per segment, and PLUMED_ACC_TASK_INTERLEAVE is gone. GPU timings of ANTIBETARMSD_SCALAR_ACC and PARABETARMSD_SCALAR_ACC in the plugin README and in CHANGES: 144 peptides, STRANDS_CUTOFF=1, the three variables 2.35 ms per step on one A100 (118 ms with the vectors, 7.8 ms on 32 CPU cores). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016vwxtV46v5FLckcMEcJrbt
ALPHARMSD computes the number of segments that are similar to an alpha helix with ALPHARMSD_SCALAR (ALPHARMSD_SCALAR_ACC in the openaccPTM plugin with USEGPU and TYPE=DRMSD), as ANTIBETARMSD and PARABETARMSD do: the segments are built from the lengths of the chains and the derivatives are added to the atoms while the segments are computed, so there is no backward pass. The vectors of the expanded input are computed only when used. About 1.7 times faster on one CPU core (0.32 -> 0.18 ms per step for 54 segments, 2.35 -> 1.42 ms for 432). SheetCount becomes SecondaryStructureCount (SecondaryStructureCount.h and .cpp, also in the plugin), with a helix mode: a segment is two halves of 15 atoms and for a helix the segment is made whole as a single chain. New regtest rt-alpha-scalar: DRMSD, OPTIMAL with LESS_THAN and NOPBC, each value printed next to the sum of the vector of the expanded input (equal). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016vwxtV46v5FLckcMEcJrbt
…ip the pairs whose switching function is zero
Behaviour change: STRANDS_CUTOFF is 1 nm by default (converted to the units
of UNITS); STRANDS_CUTOFF=0 uses all the pairs of strands as before. With
R_0, NN, MM and D_0 (no LESS_THAN) the switching function gets the D_MAX
that SwitchingFunction::set(nn,mm,r0,d0) uses for the other actions (where
the function is 1e-5) and is stretched to zero there; LESS_THAN={RATIONAL
R_0=0.08 NN=8 MM=12} gives the function of v2.10. Checked against the build
before this commit with explicit STRANDS_CUTOFF=1 and LESS_THAN={... D_MAX=}:
identical values and forces.
When the switching function is zero beyond a finite distance, the single
action (CPU and GPU) also skips the pairs of strands whose central CA atoms
are further apart than a distance computed from the references plus the
radii of the two strands: a rigorous lower bound of the distance of their
segment from every reference is then above D_MAX (DRMSD: the cross-strand
distances; RMSD: the radii of gyration). The result does not change (equal
to the sum of the vectors of the expanded input, which compute every pair),
and STRANDS_CUTOFF=0 is much cheaper: ANTIBETARMSD on 144 peptides (370800
pairs) 930 -> 160 ms per step on one core.
Regtests: references updated for the new defaults (only the sheet columns
change); rt-sheet-scalar and rt-GPU-sheet-scalar have pairs with
STRANDS_CUTOFF=0, rt-sheet-scalar prints them next to the sums of the
vectors. CHANGES (results that change, collective variables) and the
migration guide.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016vwxtV46v5FLckcMEcJrbt
ALPHARMSD gets the same default as the sheets: with R_0, NN, MM and D_0
(no LESS_THAN) the switching function has the D_MAX where it is 1e-5 and is
stretched to zero there, as in the other actions that take these keywords.
readShortcutWords always does it (the flag is gone). Checked against the
build before this commit with LESS_THAN={... D_MAX=}: identical values and
forces. The number of helical segments changes by at most 1e-5 per segment
(references of 8 regtests and 2 GPU regtests updated, alpha columns only).
Documentation: the LESS_THAN examples of the three shortcuts give D_MAX,
the LESS_THAN and R_0 keywords explain it, the formula paragraph of
ALPHARMSD explains the stretching, STRANDS_CUTOFF=1 is dropped from the
sheet examples (it is the default), and the expanded inputs in
SECONDARY_STRUCTURE_(D)RMSD show the D_MAX. The single action writes D_MAX
in the log (it is not in the description of the switching function).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016vwxtV46v5FLckcMEcJrbt
…parallel The derivative buffers of the nt threads (3 natoms+9 each) were cleared and summed by one thread. On Leonardo with 32 threads this made ALPHARMSD on 432 segments slower than with 8 (0.81 vs 0.37 ms). The buffers are now cleared with a loop over the nt buffers and summed with a loop over the derivatives, both shared between the threads in the parallel region (still correct when fewer than nt threads run, and with the same order of the sums, so the results do not change), and the radii of the strands are computed in the same region. Timings of the final benchmark (Leonardo, A100 and 8/32 cores) in the plugin README and in CHANGES. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016vwxtV46v5FLckcMEcJrbt
… sum of the buffers Leonardo, 32 cores: the three variables on 144 peptides 7.5 -> 6.1 ms per step, ALPHARMSD on 432 segments 0.91 -> 0.51 ms (0.43 ms on 8 cores). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016vwxtV46v5FLckcMEcJrbt
carlocamilloni
force-pushed
the
openaccPTM-multigpu
branch
from
November 18, 2026 21:12
d5ed95e to
b270260
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Performance with respect to master
Leonardo Booster node (32-core Intel Xeon + one A100), nvc++ 24.5,
plumed benchmarkwith a RESTRAINT on the variables (so the forcesare computed), average time per step in ms without the slowest step. GPU runs use 8 CPU threads. Systems: the amyloid of
regtest/secondarystructure/rt33(18 peptides of 8 residues, 2520 atoms: amy1) and the same replicated 2x2x2 (144 peptides, 20160 atoms: amy2).STRANDS_CUTOFF=1, DRMSD (about 5500 / 370800 pairs of strands for each sheet);STRANDS_CUTOFF=0).cases but nocut-amy2; now it is 3.4 times faster than 32 cores (7 times faster than 8 cores) on the large system.
STRANDS_CUTOFFover the 370800 pairs (about 85 ms) and did notscale with the threads.
transfers (about 0.3 ms per step for ALPHARMSD).
D_MAX(needed for the skipping).USEGPUto the DISTANCE and LESS_THAN of the shortcuts(not part of this PR; it gave no measurable difference on the GPU and does not touch the CPU path).
Summary
[ALPHARMSD], [ANTIBETARMSD] and [PARABETARMSD] now compute the number of segments that are similar to a helix or a sheet with
a single action, on the CPU and (with
USEGPU) on the GPU, instead of the chain of vector actions created by the shortcuts.For large systems this is up to 16 times faster on 32 CPU cores and about 85 times faster on the GPU than master.
ANTIBETARMSD and PARABETARMSD also get new defaults:
STRANDS_CUTOFF=1(nm), and aD_MAXfor the default switching function(also for ALPHARMSD). These defaults change the results of inputs that do not set these keywords; how to get the previous
behaviour is explained below.
The branch also contains the openaccPTM work that is not yet on master (MPI split of the GPU tasks, sums over the neighbours
without matrices, faster linked cells, action data kept on the GPU): see "Also in this branch".
Secondary structure
Single actions
ALPHARMSD_SCALAR,ANTIBETARMSD_SCALAR,PARABETARMSD_SCALAR(src/secondarystructure/SecondaryStructureCount.h/.cpp).They build the segments from the lengths of the chains (for the sheets, as pairs of strands, with the same rules as the shortcuts),
compute the distance of each segment from the references, the lowest one (PARABETARMSD) and the switching function, and add the
derivatives to the atoms in the same pass: there is no backward pass and no vector with an element per segment.
With two references the derivatives are computed only for the lowest one, and only when the switching function is not constant.
label_rmsd,label_lt,label_cut, ...).They are computed only if another action uses them (e.g. a PRINT of the vector with LESS_THAN, or LOWEST); the value of the shortcut,
or its
lessthancomponent, comes from the single action. The results are the same as the sum of the vectors (checked in theregtests
rt-alpha-scalarandrt-sheet-scalar, which print both).USEGPUandTYPE=DRMSD, the same calculation runs on the GPU (*_SCALAR_ACCin the openaccPTM plugin), with the sameper-segment code (shared functions in
SecondaryStructureCount.h). WithUSEGPUand the RMSD types the vectors are used, as before.New defaults (results change)
STRANDS_CUTOFF=1(nm) by default: the segments whose strands (CA atoms of the central residues)are further apart do not contribute. The value is converted to the units of UNITS.
R_0,NN,MM,D_0) of the three variables has aD_MAX, where it is 1e-5(1.42 nm with the defaults), and is stretched to be zero there, as
SwitchingFunction::set(nn,mm,r0,d0)does for the other actionsthat take these keywords. Each segment within the cutoff changes by at most 1e-5.
rt33(18 peptides) the number of antiparallel segments changes by about 1.2 (out of5-58) because of the cutoff; with 144 peptides by about 10 (out of 40-460). ALPHARMSD changes by about 1e-4.
D_MAXis in the expanded input (SWITCHof the_ltand_SCALARactions) and in the log of the_SCALARaction.How to get the previous behaviour
b: ANTIBETARMSD RESIDUES=all(all pairs, function without D_MAX), usingbb: ANTIBETARMSD RESIDUES=all STRANDS_CUTOFF=0 LESS_THAN={RATIONAL R_0=0.08 NN=8 MM=12}, usingb.lessthanb: ANTIBETARMSD RESIDUES=all STRANDS_CUTOFF=1 R_0=0.1b: ANTIBETARMSD RESIDUES=all LESS_THAN={RATIONAL R_0=0.1 NN=8 MM=12}, usingb.lessthana: ALPHARMSD RESIDUES=all R_0=0.1a: ALPHARMSD RESIDUES=all LESS_THAN={RATIONAL R_0=0.1 NN=8 MM=12}, usinga.lessthanSTRANDS_CUTOFF=0uses all the pairs of strands.R_0,D_0,NN,MMused before) gives the function withoutD_MAX;with LESS_THAN the number of segments is the
lessthancomponent (the value of the action is then the vector of the distances).R_0in LESS_THAN is in the length units of UNITS (0.8 withUNITS LENGTH=A).developer-doc/Migration211.md).STRANDS_CUTOFF=1andLESS_THAN={... D_MAX=...}:identical values and forces, for the sheets and the helix.
Exact skipping of the distant pairs
When the switching function is zero beyond a finite distance (the new default, or
D_MAXin LESS_THAN), the single action alsoskips the pairs of strands that are so far apart that their segment certainly has a distance from every reference larger than
D_MAX. The bound uses the distance between the central CA atoms and the radii of the two strands (DRMSD: the distances betweenthe strands in the reference; RMSD: the radii of gyration). The result does not change (equal, to the last digit, to the sum of
the vectors, which compute every pair), and
STRANDS_CUTOFF=0becomes cheap: ANTIBETARMSD on 144 peptides (370800 pairs) takes2.4 ms per step on 32 cores (63 ms on master, with every pair computed).
Documentation
D_MAXare explained in the three shortcuts and in the LESS_THAN and R_0 keywords; the LESS_THAN examples giveD_MAX.Validation
OPTIMAL and OPTIMAL-FAST, with and without NOPBC, STYLE all/inter/intra, LESS_THAN, one and two references;
--debug-forces;MPI (
rt33-mpi, and 2 ranks on the GPU); 1 to 4 threads, also with fewer OpenMP threads than requested and without OpenMP (CI).rt-GPU-sheet-scalar(references from the CPU); compute-sanitizermemcheck and racecheck clean.
secondarystructure/rt-alpha-scalar,secondarystructure/rt-sheet-scalar,targetGPU/rt-GPU-sheet-scalar.The references of the regtests that use the defaults were updated; only the columns of the three variables change.
Also in this branch (openaccPTM and neighbour sums, not yet on master)
MULTIGPU, each on its own GPU (the first processcomputes everything by default); one-pass force gathering by default on the GPU (
AVOID_THREAD_GATHERfor the low-memory version);the data of the actions is kept on the GPU between steps and copied again only when it may have changed (for the secondary structure
this alone took ss-amy2 on the GPU from 152 to 118 ms).
matrix (and the matrices of the spherical harmonics); linked cells built about 2 times faster and in parallel. Details and timings in
CHANGES and in the README of the plugin.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_016vwxtV46v5FLckcMEcJrbt