Repository navigation
Faster neighbour sums on CPU and GPU, multi-GPU support for the openaccPTM plugin. - #1635
carlocamilloni wants to merge 14 commits into
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1635 +/- ##
==========================================
+ Coverage 88.96% 88.99% +0.02%
==========================================
Files 652 655 +3
Lines 81934 82471 +537
==========================================
+ Hits 72890 73391 +501
- Misses 9044 9080 +36 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
484148f to
8ea6396
Compare
6223e6b to
185eb76
Compare
|
Hi @carlocamilloni and @GiovanniBussi I am a bit concerned about all the changes that you are making so rapidly using this AI stuff. I am grateful that you are working to make the code faster but I am also frankly a bit frustrated that you haven't had a conversation with me about the prompts that you are giving the AI engines. In other words, I am frustrated that you aren't discussing what you are trying to achieve with me before you set the agents off to make the changes to the code. The things that you are changing are pieces of codes that have taken me hours to write and I am worried that they will become unrecognisable to me and that only the AI will be able to adjust them in the future. Furthermore, it is frankly tiring to be asked to review a bunch of AI slop text every couple of days with little to no context about what you were trying to achieve. In fact I don't even know what you are looking for me to do when you ask me for a review. On Monday you sent me some code to review, I had a brief look at it, thought to myself, "I will reply later." However, by the time I set out to respond the code had already been merged so any thoughts I had on it were moot. As far as I can tell, one of the big things in this change is that you add new actions so you can avoid calculating the contact matrices and then multiplying them by vectors as I did. In other words, you take these two actions: I know that replacing three actions with one action will likely make the code faster. However, my objective with all the changes that I made to this code when I worked on v2.10 was to make it easier to understand how the calculations are performed. When you introduce combined actions back in to make the code faster you are removing the work that was done to improve comprehensibility. I believe that comprehensibility is more important than performance, especially given AI. If a person knows exactly what they are trying to do then they will be able to optimse the code to that particular purpose. This code, optimised to the particular application the person is working on, will always be faster than the sort of general purpose code that appears in PLUMED. In addition, AI makes it easy to write this type of highly specialised code. If AI makes writing this specialised and optimal code easy what is the purpose of the slow unoptimised code in PLUMED? To be clear, I am not saying that I am not willing to change the code that I have written. I am also not saying that you have to agree with my view that comprehensibility is more important than performance. What I would like to do is slow down and have some agreed vision for what we are trying to achieve here. For example, I have always wondered whether, now that we have disentangled what CVs such as Q6 and so on are computing, the code wouldn't be faster (and easier to maintain) if we just used a library like torch to do these calculations for us. In other words, I think there may be merit in deleting much of the code that I have written and using a library instead. I would also note that when Claude is adjusting the documentation it isn't using $$ around the equations. There is thus a danger that the manual ends up with a mixture of LaTeX equations and raw LaTeX. Professional science has, at times, felt to me like a bit of a meanigless pissing up the wall contest. I have had experiences of meaningful human connection when doing science. I worry, however, that this AI stuff is enhancing everyone's worse instincts and that these moments of meaningful human connection will consequently become even more rare. That is not to say AI shouldn't be used. It is to say that you have to use it in a way that is sympathetic to the people that you are notionally working with. After all, when you move fast and break things the things that get broken are always people. |
|
I agree we should definitely slow down large changes. For me it took quite some time to understand the changes proposed by me via AI My experience so far is that it works very well for localized changes, where it can do systematic improvements that make the code more manageable for the compiler and hence faster with very little maintenaibility cost For larger architectural changes, my experience is that there are an infinite number of ways to do things and it takes a lot of human time and iterations to find the best compromise in terms of performance and readability of the code. I think we need a better plan |
|
@gtribello @GiovanniBussi I agree and disagree :-) I agree that we are changing a lot and very rapidly and i agree that we can talk about these large changes before merging them. Most other changes were not changing the logic of the code itself. Here I pushed the code rewriting to keep the original flexibility but just leave it optional, indeed the code has all the previous capabilites and also still explains the actual internal logic of the actions. To get to this implementation i first began asking to get as much performance as possible without fusing actions but the final analysis was that it was not possible, in particular on GPU, because all the timings pointed to data transfer and memory management, with essentially no cost on the calculation itself (the current main already includes some optimisations) and that this was particularly relevant on the GPU, the analysis suggested a potential to get a x10-x100 speed up by fusing and so i tried. Also note that the way is implemented does not change the logic of the plugin as well, where each action is first implemented on CPU and then only specialised for GPU. Consider that to get with this PR I also run myself a lot of simulations on leonardo to get timings at multple stages. i disagree on other points, I do not learn how MD simulations work by reading GROMACS source code, i use gromacs becasuse it works and it is efficient and I may learn by chaning parameters not the code. I do not think that people use plumed source files to learn the inner working of enhanced sampling or order parameters but they can learn playing with actions. I agree that the ability to implement complex CVs as combination of simpler one is very useful and the manual and the log can still explain what is the set of operations performed and one can still do them in the expanded manner. From this point of view your original idea is still present. But at the same time plumed to me is the same as gromacs but for enhanced sampling and CVs, a state-of-the-art production tool. From this point of view I hope you agree that a GPU implementation that is twice as slow than the CPU is not very useful and that an onverall implementation that can be 10-100 times faster that the previous is preferable. The numbers for secondary structure CVs are even more impressive that this one (they are not yet updated because i still need one more round of validation). documentation latex format can be easily fixed. AI learns by doing, most good practices are already saved, others are added overtime. |
|
I tried to be calm and reasonable this morning but your last message has pissed me off. Please don't straw man what I am saying. I obviously do not think that people learn how these CVs work by reading the PLUMED source code. Similarly, I took pains in my last message to say that I appreciated the work that you are doing to speed up the code. I am not so vain that I think we should have an artificially slow version of the CVs implemented. What I do think is that we do not need a faster implementation of Q6 by next week. I do not think that this is so urgent that there isn't time to discuss with me what you are trying to achieve during one of our regular Monday meetings. Heck if you can't wait that long at least do me the curtesy of writing me an email to explain before you start. That is not hard - this is basic collaboration stuff. Heck, if we aren't in such a rush about getting things done and if take the time to have a conversation we might realise that there is something in what we are doing that is worth putting in an article. |
|
@gtribello i am sorry that my message pissed you off, that was not my intention at all. As you wrote for your side my intention was the same, was that of clarifying the rational of the work with the goal of improving performance while preserving your plan. We can talk on Monday that is easier! |
With MPI each process computes a contiguous block of the active tasks on its own GPU and the value stashes and the forces are summed, as in the CPU task manager; with SERIAL (or one process) the first process computes everything and broadcasts the results, as before. Only the part of the value stash touched by the block is cleared, uploaded and downloaded. - The low memory force gathering (AVOID_THREAD_GATHER, the default on the GPU) works with MPI: in the second pass only the first process keeps the forces of the first one; a process without tasks in the first pass uploads the positions in the second. - GPU choice: without an MD code and with several GPUs on the node (e.g. mpirun -np 4 plumed driver), process i of the node uses GPU i % nGPUs (local rank from the launcher environment), unless ACC_DEVICE_NUM is set. - Shortcuts: DISTANCES, Q1-Q6, LOCAL_Q*, LOCAL_AVERAGE, TETRAHEDRAL-like shortcuts accept SERIAL and pass it with USEGPU; COORDINATIONNUMBER and COORDINATION_MOMENTS accept USEGPU; Q* and LOCAL_AVERAGE put the contact matrix on the GPU too (as LOCAL_Q* already did). - Regtests rt-GPU-mpi-nogather (4 processes, a process without rows), rt-GPU-mpi-serial and rt-GPU-mpi-shortcuts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016vwxtUCrVzSZSsREbiW4jL
…over the neighbours without matrices From the Leonardo benchmarks (A100): splitting the tasks between the GPUs was slower for every input tested, the two-pass force gathering doubled the backward pass, and the matrices copied between CPU and GPU dominate COORDINATIONNUMBER and Q6. - With MPI the GPU actions are computed by the first process again; MULTIGPU (also accepted by the shortcuts that take USEGPU) splits their tasks. - Forces of the matrices: one pass (atomics) is the default on the GPU too; AVOID_THREAD_GATHER asks for the low memory version. DO_THREAD_GATHER is kept for compatibility. - New plugin actions COORDINATIONNUMBER_SUMSACC and Q1-Q6_SUMSACC (and Qn_DENOM_SUMSACC, src/NeighborSums.cpp): sums over the neighbours of each atom on the GPU in one pass, cells in scaled coordinates built on the CPU (any box), minimum image with Pbc, forces with atomics. Used by COORDINATIONNUMBER and Q1-Q6 with USEGPU (not with MASK, nor for COORDINATION_MOMENTS). Same results as the CPU (double precision). - SphericalHarmonic: setup() for a given l, functions in the header inline. - Regtest rt-GPU-neighbor-sums (triclinic box, 2 and 3 cells, no D_MAX, SPECIESA/B, MULTIGPU on 3 processes); the explicit Q6 chain test now runs SPHERICAL_HARMONIC on the GPU; references of two tests now exact. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016vwxtUCrVzSZSsREbiW4jL
The neighbour loop was a macro whose argument contained OpenACC pragmas, which are not processed inside macro arguments: nvc++ (linux-nvhpc CI) saw atomics on invalid expressions and the loop as a routine. The loop is now written out in the two kernels; the action data is a local copy on the device (as in the task manager); pair() is an acc routine seq; epsilon is passed as a firstprivate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016vwxtUCrVzSZSsREbiW4jL
The COMBINE of the 2(2l+1) vectors was slower on the GPU than on the CPU (Leonardo, q6-20000: 6+3 ms vs 4.5+1.2 ms), since its inputs and outputs go to the GPU and back at every step. SERIAL is still passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016vwxtUCrVzSZSsREbiW4jL
With SPECIESA/SPECIESB the forces on the columns are accumulated in one buffer per thread. The buffers were sized with OpenMP::getNumThreads(), which without OpenMP still returns PLUMED_NUM_THREADS, while only thread 0 runs: the reduction read the empty buffer of thread 1 and crashed (symfunc/rt-neighbor-sums, -small, -mpi and rt-q6-subset on the Windows and the Linux no-OpenMP builds of the CI, which use PLUMED_NUM_THREADS=2). The threads that run now mark their buffer, and only those are summed; this also covers an OpenMP team smaller than requested. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016vwxtTud9yy24ZL9dp5N2v
…v variable The sums are copied to the values with transferStashToValues and the forces on the values with transferForcesToStash (the sums already have the layout of the stash): they copy in memory order and with OpenMP. The previous serial loops over the components called Value::set/getForce per element, 15% of Q6 on 20000 atoms with 32 threads. The sums are zeroed only when this process does not compute all the rows. The openaccPTM plugin no longer reads PLUMED_NEIGHBORSUMS_SUBDIVISION: two cells per cutoff is 25-32% faster than one on the A100 too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016vwxtTud9yy24ZL9dp5N2v
… several threads per row, few rows without cells - The pair functors add the values of a pair to the sums (addValues) and contract the forces pair by pair (contract), without storing the derivatives of all the components: for the spherical harmonics the components +m and -m are done together. - Qn_SUMS also computes the coordination numbers in the same pass over the neighbours; Qn_DENOM_SUMS SUMS=label gives them as a vector and passes back the forces on them (it applies before the sums). This replaces the second pass (the ContactPairs Qn_DENOM_SUMS and the plugin's Qn_DENOM_SUMSACC). Qn_NORM computes the norms of the sums in one action; q6_norm2 (COMBINE) is still created and computed only if used. - The search around each row can be split in parts: groups of the rows of cells around it (forEachCandidate/forEachNeighbor with slice, nslices), or blocks of columns without cells. The GPU actions use several GPU threads per row (about 131k threads, at most 25 per row) and sum the parts of each row in order with a second kernel. - Without GROUP and with few rows (at most 8) the cells are not built: the columns are split in blocks shared by the threads (and by the MPI processes), with the minimum images computed in blocks (Pbc::apply) on the CPU. The results do not depend on the number of threads. - TEMPORARY, for the Leonardo measurements: PLUMED_NEIGHBORSUMS_SLICES (GPU parts per row) and PLUMED_NEIGHBORSUMS_FEWROWS (the few-rows threshold). - New regtests rt-neighbor-sums-few-rows (-mpi) and rt-neighbor-sums-parts; the references agree with the contact matrix written out (chain.dat). rt-parse-only: q1_norm is now Q1_NORM (text only). 2-core VM, -O2, ms/step forward+backward, 1/2 threads, before -> after: Q6 2000 atoms 47.9/24.9 -> 25.1/13.1; Q6 20000 528/270 -> 285/146; COORDINATIONNUMBER 2000 and 20000 3-5% faster; one atom against 100000 waters (SPECIESA/SPECIESB) 7.4/5.4 -> 2.7/2.0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016vwxtTq4ZtJbDFt8nrMV3A
…ne site against many atoms
- COORDINATION (and the other users of NeighborList) with NLISTCELLS builds the list with the
plain search when one of the two groups has at most 8 atoms: sorting all the atoms into
cells costs more than one distance per pair (equal cost at about 8 atoms). One atom
against 100000 water oxygens, list updated at every step: 7.9 -> 3.4 ms/step.
- Docs: COORDINATION ("One atom against many atoms"), COORDINATIONNUMBER and CONTACT_MATRIX
say which input to use for one site against many atoms. CHANGES/v2.11.md updated.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016vwxtTq4ZtJbDFt8nrMV3A
…lways use the core’s OpenMP. The plugin now runs no OpenMP code of its own. I’ve noted this as a rule in the progress doc. GPU: one thread per row when the cells are used. With few rows, the columns are split into blocks between the GPU threads. The splitting of a row’s cells is gone from LinkCellsView.h, which is back to master’s version. Few-rows cutoffs: on the CPU, 2 × the number of threads (at least 4), as you chose; on the GPU, 512. Both temporary environment variables are gone, and the plugin README is updated.
… plugin On one A100 (8 CPU threads) / 32 CPU cores, ms per step: COORDINATIONNUMBER 3.7 / 4.7 (20000 atoms), 17 / 25 (100000); Q6 13 / 17, 59 / 94 (Leonardo, round 8). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016vwxtTq4ZtJbDFt8nrMV3A
…ings With the cheaper cells (Leonardo, 32 threads, 100000 columns) all the pairs cost as much as the cells at about 44 rows: 64 rows took 4.03 ms without cells against ~3.5 with them. The threshold is now the number of threads (at least 4; unchanged up to 4 threads). CHANGES and the plugin README have the round-9 timings. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016vwxtTq4ZtJbDFt8nrMV3A
d5ed95e to
b270260
Compare
runAllTasksACC and applyForcesWithACC took the action data (input_type) by value, a deep copy on the host, and moved it to the device with enter/exit data at every call, twice per step. For large action data this dominated the time of the action: e.g. the 370k segments of the secondary structure variables of 144 peptides are 45 MB of indices, and SECONDARY_STRUCTURE_DRMSD_ACC took 14 ms forward and 10 ms backward per step with 5000 active segments (4 and 1 ms on the CPU). The task manager now keeps a copy of the action data on the device of the action and passes the kernels a pointer to it (present(ad[0:1])). The copy is made and uploaded again only when the data may have changed: ParallelTaskManager counts the calls of setActionInput and of the non-const getActionInput (getActionInputGeneration), which is how actions change their input (e.g. the neighbour lists of the contact matrix, the regions of the volumes, the bookkeeping of the matrix products), so those actions still upload it at every step, as before. The copy is removed from the device when the task manager is destroyed. Tested with the plugin built with g++ and the host stub of the OpenACC runtime (51 targetGPU regtests pass; the stub ignores the data directives, so the residency itself is only tested with nvc++). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016vwxtV46v5FLckcMEcJrbt
Summary
Faster neighbour sums on CPU and GPU, and multi-GPU support for the openaccPTM plugin.
USEGPU).MULTIGPU), the GPU chosen by node-local rank, and one-pass force gathering by default.Values are unchanged. Forces change only in the last digits (relative 1e-14), and with
SPECIESthey no longer depend on the number of OpenMP threads.Overall, on a Leonardo Booster node (32-core Xeon 8358, A100), 20000 atoms:
See "Performance" below for the baselines and for a same-build comparison.
What changes
1. Fused neighbour sums (core,
src/symfunc/NeighborSums.{h,cpp})COORDINATIONNUMBER_SUMS,Q1/Q3/Q4/Q6_SUMS,Qn_DENOM_SUMS,Qn_NORM. They are created by the shortcuts as suffixed variants (not meant to be written by hand).MASK, COORDINATIONNUMBER and Q* compute the sums per atom directly from the linked cells:LinkCellsView;q6_denom) in the same pass as the sums of the spherical harmonics;q6_norm) computed by one action instead of a COMBINE of 2(2l+1) vectors.SPECIES, each atom gathers its force from its own neighbours: no atomics, and the forces do not depend on the number of threads;SPECIESA/SPECIESB, the column forces go to per-thread buffers that are reduced in a fixed order.cn,q6_sp.*(exactly the 2(2l+1) components),q6_denomandq6_normkeep their labels and shapes.cn_mat,q6_mat,q6_sh, …) are still created and are computed only if another action uses them.q6_norm2(the COMBINE) is still created, dormant.SPECIESA=1 SPECIESB=<all waters>). Up tomax(4, threads)rows on the CPU, or 512 rows on the GPU:SERIALis accepted.D_MAX, or with a box smaller than 3 cutoffs, all pairs are computed withPbc::distance.2. openaccPTM plugin
USEGPU, COORDINATIONNUMBER and Q* use the same row kernels on the GPU (*_SUMSACC):MULTIGPUsplits the tasks between the ranks, each on its own GPU;SERIALis accepted by all the shortcuts that takeUSEGPU(DISTANCES, Q*, LOCAL_Q*, LOCAL_AVERAGE, TETRAHEDRAL-like);ACC_DEVICE_NUMis not set, a rank uses GPU (node-local rank mod number of GPUs). The node-local rank is read from the OpenMPI, MVAPICH, Intel MPI, PMI, PALS, Flux or SLURM variables. The choice is written in the log.AVOID_THREAD_GATHERgives the previous two-pass, low-memory version.USEGPU.USEGPU:* the norm is computed on the CPU (it was slower on the GPU).VMEAN/VSUMthe GPU flags were passed to a COMBINE of scalars, which rejects them.3. Docs
How to get the previous behaviour
MASKand for COORDINATION_MOMENTS. To force it elsewhere, write the chain out: CONTACT_MATRIX + MATRIX_VECTOR_PRODUCT for the coordination numbers; for Q*, the CONTACT_MATRIXCOMPONENTS+ SPHERICAL_HARMONIC + MATRIX_VECTOR_PRODUCT chain of the expanded shortcut (e.g.plumed driver --parse-only, or the manual). There is no keyword to switch off the fused path.Qn_NORM) and denominator in the same passUSEGPUwithout the plugin loadedUSEGPU, as before.MULTIGPUis opt-in.ACC_DEVICE_NUM(or let the MD code assign the GPU) to choose the GPU explicitly.AVOID_THREAD_GATHERgives the two-pass, low-memory version (the old GPU default; about two times slower, uses less memory).fewRows(): CPUmax(4, threads), GPU 512). Results agree with the cells path to 1e-13.Output that changes:
AVOID_THREAD_GATHERexplicitly.Q1_NORM).No other reference changed.
Performance
How to read the numbers.
plumed benchmark, forward plus backward, averaged without the slowest step.Same build, fused vs matrix chain (2-core VM, -O2; the chain written out by hand; includes the cells of half the cutoff already in master)
The columns come from successive sessions on the same VM, which is noisy (±10%). The last column is the code before the linked-cells and few-rows changes, which are measured on Leonardo below.
Leonardo (32 CPU cores / 1 A100 with 8 host threads)
The "now" columns include the faster linked cells of
linkcells-threaded(already in master): they took COORDINATIONNUMBER 20000 on 32 cores from 4.70 to 3.96 ms.One site against many atoms (one atom or a few against 100000 water oxygens)
SPECIESA=1(2-core VM, 1 / 2 threads)NLISTis still the fastest (0.59 ms withNL_STRIDE=10). The docs say so.Linked cells alone (Leonardo, µs per build, 20000 / 100000 atoms; from
linkcells-threaded, already in master)The old code stopped improving at 4 threads, because placing the atoms in the cells was serial. With the new cells, COORDINATIONNUMBER with 20000 atoms on 32 cores goes from 4.70 to 3.96 ms per step.
GPU force gathering (the matrix chain on the GPU, COORDINATIONNUMBER 20000 written out)
Tests
chain.datin each directory) and agree with the fused output at the regtest precision.PLUMED_NUM_THREADS=2, both with gcc+OpenMP+MPI and with clang without OpenMP;