Abort undersized auto-distributed Qureg instead of replicating it

[iarejula-bsc]

In a multi-node environment, createQureg() can currently return a Qureg which is replicated across every node rather than distributed between them. Every node then redundantly simulates the same state, so all but one of the allocated nodes do no useful work, and nothing warns the user. The auto-deployer reaches that outcome by two routes:

• the state has too few amplitudes to divide evenly between the nodes (numQubits < log2(numNodes)), so it cannot be distributed at all
• the state would divide evenly, but distribution is judged not worthwhile, because each node would be left with fewer than MIN_NUM_LOCAL_QUBITS_FOR_AUTO_QUREG_DISTRIBUTION (26) local qubits.

The solution provided is to require that an automatic deployment utilises every node of the environment, and to abort with a clear error otherwise, the same way already done in validation (e.g. the "node count must be a power of 2" check), so callers can react instead of QuEST silently proceeding. Each route reports its own reason: how many qubits would be needed to divide the state between the nodes, or how many amplitudes each node would be left holding.

Explicitly requesting a non-distributed Qureg (via createCustomQureg() or createQuregFromEnvDeployments()) remains permitted, since that replication is deliberate; only deployments left to the auto-deployer are affected.

reportQuESTEnv() now reports the sizes which can no longer be automatically deployed, rather than advertising a deployment which createQureg() would reject.

I would like to discuss the use of Malleability instead of aborting. Where the runtime supports malleability (DMR library), a better response is available, shrink the job to a node count at which the Qureg does distribute, and carry on, which would turn this error into a resize and leave the abort as the fallback for whatever malleability cannot cover. This would not affect the core of Quest and would be low intrusive, as it would only modify the autodeployer, which i think it fits the use of malleability.

Disclosure: this contribution was developed with support of Opus.

[TysonRayJones]

Could you elaborate in words the decision made? It sounds like you propose to remove the automatic disabling of distribution. The existing functionality is very useful to ensure that the chosen parallelisations/deployments are performant for the user's Qureg. Forcing distribution, when the environment is distributed, will pointlessly cripple performance. Keep in mind that a user may be creating multiple Qureg of varying sizes within the same execution, and it is senseless to force the small ones to distribute their data. Please see instead createForcedQureg

[iarejula-bsc]

I am not removing the distribution, but the replication (same qureg on all nodes).

This comes because I have been doing benchmarks on Quest for the last months. I realized that after the 26 local qubits limit, the autodeployer just wastes resources. As you can see in the image, with 32 nodes it uses all resources but only 1 node is useful, as the rest are replicas with the same result. Personally, I do not see why this is worth it, nor was I aware of this behavior until I saw the benchmarks and analyzed the code to understand the issue of 32 nodes being the worst performant (which is not a good ux, cuz if i did not run the benchmark i wouldnt be aware of 32 is worse than 16 and if i did not read the code i would not know the reason of this behaviour)

The idea of this PR is to notify the user and avoid wasting resources, but still allow the user to use this functionality in the "no auto" mode, if they really want it.

I understand that this limits the scenario where a user allocates 16 nodes for a circuit of 30 qubits and another of 5 qubits: the first one distributes, and the second one fails. I believe that not wasting resources is the better default, with wasting them being opt-in, I am open to discussing it if you disagree and think it is worth to discuss it.

However, I feel the best option here is to use malleability to reallocate the allocated nodes, but that is another discussion.

[TysonRayJones]

Hi there,

There still seems to be a misunderstanding - the "wasting" of computation across processes, when a Qureg has been replicated across them, is deliberate. It is a speedup in settings where some Qureg are distributed, and others are not - which is the expected and standard scenario when a user is deploying a multi-node simulation (otherwise, there is no reason a user should ever launch their <26 qubit job with multiple MPI processes; they are inviting a slowdown!).

Perhaps you can see the speedup more clearly when it comes time to perform some two-Qureg computation between a distributed and non-distributed Qureg. The non-distributed Qureg must contain corresponding data in every process spanned by the distirbuted Qureg. Under your proposal of not-duplicating the non-distributed Qureg, then at that point in time, we must broadcast the non-distributed Qureg from some process, say 0, to every other process. That broadcasting is superfluous communication, which is our main contribution to runtime in our typical network-bandwidth-bound settings. So we are pointlessly waiting for the single process responsible for the non-duplicated Qureg to broadcast its entire state, to every other node! That's a greater bottleneck then typical pair-wise communication between processes (QuEST's main cost during gate application), since at least there, all pairs of nodes send/recv independently of one another. But in this scenario, process 0 does all the sending!

Consider instead the current scheme, where every node is always simultaneously evolving the non-distributed Qureg. At the time of the two-Qureg computation with a distributed Qureg, no communication is necessary since every process already has all needed data, and the computation can proceed immediately.

The superfluous communication invited by not-duplicating non-distributed Qureg is actually still worse than this. QuEST boasts API agnosticism to the deployment mode; so decisions the user makes in their control flow (e.g. if (getProb(smallQureg) > 1): applyX(bigQureg)) should always work, even when distributed. But in your scheme, this logic will fail. The user would need to broadcast the result of getProb(smallQureg), which is only known to process 0, to all other processes which share bigQureg. Otherwise, the other processes do not know the output of getProb(), cannot determine the if branch faithfully, fall out of consensus/synchronisation, and the system will likely deadlock.

So, lots of additional, superfluous, slowing communication is necessary when trying to let "non-processes relax" during simulation of a non-distributed, non-duplicated Qureg. It's a strict slowdown. Hence, your benchmarking results are very suspicious to me. I suspect you are measuring a special case without performing the additional communication needed of the general case (maybe you're simulating a single Qureg - the scenario where a distributed launch is totally pointless). That you see duplication causing a slowdown is very suspicious; process 0 performing superfluous, embarrassingly parallel computation should not slowdown independent process 1, for example. Perhaps you are oversubscribing a single machine to pointlessly use distribution? This will, of course, not yield representative performance data