The load-balancer dataplane gate asks for a spread the runtime hands out at random: six connections landed on one backend, and the suite read a working balancer as one that distributes nothing #559

Client: octl / the runtime's own load balancer · Runtime: --vm incus-ovn, Incus 7.2.0 · Suite: tools/conformance/outscale/balancer.sh · Measured: 2026-08-27

The witness

FEINT_VM=incus-ovn mise run conformance failed 1248 s in, at the balancer dataplane suite, on a balancer that was working:

- an internal load balancer, its two machines registered
  ok: feint-lbu-data answers on 10.186.7.7
- the balancer distributes to both machines
FAIL: no connection reached i-3b55a58b:  i-7bdee39d i-7bdee39d i-7bdee39d i-7bdee39d i-7bdee39d i-7bdee39d

Both backends had answered directly in the same pass (ok: both backends answer directly), both were registered, and the emulator logged no withheld-backend warning for this balancer — so the spec was whole and the two backends were in it. Six connections simply landed on the same one.

Why it is the assertion and not the balancer

The suite was replayed three times on the same tree, each against an emulator of its own. All three passed, and the distributions are the finding:

And within one replay, the second probe set of the same balancer split 3/3 where the first had split 5/1. The runtime hashes each connection independently; nothing about the balancer changes between the two probe sets.

So case "$hits" in *"$vm_a"*) over six probes asks for a spread the runtime never promised. With two backends and an independent choice per connection, the run is red whenever all six land the same way: 2 × (1/2)⁶ ≈ 3.1 % of runs, with the balancer working perfectly.

Why it matters here rather than being a nuisance

Three percent is often enough that the answer becomes "re-run it". That is the reflex CLAUDE.md names as worse than having no gate — and it lands on the one suite whose whole subject is the dataplane, so the next real distribution defect will be read as "the flake again". #483 is exactly a balancer that distributed nothing, and this assertion is what stands between that defect and a green run.

Two ways out, and they are not equivalent

• More probes. Twenty-four connections put the all-one-side probability at 1.2 × 10⁻⁷. It keeps the property the suite means to state and costs eighteen more curls.
• Assert what the runtime promises. fnl_balancer_delivers in tools/conformance/functionallib.sh already takes the other road for the same question: every reply must name a backend the pack recorded as delivered, and none may ever name one it recorded as withheld. That is a property of the runtime's own record rather than of a hash, and it cannot flake. It does not, on its own, catch a balancer that stopped using one of two live backends — which is what the six-probe spread is reaching for — so the two are complementary rather than a swap.

What is not measured

• Whether OVN's affinity_timeout (off by default here) or the sequence of source ports changes the distribution.
• Whether the skew is the same with three or more backends, or with a client outside the balancer's own subnet.
• The same assertion in tools/conformance/exoscale/ and in the stacks: only the Outscale suite was replayed.

The test that fails without a fix

None, and that is the shape of it: the control is red at random rather than never, so nothing distinguishes a run that measured the property from a run that got lucky.