[w180112]

Summary

Distributor workers call rte_distributor_get_pkt() to pull packets off
the rte_distributor, and go RCU-offline while blocked there. When the
ctrl thread resets a subscriber's config, it closes the per-ccb data-plane
gate and then calls rte_rcu_qsbr_synchronize() to wait for every lcore to
pass a quiescent point before touching the NAT / IPv6 firewall tables. That
synchronize does not wait for an offline worker, so a packet the RX lcore
already passed through the gate and handed to the distributor can still be
sitting in flight when the reset completes, and the worker can go on to
process it against the just-reset tables after it comes back online.

This PR closes that window from the worker side: right after a distributor
worker comes back online, before it touches any per-ccb table, it re-checks
the same gate the RX lcore already checked and drops the packet if the gate
has since closed. The worker loops are extracted into
dp_wan_dist_worker_process_pkt() / dp_lan_dist_worker_process_pkt() so
the gate re-check (and the rest of the per-packet logic) is unit-testable.

What changed

• src/dp.c / src/dp.h: extract the single-packet body of the WAN/LAN
  distributor worker loops into two functions and add the gate re-check
  (pppd_dp_gate_open for IPv4, pppd_ipv6_dp_gate_open for IPv6) right
  after the worker comes back online, before any per-ccb table is touched.
  A closed gate drops the packet and counts it against the matching port.
• unit_test/dp_test.c: new tests for both worker functions covering
  gate-closed (packet dropped, hash/tables untouched) and gate-open
  (original processing path unchanged).
• e2e_test/phases/phase41_distributor_mode.sh (new) plus a small
  datapath_mode_lib.sh helper lib: a new e2e phase that switches the node
  to software distributor mode (EnableDDP = false) and re-runs the
  suite's IPv4 NAT UDP/TCP traffic and IPv6 firewall UDP traffic through
  it, checking that both WAN and LAN distributor workers actually move
  packets. The original config is saved before the switch and restored
  byte-identically afterward (verified by sha256), and the restore is also
  wired into the harness's existing cleanup/trap path so an interrupted run
  still puts the node back in its original mode.
• Assertions added by this phase go through small lib predicates
  (e2e_counter_increased, e2e_all_flows_replied, e2e_dist_worker_roles)
  registered with the harness's self-validation fixtures, including
  negative samples that confirm each predicate actually catches the defect
  it targets.
• e2e_test/phases/phase37_ipv6_firewall.sh: the IPv6 fixture setup used by
  the existing phase 37 is factored out so phase 41 can reuse it unchanged.
• e2e_test/run_e2e_test.sh: wires in the new phase and lib, and renames
  phase41_summary.sh to phase42_summary.sh so the new distributor-mode
  phase keeps its natural slot ahead of the summary phase.

Step 190 root cause (IPv6 UDP through the distributor)

While developing phase 41, the new IPv6 step was flaky under one ordering:
if the node is switched into distributor mode and the IPv6 UDP burst is
sent immediately, the lab BRAS has not yet learned the WAN host's MAC via
neighbor discovery and drops every packet as drop:route until it does.
The lab BRAS only sends a neighbor solicitation retry every 10 s once past
its fast-retry window, so a burst that lands inside that window can be
dropped in full.

This was confirmed with controlled repro runs against the real bench BRAS:
3/3 runs that add the WAN address right after a solicitation tick and then
burst immediately reproduce the failure verbatim (flows_with_reply=0 of 16, all 320 packets recorded by the BRAS as drop:route, BRAS resolves
the upstream MAC several seconds after the burst ended); 3/3 runs that wait
for the BRAS to resolve the MAC (observe its neighbor advertisement) before
bursting pass cleanly (16/16, zero drops). This is a timing artifact of the
lab's neighbor-discovery retry interval, not a data-plane defect — the
existing phase 37 IPv6 step already avoids it because it pings first.

The fix is for Step 190 to wait on the existing _p37_ping6_ok helper
(same one phase 37 uses) before running the UDP echo burst, which forces
neighbor resolution to complete first.

Test results

• make test: all 2440 tests passed.
• make all: production rebuild, clean.
• Full e2e (no flags — local validation + full run + full drill run):
  • LOCAL CORRECTNESS VALIDATION: 194/194 predicates OK
  • Total: 206 Pass: 206 Fail: 0 Skip: 0 / RESULT: ALL PASSED
  • VALIDATE RESULT: 26/26 caught, 0 polluted

Note

On its own, this change only narrows the race window; it does not close it
completely. The reset path (ppp_init_config_by_user) closes the gate and
clears the NAT / IPv6 firewall tables, but does not yet call
rte_rcu_qsbr_synchronize() between those two steps on this branch's base,
so a worker that is online and already past this re-check at the moment of
reset can still race the table clear. Full protection depends on a
separate, already-written fix that adds that synchronize call landing on
master first.

🤖 Generated with Claude Code