Description
The failureRunner - shares sink base across remote fault mix test in packages/cli/src/bench/scenarios/failure.test.ts is sensitive to CI load.
The test runs a closed-loop load plan for 25 ms with a concurrency of one. It then expects more than one request to have completed and checks that both alternating recipient inboxes were used:
assert.ok(measurement.requests.total > 1);
Remote fault requests are serialized, and the load generator stops dispatching once the wall-clock deadline has passed. If the first asynchronous request or CI scheduling consumes most of the 25 ms window, the measurement contains only one request even though the failure runner is behaving correctly.
This happened in the Node.js test job for PR #949:
https://github.com/fedify-dev/fedify/actions/runs/29640927713/job/88071217632?pr=949
The job passed 558 of 559 tests. The failing test took about 87 ms and failed because measurement.requests.total was not greater than one. PR #949 does not modify the CLI package or this test path, and the same test passed on the base commit and in the Bun job.
I also ran the isolated test 30 times locally without reproducing the failure. That result, together with the short real-time deadline and the unrelated PR changes, suggests that this is a timing-dependent flaky test rather than a regression.
Expected behavior
The test should verify that the remote fault variants share the configured sink base without depending on how many asynchronous operations a runner can complete within 25 ms.
Suggested approach
Use the Clock injection already supported by the failure runner to control time deterministically, or structure the test around a fixed number of dispatches. The test should still verify that both alternating inboxes are reached and that the requests succeed.
Increasing the duration alone would make the failure less frequent but would leave the test dependent on host load.
Description
The
failureRunner - shares sink base across remote fault mixtest in packages/cli/src/bench/scenarios/failure.test.ts is sensitive to CI load.The test runs a closed-loop load plan for 25 ms with a concurrency of one. It then expects more than one request to have completed and checks that both alternating recipient inboxes were used:
Remote fault requests are serialized, and the load generator stops dispatching once the wall-clock deadline has passed. If the first asynchronous request or CI scheduling consumes most of the 25 ms window, the measurement contains only one request even though the failure runner is behaving correctly.
This happened in the Node.js test job for PR #949:
https://github.com/fedify-dev/fedify/actions/runs/29640927713/job/88071217632?pr=949
The job passed 558 of 559 tests. The failing test took about 87 ms and failed because
measurement.requests.totalwas not greater than one. PR #949 does not modify the CLI package or this test path, and the same test passed on the base commit and in the Bun job.I also ran the isolated test 30 times locally without reproducing the failure. That result, together with the short real-time deadline and the unrelated PR changes, suggests that this is a timing-dependent flaky test rather than a regression.
Expected behavior
The test should verify that the remote fault variants share the configured sink base without depending on how many asynchronous operations a runner can complete within 25 ms.
Suggested approach
Use the
Clockinjection already supported by the failure runner to control time deterministically, or structure the test around a fixed number of dispatches. The test should still verify that both alternating inboxes are reached and that the requests succeed.Increasing the duration alone would make the failure less frequent but would leave the test dependent on host load.