Skip to content

error-handler: warnings from fibers started by the test and from failing tests are lost #430

Open

Description

@vudaltsov

Summary

With ErrorHandlerPlugin(failOnError: true) (testo 0.10.59, testo/error-handler 0.1.1), two kinds of PHP warnings
are lost entirely: the test result does not reflect them and they do not appear anywhere in the output.

  1. A warning raised by a fiber that the test started (a task of an event loop), while the fiber of the test is suspended.
  2. A warning raised by the test itself when the test then fails on something else, typically an assertion that fails
    because of the warning.

Both come from the design: the plugin takes its handler off the stack while the test fiber is suspended, and it judges
errors only after the test has finished.

Reproduction

A toy event loop, in the spirit of any fiber scheduler: the test runs in a fiber, and the tasks it spawns run
while it is suspended.

final class Loop
{
    /** @var list<\Fiber> */
    public static array $ready = [];

    public static function spawn(\Closure $task): void
    {
        self::$ready[] = new \Fiber($task);
    }

    /** Lets the spawned tasks run, as an await on I/O or a timer would. */
    public static function yield(): void
    {
        \Fiber::suspend();
    }

    public static function run(\Closure $main): mixed
    {
        $fiber = new \Fiber($main);
        $fiber->start();
        while (!$fiber->isTerminated()) {
            while (($task = array_shift(self::$ready)) !== null) {
                $task->start();
            }
            $fiber->resume();
        }

        return $fiber->getReturn();
    }
}

final class LoopInterceptor implements TestRunInterceptor
{
    public function runTest(TestInfo $info, callable $next): TestResult
    {
        return Loop::run(static fn() => $next($info));
    }
}

final class LoopPlugin implements PluginConfigurator
{
    public function configure(Container $container): void
    {
        $container->get(InterceptorCollector::class)->addInterceptor(LoopInterceptor::class);
    }
}
// testo.php
return new ApplicationConfig(
    suites: [
        new SuiteConfig(
            name: 'repro',
            location: [__DIR__ . '/tests'],
            plugins: [new ErrorHandlerPlugin(failOnError: true), new LoopPlugin()],
        ),
    ],
);
final class ReproTest
{
    #[Test]
    public function warningInSpawnedTask(): void
    {
        Loop::spawn(static function (): void {
            $data = file_get_contents('/no/such/file');
        });
        Loop::yield();

        Assert::true(true);
    }

    #[Test]
    public function warningInTheTestItself(): void
    {
        $data = file_get_contents('/no/such/file');

        Assert::same('', $data);
    }
}

Result (PHP 8.6, display_errors=1, error_reporting=E_ALL):

 ReproTest [test]
   ✓ warningInSpawnedTask (20ms)
   ✗ warningInTheTestItself (8ms)
...
    Failed assertion that `""` is the same as `false`: expected `false`, got `""`.

No such file appears nowhere in the output of the run, for either test.

Why

1. The handler leaves the stack while the test fiber is suspended

ErrorHandlerInterceptor::run() wraps the test into a fiber of its own and, on every suspension,
calls HandlerScope::suspend(), which removes the capturing handler (and everything above it) until the test
is resumed. The intent is good: sibling tests that are interleaved in one process must not see each other's errors.

But for a test that drives an event loop, "the test fiber is suspended" is exactly the moment when the work
of the test happens: the tasks it spawned, timers, I/O callbacks. Their errors are raised while the plugin's handler
is not installed, so they are neither captured nor reported, and the test passes. The more asynchronous the code,
the less the plugin sees: in a typical async test almost all code under test runs outside the test fiber.

There is no way to tell "a fiber of an interleaved sibling test" from "a fiber that this test started" by looking
at the stack alone. Some ideas:

  • an opt-out of the per-suspension scoping for suites that do not interleave tests (it is a property of the runner,
    so the plugin may even know it on its own);
  • attributing an error to a test by the fiber that raised it and the fibers that test has started, if the runner
    can tell which test a fiber belongs to.

2. The error is judged after the test, and only for a passing test

failOnError upgrades only a passing test. A warning usually makes a function return false/null, the test
goes on with that value and fails on the next assertion. The failure then names the symptom
(expected false, got ""), and the cause is gone: it is not in the failure, and the stderr message
that the plugin would have written is not in the output either (I suppose because PHP's own printing is suppressed and the message log of a failed result is not rendered, but I have not dug into it).

At the very least, the captured errors could be attached to the output of a failed test too. A stronger option
is a mode that throws \ErrorException from the handler at the point of the error (like PHPUnit's
convertWarningsToExceptions used to): the test then stops at the line that caused the problem, with its stack
trace, and the error cannot be lost by later code or by a later failure. For code that runs in fibers, throwing
in place is also the only behavior that reports the error in the context where it happened.

Environment

  • testo/testo 0.10.59, testo/error-handler 0.1.1
  • PHP 8.6.0RC7, Linux

Activity

  1. roxblnfk commented on Nov 19, 2026

    @roxblnfk
    Member

    Thanks for the thorough report. Three different things turned out to be behind it.

    failOnError was never on in the repro. A plain plugins: [...] list is merged with the default plugins, and the default ErrorHandlerPlugin instance won over yours. Fixed in #434: the instance configured for the suite now overrides the default.

    Warnings from spawned tasks. The loop interceptor needs to sit closest to the test:

    #[InterceptorOptions(order: InterceptorOptions::ORDER_ASYNC_COROUTINE)]
    final class LoopInterceptor implements TestRunInterceptor

    That is the slot for an interceptor that drives the test fiber itself, see RunInRevoltInterceptor. At the default order the loop sits outside the per-test guards. When the test suspends, they step aside, including the error handler, so the tasks run outside the test's context. In the inner slot the loop stays inside it. With your repro and only the order changed, warningInSpawnedTask fails with ErrorException. Documented for plugin authors in #439.

    Warnings lost on failed tests. This one was a real bug: the plugin's messages were dropped before reaching any reporter, for passing and failing tests alike. Fixed in #436 and #437. A warning now shows on STDERR as soon as it is raised, next to the test that raised it.

    Throwing \ErrorException at the point of the error is tracked in #438.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions