Skip to content

CI fails on a slow runner: the author's tests run on vitest's 5 s limit, raised one test at a time #255

Closed

Description

@alekswebnet

CI on main failed again on 2026-11-20 (run 44701664919, the merge of #254): in packages/author, bundle.test.ts "writes every content type's or none" timed out at 5011 ms.

It is the third failure of one kind in three days, and each was another test:

Date Run Test Took
2026-11-18 main build.test.ts, an Interactive Video's time past the end of its video 5010 ms
2026-11-19 #244's branch bundle.test.ts, bundle --all 5025 ms
2026-11-20 main bundle.test.ts, every content type's bundle or none 5011 ms

They are 3 of the 88 CI runs since 2026-11-13, and the only ones that failed for no fault in the change. Each time the test that failed got a limit of its own, 20 s (#242), and the next one failed.

Why

The author's Node tests run on vitest's default limit, 5 s a test. They build packages from the 31 MB library pack and write every content type's bundle, which a CI runner does two to four times slower than a laptop. Over three CI runs:

Test Fastest Slowest Its limit
The Course Presentation example, built, read and built again 10.5 s 13.1 s 60 s, its own
The Interactive Video example, the same 5.8 s 7.5 s 60 s, its own
bundle --all 4.1 s 5.4 s 20 s, its own
An Interactive Video's time past the end 4.1 s 5.3 s 20 s, its own
Every content type's bundle or none 3.6 s 5.0 s 5 s, the default
A poster sized by what it holds 2.3 s 3.1 s 5 s, the default

So a test passes only while somebody has given it a number, and any new one that builds a few packages is the next failure. embed, mcp, the demo and the skills project set one limit for the whole package, 120 s, and none of them has timed out.

Proposed

  • The limit is the package's. The author's Node project sets testTimeout and hookTimeout to 120 s in its config, as those four do. The five limits tests carry there go: each is lower than the package's, or the same.
  • Verify gets the same, in a config of its own. Every test of it starts a browser, and each heavy one carries its own number today, so a new one without a number would fail the same way.
  • The rule is written down in design/testing.md: a limit stops a hang and does not time a test, so it is set once for a package, for a slow runner, and a test sets its own only where the time is what it tests.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions