Skip to content

test: a test's time limit is its package's, for the author package and the verifier - #256

Merged
alekswebnet merged 2 commits into
mainfrom
fix/test-limits-per-package
Nov 20, 2026
Merged

alekswebnet merged 2 commits into
mainfrom
fix/test-limits-per-package

Conversation

@alekswebnet

@alekswebnet alekswebnet commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Closes #255.

CI on main failed after #254 merged: a test in packages/author timed out at 5011 ms on vitest's 5 s default. It is the third such failure in three days (5010, 5025 and 5011 ms), another test each time. After each one, the test that failed got a number of its own, and the next one failed.

What changes

  • packages/author/vitest.config.ts: the Node project sets testTimeout and hookTimeout to 120 s, as embed, mcp, the demo and the skills project already do for the whole package.
  • The author's tests lose the five numbers they carried (20 s twice, 60 s twice, 120 s once): each is lower than the package's limit, or the same.
  • packages/verify/vitest.config.ts is new, with the same limit. Every test there that plays a package starts a browser, and each heavy one carried its own number, so a new one written without a number would have failed on the default.
  • The verifier's tests lose their sixteen numbers of 120 s or less. The three that ask for 240 s stay, and so does { timeout: 1000 } on the test whose point is the time.
  • design/testing.md has the rule: a limit stops a hang and does not time a test, so it is set once for a package, for a slow runner.

Why this and not another number on one test

Over three CI runs, these are the author tests that take longest on a runner:

Test Fastest Slowest Its limit before
The Course Presentation example, built, read and built again 10.5 s 13.1 s 60 s, its own
The Interactive Video example, the same 5.8 s 7.5 s 60 s, its own
bundle --all 4.1 s 5.4 s 20 s, its own
An Interactive Video's time past the end 4.1 s 5.3 s 20 s, its own
Every content type's bundle or none 3.6 s 5.0 s 5 s, the default
A poster sized by what it holds 2.3 s 3.1 s 5 s, the default

The last two had no number. The first of them failed this run, and the second is next.

Tested

  • Under load: bundle.test.ts and layout.test.ts with 30 busy processes on 10 cores, to stand in for a slow runner.
    • On main, the two tests with no number fail, at 5006 and 5014 ms.
    • On this branch all 33 pass.
  • The suites: the author package's 278 tests and the verifier's 29 pass, Chromium included.
  • Type check: passes in both packages.

Not changed

  • The player's unit project stays on the default. Two tests there carry 30 s of their own and run in under a second on a runner. Moving them would start the player's whole browser suite for this change.
  • cmi5, normalize, the runtime and the extension's tests run in milliseconds and keep the default. The rule says when a package gets a limit: when one of its tests first needs a number.

Summary by cubic

Moves test time limits from individual tests to the package config for packages/author and packages/verify, fixing CI failures from vitest's 5-second default on slow runners.

  • Sets testTimeout and hookTimeout to 120 s in both packages' vitest.config.ts, and to 180 s in the author's browser project.
  • Removes per-test timeouts of 120 s or less from the author's and verifier's tests; the three 240 s limits and the test measuring time stay.
  • Documents in design/testing.md that a package sets its own limit, and a test names one only to ask for more or where time is what it tests.

Written for commit 9b61284. Summary will update on new commits.

View guided diff Turn on auto-fix

…d the verifier (#255)

CI failed on main for the third time in three days on a test of
packages/author crossing vitest's 5 s on a slow runner: 5010, 5025 and
5011 ms, another test each time, each after the one before it got a
number of its own. Two more ran within reach of it, at 3.6 to 5.0 s and
2.3 to 3.1 s over three CI runs.

The author's Node project now sets testTimeout and hookTimeout to 120 s
in its config, as embed, mcp, the demo and the skills project do, and
the five numbers its tests carried go. The verifier gets the same in a
config of its own, and its sixteen numbers of 120 s or less go: the
three that ask for 240 s stay, and the one that tests a time.

design/testing.md has the rule: a limit stops a hang and does not time
a test, so it is set once for a package, for a slow runner.

Closes #255.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 8 files

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread design/testing.md Resolved
Comment thread packages/author/vitest.config.ts Resolved
@alekswebnet
alekswebnet merged commit 58d7b4f into main Nov 20, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI fails on a slow runner: the author's tests run on vitest's 5 s limit, raised one test at a time

1 participant