Repository navigation
test: a test's time limit is its package's, for the author package and the verifier - #256
Merged
Merged
Conversation
…d the verifier (#255) CI failed on main for the third time in three days on a test of packages/author crossing vitest's 5 s on a slow runner: 5010, 5025 and 5011 ms, another test each time, each after the one before it got a number of its own. Two more ran within reach of it, at 3.6 to 5.0 s and 2.3 to 3.1 s over three CI runs. The author's Node project now sets testTimeout and hookTimeout to 120 s in its config, as embed, mcp, the demo and the skills project do, and the five numbers its tests carried go. The verifier gets the same in a config of its own, and its sixteen numbers of 120 s or less go: the three that ask for 240 s stay, and the one that tests a time. design/testing.md has the rule: a limit stops a hang and does not time a test, so it is set once for a package, for a slow runner. Closes #255.
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 8 files
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
… says both projects' (#255)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #255.
CI on main failed after #254 merged: a test in
packages/authortimed out at 5011 ms on vitest's 5 s default. It is the third such failure in three days (5010, 5025 and 5011 ms), another test each time. After each one, the test that failed got a number of its own, and the next one failed.What changes
packages/author/vitest.config.ts: the Node project setstestTimeoutandhookTimeoutto 120 s, as embed, mcp, the demo and the skills project already do for the whole package.packages/verify/vitest.config.tsis new, with the same limit. Every test there that plays a package starts a browser, and each heavy one carried its own number, so a new one written without a number would have failed on the default.{ timeout: 1000 }on the test whose point is the time.design/testing.mdhas the rule: a limit stops a hang and does not time a test, so it is set once for a package, for a slow runner.Why this and not another number on one test
Over three CI runs, these are the author tests that take longest on a runner:
bundle --allThe last two had no number. The first of them failed this run, and the second is next.
Tested
bundle.test.tsandlayout.test.tswith 30 busy processes on 10 cores, to stand in for a slow runner.Not changed
Summary by cubic
Moves test time limits from individual tests to the package config for
packages/authorandpackages/verify, fixing CI failures from vitest's 5-second default on slow runners.testTimeoutandhookTimeoutto 120 s in both packages'vitest.config.ts, and to 180 s in the author's browser project.design/testing.mdthat a package sets its own limit, and a test names one only to ask for more or where time is what it tests.Written for commit 9b61284. Summary will update on new commits.