WPT: skip ref tests that depend on JavaScript to render - #708
Open
nicoburns wants to merge 1 commit into
Open
Conversation
Contributor
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Many WPT ref tests require JavaScript to render correctly (e.g.
css/css-grid/grid-definition/grid-auto-repeat-dynamic-001.htmlforces a reflow and then mutates styles from a<script>). Since Blitz doesn't run JS, these were being run and producing bogus PASS/FAIL results. Scanning the WPT repo, ~4100 of ~24700 css/ ref tests contain script, and ~98% of those depend on it; the remaining ~2% only use script for waits we already perform (fonts/load) before taking a screenshot.New
script_detection::uses_nontrivial_script(html) -> boolclassifies a document as script-dependent unless every script in it is trivial:<script src>is trivial only for the known wait helpers/common/reftest-wait.jsand/common/rendering-utils.json*=handler attributes) are trivial only if, after stripping comments, every string literal and identifier is on a "wait then screenshot" allowlist, e.g.:document.fonts.ready.then(takeScreenshot)onload = () => requestAnimationFrame(takeScreenshot)document.documentElement.classList.remove("reftest-wait")setTimeout, worklets, unknown helpers, unclosed<script>tags) → script-dependentprocess_test_filenow SKIPs ref tests where this returns true (mirroring the existing crashtest behaviour), andrender_reference_and_comparealso skips tests whose reference file depends on script.On the default
css/css-flexbox+css/css-gridsuites this newly skips 202 tests (78 previously "passing" coincidentally, 124 failing); pass rate of tests run goes 45.92% → 46.41%, and the script (X) failure bucket drops to 0.Link to Devin session: https://dioxus.staging.devinenterprise.com/sessions/7841ac50226f4927a65d4f39ea83715f
Requested by: @nicoburns
WPT results
0 newly passing, 1748 newly failing (net -1748), 2316 other status changes.
Full diff (4064 changed tests)
Generated by the WPT workflow.