Skip to content

wpt: add --diff flag that diffs a run against the latest main-branch results - #671

Open
nicoburns wants to merge 1 commit into
mainfrom
devin/1786312567-wpt-diff-flag
Open

wpt: add --diff flag that diffs a run against the latest main-branch results#671
nicoburns wants to merge 1 commit into
mainfrom
devin/1786312567-wpt-diff-flag

Conversation

@nicoburns

@nicoburns nicoburns commented Aug 9, 2026

Copy link
Copy Markdown
Member

Summary

cargo run -rp wpt <suite> --diff now compares the local run against the latest main branch results, so regressions/improvements are visible locally without going through CI.

The baseline is https://dioxuslabs.github.io/blitz/wptreport.json.zst (same artifact the website consumes). It is cached in wpt/cache/ (gitignored) as the raw .zst body plus the response's Last-Modified value, and every --diff run revalidates with If-Modified-Since, so a 304 skips the download. Download failures fall back to the cache; if there's no cache either, the run still completes and only the diff is skipped.

Diffing uses wptreport::aggregate (bumped wptreport 0.0.5 -> 0.0.13, which added aggregate/merge and WptRunInfo::linux_distro). The baseline is filtered to the suites being run, so a partial run (e.g. css/css-flexbox/alignment) doesn't report every other test as removed. Tests whose subtest counts changed are printed with their before/after status, classified as improved or regressed by pass fraction, followed by a summary:

Diff against main (708eb61323e9dda6d57d6f0a667512bb4dfd0068)
=====================
FAIL(3/5) => PASS(5/5) css/css-flexbox/foo.html
ADD  css/css-flexbox/bar.html
---

   0 tests REGRESSED
   1 tests IMPROVED
   1 tests ADDED
   0 tests REMOVED

Link to Devin session: https://dioxus.staging.devinenterprise.com/sessions/ea75dffa87234844b3902d8f14a202c7
Requested by: @nicoburns

WPT results

No changes in test results compared to main.

Generated by the WPT workflow.

@nicoburns nicoburns self-assigned this Aug 9, 2026
@staging-devin-ai-integration

Copy link
Copy Markdown
Contributor

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR that start with 'DevinAI' or '@devin'.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant