e2e: publish the results of a run packed, and serve them - #795
Merged
Merged
Conversation
klihub
added this pull request to stack #796
September 16, 2026 11:44
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
klihub
force-pushed
the
e2e-results-server
branch
from
September 16, 2026 13:22
563a380 to
561fb20
Compare
klihub
changed the title
klihub
force-pushed
the
e2e-results-server
branch
2 times, most recently
from
September 16, 2026 14:09
e3c79c3 to
00280af
Compare
klihub
force-pushed
the
e2e-results-server
branch
from
September 17, 2026 11:41
00280af to
3f6fc7f
Compare
klihub
force-pushed
the
e2e-results-server
branch
2 times, most recently
from
September 17, 2026 12:09
01acf84 to
580b75c
Compare
klihub
force-pushed
the
e2e-results-server
branch
from
September 17, 2026 12:46
580b75c to
d708577
Compare
Move our e2e-runner and the result serving caddy file to scripts/testing/nightly. Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Krisztian Litkey <krisztian.litkey@intel.com>
klihub
force-pushed
the
e2e-results-server
branch
2 times, most recently
from
September 18, 2026 07:36
5f08b3b to
5e3e3d9
Compare
Pack everything to a single results.tar.zst. Remove the packed artifacts, leaving behind the report, results.json, status.txt and summary.txt. This results in significant space savings, eg. 117M test artifacts get packed to cc. 2.3M. This allows keeping test results around easily for months before cleaning them up. Add a specialized server to e2e-report for serving results over HTTP. Our server knows about and understands packed artifacts, and knows how to serve individual pieces of content from them. We serve nothing outside the result root directory. We serve nothing but Nothing but regular files of a packed archive. We serve requests so that nothing gets cached without revalidation. 'make e2e-report' builds e2e-report to build/bin. Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Krisztian Litkey <krisztian.litkey@intel.com>
Add --pack-results to pack up the results once the run is reported on. It also keeps the artifacts of every test and the coverage data of each as well unless told otherwise. Off by default. Browsing a packed needs 'e2e-report serve'. Document setting the nightly up from cron and serving what it publishes, with a systemd unit for the server. Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Krisztian Litkey <krisztian.litkey@intel.com>
klihub
force-pushed
the
e2e-results-server
branch
from
September 18, 2026 10:21
5e3e3d9 to
3004564
Compare
Add --live-index to the serve subcommand. With it the index of the runs is built from the runs found under the result root for every request, instead of being read from the index.html there. This serves a root nothing has indexed, and one whose runs come and go without anyone running 'e2e-report index' afterwards. A row names the directory it links to, so a link works even for a run whose report calls the run something else. A run with no report of its own yet, which is what one still going looks like, is listed too. Its row links to the directory of the run instead of to a report which is not there, so the link works and lists what the run has collected so far. We build the index reading only. reportIndex reports on a run which has none, and a server has no business doing that: the unit hands it the results read-only, and reporting would overwrite the status of a run still going. A packed run is indexed exactly like an unpacked one, as both keep results.json and index.html outside the archive. Nothing about a row depends on whether a run is packed. Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Krisztian Litkey <krisztian.litkey@intel.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Publish the results of a run packed, and serve them.
A run of the whole suite collects some 120M, most of it the command transcripts
and the plugin logs of each test, and most of that the same lines over and over.
In one solid archive it takes 2.3M, so nothing has to be thrown away to keep the
results of months of nightly runs.
e2e-report packwrites what a run collected toresults.tar.zstand removeswhat it packed. The report,
results.json,status.txtandsummary.txtstaywhere they are, so how a run went is readable without unpacking anything.
e2e-report serveserves a result root over HTTP, the packed runs as if theirarchives had been extracted where they are: a report links to the same paths
whether its run is packed or not, and directory browsing works either way.
e2e-runner --pack-resultspublishes a run this way, and then keeps theartifacts of every test and the coverage data of each as well, since with the
results packed there is nothing to gain by pruning them.
into with one, so the artifacts a test packed up can be read a file at a time
instead of being downloaded. That works for the archive of a run, for the
artifacts of a test, and for the latter inside the former. The report links to
the artifacts of a test both ways, the tarball first, as a plain file server
can only ever hand it over.
Packing is off by default: a packed run needs the server to browse.
tar --zstd -xfgets the results out without one.e2e-runnerand the caddy configuration move toscripts/testing/nightly, whichalso gets a README of how to set the nightly up from cron and how to serve what
it publishes, and a systemd unit for the server.
Measured on a published nightly run: 17M as published today (pruned, with the
artifacts of each test in a tar.xz) packs to 2.8M, and the same run with nothing
pruned at all, 117M and 4693 files, packs to 2.3M. Reading a file from an
archive takes 20-300ms, decompressing it up to that file; only the listing of
each archive is kept.
Standard library plus
klauspost/compressfor zstd, which was already in themodule graph. Reading a
.tar.xzshells out toxz, rather than taking adependency for the sake of the tarballs of tests published so far; without it
such a tarball is still served, it just cannot be served into.
Tests cover packing, listing, serving packed and unpacked runs, browsing into
.tar.xz,.tar.gzand.tarboth on their own and nested in a run archive,content types, and that neither a path leading out of the result root nor a
doctored archive gets anything else served.