Conversation
klihub
force-pushed
the
devel/e2e-runner/improved-collection-and-reporting
branch
2 times, most recently
from
September 15, 2026 21:19
bf18250 to
6cf8faf
Compare
Add a coverage package which serves the coverage data of a plugin built with go build -cover over the instrumentation HTTP server: the meta-data, a snapshot of the counters, and the ID of the binary tying the two together. The ID is the one the go runtime names the files it writes to $GOCOVERDIR with, dug out of the meta-data header, so that data served over HTTP and data dumped at exit merge as they are. This is how we get the coverage of a plugin which is still running, or which never gets to exit gracefully. Serve it from the resource manager, so from the nri-resource-policy-* plugins, and only with the test APIs enabled. runtime/coverage works in a binary built with -cover and never in a test binary, so the tests build a helper to check that what we serve is what go tool covdata expects. Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Krisztian Litkey <krisztian.litkey@intel.com>
COVER=1 builds the resource-manager-based plugins with coverage instrumentation. Instrument our own packages only, and with atomic counters: the plugins are concurrent, and clearing the counters at runtime does not work without them. Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Krisztian Litkey <krisztian.litkey@intel.com>
Collect the coverage data of the plugins the tests exercise, and report the coverage of each plugin and the total at the end of a run. The tests clear the counters when they launch a plugin and ask it for a dump once done. make e2e-tests builds the plugins with COVER=1 so that a run has something to collect; collecting never fails a test. go tool covdata percent cannot report per plugin, and prints a package which has no statements without a percentage and without a line break, running the next package into it, so calculate the numbers from the profile instead, weighted by statements the way go tool cover does. Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Krisztian Litkey <krisztian.litkey@intel.com>
Print a single pod in vm-wait-pod-regexp, and never one which is
terminating. Otherwise port forwarding got two names while one pod was
terminating and another starting, and kubectl refused:
error: TYPE/NAME and list of ports are required for port-forward
Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Krisztian Litkey <krisztian.litkey@intel.com>
Collect the results of a VM and let anything else be, and summarize every collected VM instead of the ones whose name matches n[0-9]*-c*. The results of a topology like s8c4k never made it into the summary, and as the verdict of a run is a grep for FAIL in it, a run whose only failures were there was published as passing. Rename the runtime logs to .txt like the other logs, so that a browser shows them, and clone the repository into CLONED_REPO rather than into an unset variable. Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Krisztian Litkey <krisztian.litkey@intel.com>
Generate a coverage report of every run and publish it with the results. This has to happen before the worktree goes away: the browsable report embeds the sources of the revision which was tested. Remove the results of a run once they are older than RETENTION_DAYS, 100 by default, and the coverage data of its tests once its report exists: --retention-days <days> 0 keeps every result, whatever its age --retention-keep <count> always keep this many latest results --keep-coverage-data keep the data collected for each test Keep the data merged over a run in either case: the profile and the report can be regenerated from it. Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Krisztian Litkey <krisztian.litkey@intel.com>
Add e2e-report, which writes the report of a run: the failures first, each with the reason it failed and with links to the test log, the log of the plugin, the runtime log, the commands the test ran and the test itself on github at the revision which was tested, then the coverage of each plugin and of everything instrumented, then every test case with its artifacts. A run which never got to a test says so and points at the log of the runner, and a test which never wrote a verdict is told apart from one which failed. The numbers go to results.json as well, and the index of all runs is generated from those. report-coverage.sh takes several directories and where to write the report to, so that the coverage of one plugin can be reported across the topologies it ran on, and writes summary.json for the report of a run to pick up. Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Krisztian Litkey <krisztian.litkey@intel.com>
The command transcripts and the plugin logs are the bulk of what a run publishes, and nobody reads either for a test which passed: pack them into an artifacts.tar.xz per test, and leave those of the tests which failed browsable as they are. Packing beats compressing the files one by one, the transcripts being 3090 tiny files: 824K packed against 15M gzipped, most of which is the block each file takes. --keep-artifacts all|failed|none whose artifacts to keep as they are A test keeps its own log either way, nothing is removed unless it made it into the tarball first, and an unknown value keeps everything. Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Krisztian Litkey <krisztian.litkey@intel.com>
Remove the byte compiled python of the pyexec snippets from the results. Record the verdict of a run in a status.txt and point a latest symlink at the newest run, so that the cronjob can tell a bad run from a good one without reading html or json. The exit status keeps saying whether we did our job, not whether the tested system passed. Give the report dark mode, a badge for each verdict, and a section per VM which folds away when everything in it passed. Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Krisztian Litkey <krisztian.litkey@intel.com>
Take --runtime and --full-build|--minimal-build, and keep deciding by the day of the year when neither is given. Refuse a runtime which is neither containerd nor crio right away, instead of half an hour later. Export the options which default from the environment across the re-exec from the worktree of the revision under test: --retention-days and its like only ever took effect when they were given as variables. Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Krisztian Litkey <krisztian.litkey@intel.com>
The results of the runs are static files, so serving them takes a file server with browsing for the directories which have no index of their own. Take the directory to serve and the port from the environment. Caddy also sniffs the type of a file which has no extension, so the command transcripts of a test case show up in a browser instead of being downloaded. Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Krisztian Litkey <krisztian.litkey@intel.com>
Resolve the path of the worktree with realpath -m, and create the
directory the worktrees go into. realpath needs every component but the
last one to exist and prints nothing at all when one does not, so the
first run on a host asked git to add a worktree of "", which dies of an
internal error:
BUG: builtin/worktree.c:498: How come '' becomes empty after
sanitization?
Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Krisztian Litkey <krisztian.litkey@intel.com>
Prune anything with results in it, and leave the links and whatever else is kept next to them alone. Order the runs by when they ran, told from their name or from the timestamp of their directory: reporting on a run writes into it, so the timestamp alone does not say when it ran. Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Krisztian Litkey <krisztian.litkey@intel.com>
Index anything with results or a log of a run in it, give a run which has no report one of its own, and order them by when they ran, told from the name of a run or from the timestamp of its directory. Link each policy of the coverage table to the source of its plugin at the tested revision, and recognise the ssh spellings of a remote, which is what the nightly is cloned from. Right-align the header of a column whose cells are right-aligned, call the log of run.sh a test log, and leave out a link to an empty file, such as the pyexec output of a test whose last pyexec printed nothing. Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Krisztian Litkey <krisztian.litkey@intel.com>
A test also leaves behind the python snippet of its last pyexec, the state it ran against, what it printed and the cache of the plugin, some 160 kilobytes per test and of interest only when a test failed. Pack those away as well for the tests which passed, which takes a run of the whole suite from 25M to 17M, and link them from the report for the tests which failed. Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Krisztian Litkey <krisztian.litkey@intel.com>
Tee the output of a run when we have a terminal to write to, and keep redirecting it to the log when we have not: from cron there is nobody watching, and printing anything would have the whole log of the run mailed to whoever gets our output. Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Krisztian Litkey <krisztian.litkey@intel.com>
Say RUNNING in the status of a run at the start, take a log written to within the hour as one still going and a quiet one as aborted, and report on a run as soon as its directory is there, so the index has it from the start. Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Krisztian Litkey <krisztian.litkey@intel.com>
klihub
force-pushed
the
devel/e2e-runner/improved-collection-and-reporting
branch
from
September 16, 2026 07:02
6cf8faf to
b73147b
Compare
Collaborator
Author
|
Closing in favor of a stacked set of PRs. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note: This PR is manually stacked on top of (IOW includes) #786. Please review that one first.
Improve our e2e nightly runner and reports.
and links to the logs, the commands and the test source at the tested
revision, then the coverage of each plugin, then every test case. Plus an
index of all runs, and a status.txt and a latest symlink for the cronjob.
the tests which passed: a full run publishes 17M instead of 93M.
--retention-keep, --keep-coverage-data, --keep-artifacts.
summary, so a run whose only failures were there was published as passing;
the first run on a host died of a git internal error.
Exit status semantics are unchanged: non-zero only if the runner itself failed,
never because the tested system failed.