test: Add the evmone test command - #1685
Open
chfast wants to merge 11 commits into
Open
Conversation
Contributor
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #1685 +/- ##
========================================
Coverage 97.98% 97.99%
========================================
Files 181 182 +1
Lines 16199 16307 +108
Branches 3724 3747 +23
========================================
+ Hits 15873 15980 +107
Misses 248 248
- Partials 78 79 +1
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Pull request overview
Adds a unified evmone test command for state and blockchain fixtures.
Changes:
- Adds lazy fixture collection, classification, filtering, and execution.
- Exposes per-case state and blockchain loaders.
- Updates empty/skipped-run exit semantics and integration coverage.
Reviewed changes
Copilot reviewed 16 out of 16 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
tools/evmone/main.cpp |
Adds the test subcommand. |
test/utils/test_driver.hpp |
Defines the nothing-verified exit code. |
test/utils/test_driver.cpp |
Handles fully skipped runs. |
test/utils/test_collector.hpp |
Declares collection settings and API. |
test/utils/test_collector.cpp |
Classifies and runs mixed fixtures. |
test/utils/statetest.hpp |
Exposes single-case loading. |
test/utils/statetest_runner.cpp |
Documents empty-post behavior. |
test/utils/statetest_loader.cpp |
Implements single-case loading. |
test/utils/CMakeLists.txt |
Builds the collector. |
test/utils/blockchaintest.hpp |
Exposes single-case loading. |
test/utils/blockchaintest_loader.cpp |
Publishes the case loader. |
test/unittests/test_driver_test.cpp |
Tests skipped-run semantics. |
test/integration/testcmd/one_unsupported_case.json |
Covers mixed supported and unsupported cases. |
test/integration/testcmd_fault/unrecognised_case.json |
Covers malformed fixture cases. |
test/integration/testcmd_cases/case_after_exception.json |
Covers continuation after exceptions. |
test/integration/CMakeLists.txt |
Adds command-level integration tests. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+147
to
+149
| // A fixture file in which this tool ran nothing it was asked for is skipped, not passed. | ||
| if (!any_ran && !declined.empty()) | ||
| throw UnsupportedTestFeature{declined.front().second}; |
Comment on lines
+180
to
+181
| cmd.add_flag("--trace-summary", opts.settings.trace_summary, | ||
| "Report each test's execution summary; --trace reports it as well."); |
Comment on lines
+42
to
+44
| auto format = info->find("fixture-format"); | ||
| if (format == info->end()) | ||
| format = info->find("fixture_format"); // The spelling EEST also accepts. |
chfast
force-pushed
the
test/evmone-test-cmd
branch
4 times, most recently
from
August 31, 2026 20:52
2cfb252 to
3460a62
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.
Suppressed comments (1)
test/utils/test_collector.cpp:152
- When
-kmatches no cases in a directory, every collected file returns normally and is counted as passed, so the command exits successfully despite executing no fixture. This also makes filtering behavior depend on whether the same fixture was named as a file (which correctly returnsNOTHING_VERIFIED). Mark a file with no runnable selected cases as skipped so an empty directory selection propagatesNOTHING_VERIFIED.
// A fixture file in which this tool ran nothing it was asked for is skipped, not passed.
// TODO: A file whose cases -k all deselected still passes, as it did before this command
// existed, so a filter which matches nothing reports a tree of passing tests. Skip it
// instead, and an empty selection reaches NOTHING_VERIFIED on its own.
if (!any_ran && !declined.empty())
throw UnsupportedTestFeature{declined.front().second};
chfast
force-pushed
the
test/evmone-test-cmd
branch
4 times, most recently
from
September 1, 2026 08:22
0080993 to
cdc9e3d
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 14 out of 14 changed files in this pull request and generated 1 comment.
Suppressed comments (1)
test/utils/test_collector.cpp:148
- With a directory root,
-kmatching no case leaves each collected file as passed, so the command exits successfully despite verifying nothing; the same filter on a named file correctly returnsNOTHING_VERIFIED. Treat a fixture file where no selected case ran as skipped so an entirely empty selection produces the non-success exit code.
// A fixture file in which this tool ran nothing it was asked for is skipped, not passed.
// TODO: A file whose cases -k all deselected still passes, as it did before this command
// existed, so a filter which matches nothing reports a tree of passing tests. Skip it
// instead, and an empty selection reaches NOTHING_VERIFIED on its own.
if (!any_ran && !declined.empty())
throw UnsupportedTestFeature{declined.front().second};
Comment on lines
+19
to
+20
| // TODO: A test with an empty "post" checks nothing and still passes, so a gutted fixture is | ||
| // indistinguishable from a working one. Report a test which verified nothing as a failure. |
chfast
force-pushed
the
test/evmone-test-cmd
branch
6 times, most recently
from
September 1, 2026 13:25
e9d827d to
1d8a9ca
Compare
This was referenced Sep 2, 2026
chfast
force-pushed
the
test/evmone-test-cmd
branch
2 times, most recently
from
September 2, 2026 05:51
81b6c18 to
4686ff5
Compare
chfast
force-pushed
the
test/evmone-test-cmd
branch
from
September 2, 2026 06:04
4686ff5 to
894674a
Compare
One command runs both fixture formats, deciding which each test case is when it runs it: "_info.fixture-format" names it, and a case without one is recognised by the shape only that format has. A format the tool does not run is skipped, as is a file holding no test at all, which is what EEST's shared pre-allocation is. Anything else it cannot recognise is a fault in the fixture. The two test tools are untouched and keep working. Claude-Session: https://claude.ai/code/session_016UHPAGwcwXMjqhTLpT31K7
chfast
force-pushed
the
test/evmone-test-cmd
branch
from
September 2, 2026 06:49
894674a to
910136c
Compare
An entry the tool did not recognise was a fault or a skip depending on whether the rest of its file holds fixtures, a bool threaded from the collector through a lambda into run_fixture to feed one branch. Loading the file settles it instead: a file with no fixture in it is skipped whole and nothing in it reaches execution, so an unrecognised entry in a fixture file is simply a fault. Named on the command line, such a file is one skipped test rather than one per entry. Claude-Session: https://claude.ai/code/session_016UHPAGwcwXMjqhTLpT31K7
Three of the command's ctests were named for behaviour they could not observe: deleting the -k filter from a directory run, hardcoding the trace summary off, or taking the last decline reason rather than the first all left the integration suite green. One run with -k and --trace-summary over a directory holding a failing case covers the first two, and a second declining case in all_unsupported names which reason wins. The two fault tests differed only in the filter, so they fold into one. Claude-Session: https://claude.ai/code/session_016UHPAGwcwXMjqhTLpT31K7
A fixture was taken for a blockchain test on a "blocks" key alone, so any JSON sidecar carrying one was run as a test and failed the whole run with "key 'genesisBlockHeader' not found", while the symmetric miss on the state shape was a quiet skip. Each shape now needs the state it starts from and what is applied to it: "pre" with "blocks", or "pre" with "transaction". Neither reads an expectation, so a fixture whose expectations are missing is still run rather than passing as no test at all, and "pre" is in every fixture of every format. classify() returns the format alone, the two ways of not running one among its values, so run_fixture switches over it exhaustively and no return site can label a fixture wrongly. Claude-Session: https://claude.ai/code/session_016UHPAGwcwXMjqhTLpT31K7
items() walks an array by index, so a top-level JSON array was collected as fixtures named "0", "1", ... and run. A document which is not an object holds no fixture and is not a test. An ifstream which never opened reads as EOF, so every I/O failure was reported as a JSON syntax error in a file which has none, a 0-byte file and an unreadable one giving the same message. Check the stream. The help and the collector both still described a named file as one test per case in it, which a file holding no fixture at all has not been since it started being skipped whole. Claude-Session: https://claude.ai/code/session_016UHPAGwcwXMjqhTLpT31K7
Its comment described being skipped one entry at a time, which stopped being how a file with no fixture in it is read, and pinned a file count from one release into a checked-in fixture. Claude-Session: https://claude.ai/code/session_016UHPAGwcwXMjqhTLpT31K7
The per-fixture catch built one string where the two-argument fail() splits the check from its detail, and only the detail is indented line by line, so a what() of several lines broke the report's columns. It also caught nothing but std::exception, where the run loop around it catches everything. Claude-Session: https://claude.ai/code/session_016UHPAGwcwXMjqhTLpT31K7
--trace turned the progress row into a line per test so its output could be told apart, but --histogram wrote its table into the row mid-line. Both write per test, so both decide the same way. Claude-Session: https://claude.ai/code/session_016UHPAGwcwXMjqhTLpT31K7
The tests hold the settings and the VM by reference, which only the caller can honour, and dropping the return value turned a tree which could not be collected from a failure into a silent success. Claude-Session: https://claude.ai/code/session_016UHPAGwcwXMjqhTLpT31K7
Each test captured the fixture it was made from, and the capture copies the subtree it names, so a file's whole document was held once more over again, and a listing paid for every copy to read nothing but the name. Hold the document once and index it when the test runs: peak memory on a 3MB file drops by about a quarter. Claude-Session: https://claude.ai/code/session_016UHPAGwcwXMjqhTLpT31K7
Collected from a directory a whole file is one test, so a file which ran anything reported nothing about the fixtures it declined: two of three were dropped from a run which said "1 passed" and exited 0. The day a fixture stops being one this tool runs, its coverage would go with it and the count would not move. Name each on the error stream, which stays empty on a release whose files are each of one format. Claude-Session: https://claude.ai/code/session_016UHPAGwcwXMjqhTLpT31K7
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
One
evmone testcommand runs both state and blockchain fixtures,deciding which format each test case is when it runs it:
_info.fixture-formatnames it, and a case without one is recognisedby the shape only that format has.
What the tool cannot run is treated three ways, because a fixture
directory holds more than fixtures. A format it does not run, engine
fixtures above all, is skipped. A file holding no test at all is skipped
whole, which is what EEST's shared pre-allocation is: 35715 files in one
release. Anything else it cannot recognise, in a file whose other cases
are tests, is a fault in the fixture and fails.
Collection reads no fixture, so a directory costs a walk.
evmone-statetestandevmone-blockchaintestare untouched and stillpass CI; a later PR moves CI to
evmone testand deletes them.Stacked on #1692 and #1693: only the last commit is this PR.
Verified against them on three fixture releases, verdict for verdict:
2710 state tests, 2827 blockchain tests with 2 skipped, and 8172 / 8612
on
tests@v20.0.1.