FlintCoverage

Measures how much of vanilla Minecraft's code the FlintBenchmark tests actually run.

It starts a vanilla server with the JaCoCo agent attached, runs the whole benchmark against it with FlintCLI, and builds an HTML coverage report. The report shows each class as decompiled source, with covered, partly covered and missed lines and branches marked.

Caution

Don't share reports anywhere. A report contains decompiled Minecraft source code, and Mojang's EULA does not allow redistributing it. Don't commit reports, upload them as CI artifacts, host them (GitHub Pages or anything else), or post them in chats. Everyone generates their own copy locally. Sharing the numbers (percentages, class names, line counts) is fine.

Requirements

• Linux (the script uses mkfifo, nproc and sha1sum)
• Java 25
• git, curl, jq, unzip, zip, python3
• FlintCLI built in release mode: cargo build --release in a FlintCLI checkout next to this repo (../FlintCLI), or point FLINTMC at the binary
• Port 25599 free (or set PORT)

Usage

./coverage.sh                    # FlintBenchmark master
BENCHMARK_REF=my-branch ./coverage.sh
./coverage.sh path/to/tests      # a local tests folder

Variable	Default
FLINTMC	../FlintCLI/target/release/flintmc	flintmc binary
BENCHMARK_REF	master	FlintBenchmark branch, tag or commit
MC_VERSION	26.2	Vanilla version. Must match the protocol FlintCLI speaks
PORT	25599	Server port

The first run downloads the server, JaCoCo, Vineflower and FlintBenchmark into cache/ and takes about 4 minutes. Later runs take about 3.

The report ends up in out/html/index.html, with out/coverage.csv and out/coverage.xml next to it. Each run replaces out/, so copy the folder if you want to keep an older report. The script exits with flintmc's status, so failing tests make it fail too.

Reading the report

• Read branch coverage. Every if, switch case and &&/|| outcome counts. Line coverage is inflated by server startup, which runs every block's constructor and registry setup, so even untested mobs show a few percent of lines.
• Covered means the code ran, not that a test checked it. Placing a block next to the one under test runs the neighbour's update logic whether or not anything asserts on it. Treat the numbers as an upper bound on what the suite can catch. Mutation testing is the way to measure whether the assertions catch changes.
• Only these packages are measured, including subpackages: world.level.block, world.level.material, world.level.redstone, world.level.portal, world.item, world.entity. Edit PACKAGES in coverage.sh to change that. Logic that lives elsewhere (for example nether portals in world.level.portal) needs its package added or it won't show up.
• Line numbers refer to the decompiled source, not Mojang's original files.
• Two runs of the same tests differ by about 15 branches, mostly item-entity timing.
• Tests place blocks with commands, so placement logic like getStateForPlacement shows as missed unless a test places blocks the way a player does.

How it works

1. Unpacks the server bundle and decompiles the in-scope classes with Vineflower, which also exports a mapping from original to decompiled line numbers (cached per version and package list).
2. LineRemap.java rewrites each class's line numbers to match the decompiled source. Only debug info changes, never behaviour.
3. Starts a superflat server on the rewritten classes with the JaCoCo agent, ops the bot and turns off natural mob spawning so mob AI doesn't count as covered.
4. Runs flintmc against it, stops the server so the coverage data is written, and generates the report.

The server has to run on the rewritten classes. JaCoCo uses line boundaries to place its probes, so changing line numbers after the run would make the coverage data not match.