Skip to content

Reduce test wall-clock time without weakening coverage#309

Merged
IlyaasK merged 4 commits into
mainfrom
ilyaas/reduce-test-wallclock
Jul 16, 2026
Merged

Reduce test wall-clock time without weakening coverage#309
IlyaasK merged 4 commits into
mainfrom
ilyaas/reduce-test-wallclock

Conversation

@IlyaasK

@IlyaasK IlyaasK commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

What

Reduce the Hypeman test suite wall-clock time by removing measured lifecycle overhead:

  • cancel the full native compression process group when its context expires
  • use immediate VM shutdown only in test teardown while retaining the normal cleanup path
  • prepare VM-ready test images once per run instead of repeatedly unpacking layers
  • use 512 MiB guests for lifecycle tests that do not exercise memory capacity

Why

Wall-clock profiling showed that the suite was spending substantial time in repeated image preparation, graceful shutdown timeouts, oversized lifecycle guests, and a cancellation path that killed the shell but left its child process holding output pipes open.

These changes address those root causes directly. They do not shard the suite or add a test matrix.

Result

Linux Run tests step:

  • before: 259 seconds
  • after: 190 seconds
  • reduction: 69 seconds, or 26.6 percent

The full Linux job, macOS job, and install checks passed.

Test strength

No tests, assertions, VMM backends, or integration scenarios are removed.

Production DeleteInstance behavior remains unchanged. Immediate shutdown is restricted to test cleanup and terminal test teardown. Dedicated memory and resource-capacity tests retain their larger guest sizes.

The compression cancellation regression test also asserts that cancellation returns within one second, so the original 30-second child-process leak cannot silently return.

Verification


Note

Medium Risk
Production snapshot compression cancellation behavior changes (process-group kill); delete graceful-shutdown skipping is test-only but lives on the shared delete path.

Overview
Cuts integration test wall-clock by attacking repeated setup and teardown overhead, without dropping scenarios or assertions.

Prewarm builds VM-ready images once per CI run: test-prewarm now runs CreateImage + WaitForReady for each mirrored image, and EnsureImageReady prefers HYPEMAN_TEST_PREWARM_DIR so tests seed from that cache instead of rebuilding layers. Linux CI chowns the prewarm directory so the job user can write under /ci/prewarm.

Faster test VM teardown: deleteInstanceWithOptions gains skipGracefulShutdown; deleteTestInstanceNow uses it for cleanups while public DeleteInstance behavior is unchanged.

Smaller default lifecycle guests: many integration tests use lifecycleTestMemorySize (512 MiB) where memory size is not the subject; capacity-focused tests keep larger sizes.

Compression cancel fix (production): native zstd/lz4 helpers run in a process group and get SIGKILL on the whole group when the context is canceled, so Wait does not hang on orphaned children; the regression test now requires cancellation within one second.

Reviewed by Cursor Bugbot for commit de5fe4f. Bugbot is set up for automated code reviews on this repo. Configure here.

@IlyaasK IlyaasK requested a review from sjmiller609 July 16, 2026 13:55
Wait for the native helper child before canceling so the regression always exercises process-group termination. Also apply the two clarity cleanups found during self-review.

@sjmiller609 sjmiller609 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice, thanks

@IlyaasK IlyaasK merged commit 5685f4c into main Jul 16, 2026
11 checks passed
@IlyaasK IlyaasK deleted the ilyaas/reduce-test-wallclock branch July 16, 2026 14:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants