Skip to content

ci: replace OpenVMM target cache with sccache - #127

Merged
Pedro Henrique Penna (ppenna) merged 3 commits into
devfrom
ci/clean-target-sccache
Sep 19, 2026
Merged

Pedro Henrique Penna (ppenna) merged 3 commits into
devfrom
ci/clean-target-sccache

Conversation

@ppenna

@ppenna Pedro Henrique Penna (ppenna) commented Sep 19, 2026

Copy link
Copy Markdown
Contributor

Summary

  • replace shared compiled OpenVMM target/ caches with clean Cargo targets and persistent runner-local sccache
  • install pinned, SHA-256-verified sccache 0.18.0 through both Linux and Windows runner setup scripts
  • configure runner services with RUSTC_WRAPPER=sccache, CARGO_INCREMENTAL=0, and a persistent 10-GiB cache
  • report per-job sccache statistics for OpenVMM binary and test jobs

Root cause

Windows runs 35418573985 and 35420849296 intermittently failed while relinking the cached proc-macro DLL windows_interface-ec5c95fb373e8bb3.dll. Investigation on azure-windows-2 reproduced the exact LNK1104 when the cached DLL was held during a path-invalidated rebuild. The shared target cache was created under the former runner workspace and Cargo-home roots, then restored under the new roots, forcing a broad rebuild over loadable cached DLLs.

A Defender exclusion would only narrow the timing window. Clean target directories remove the stale DLL completely; sccache retains content-addressed compiler reuse without restoring live linker outputs.

Runner rollout

The updated setup scripts were applied successfully to all 15 registered Azure runners:

  • azure-windows-1 through azure-windows-5
  • azure-kvm-1 through azure-kvm-3
  • VMSS-named KVM runners for instances 4 and 5
  • azure-azlinux-1 through azure-azlinux-5

All runners completed their built-in environment validation and returned online and idle.

Validation

  • complete Windows validate-nvx: 236 NVX tests (1 skipped), 4 host-profile tests
  • Ruff lint and formatting
  • Pyright for Linux and Windows
  • ShellCheck and shfmt for setup-linux-runner.sh
  • PowerShell syntax validation for setup-windows-whp.ps1
  • pinned Windows sccache binary exercised through idempotent connect/start, stats, and stop operations
  • Linux and Windows canary provisioning followed by full 15-runner rollout
  • exact-head CI run 35431797238, attempt 1: success
  • final sccache hit rates: KVM 99.64%, MSHV 7.28%, Windows 6.65%; zero cache write errors
  • Copilot follow-up review: approval recommended, no findings

Copilot AI balanced review requested due to automatic review settings September 19, 2026 07:30

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The sccache lifecycle can fail jobs after cancellation or extended idle periods.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1 High severity

Open (1)
What changed in this PR

Replaces shared OpenVMM build caches with persistent runner-local sccache to avoid stale linker artifacts.

Changes:

  • Installs pinned, checksum-verified sccache on Linux and Windows runners.
  • Integrates cache lifecycle and statistics into OpenVMM CI jobs.
  • Adds runner validation, documentation, and regression checks.
File Description
scripts/​test_nvx_tools.py Tests sccache configuration and cache removal.
scripts/​setup/​setup-windows-whp.ps1 Provisions and configures sccache on Windows.
scripts/​setup/​setup-linux-runner.sh Provisions and configures sccache on Linux.
scripts/​setup/​README.md Documents the new caching model.
.github/​workflows/​ci.yml Replaces target caches with sccache actions.
.github/​actions/​validate-runner/​action.yml Validates runner sccache configuration.
.github/​actions/​sccache/​action.yml Manages per-job sccache statistics and lifecycle.

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread .github/actions/sccache/action.yml Outdated
Copilot AI review requested due to automatic review settings September 19, 2026 07:50

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

It changes privileged fleet-wide runner provisioning and has an unresolved Linux setup-path failure.

Review effort: Balanced
Findings: 1 High severity

Open (1)
Previously missed (1)

In code that hasn't changed since last review

Medium severity Guard writable check when runner is not configured

scripts/​setup/​setup-linux-runner.sh:640

Fresh-host provisioning without --runner-name still leaves configure_runner=false. That path skips install_runner and protect_runner_installation, so _work/_sccache is not created; this unconditional writable check then fails an otherwise supported setup at the end. Guard the check with the existing .runner marker, as the runner-service validation below already does.

Copilot AI review requested due to automatic review settings September 19, 2026 08:21
@ppenna

Copy link
Copy Markdown
Contributor Author

Also addressed the fresh-host setup concern in 6a64cc9: the Linux _work/_sccache writable check now runs only when the .runner marker exists, matching the surrounding service validation boundary.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The cache migration is consistent across provisioning, validation, CI, documentation, and tests.

Review effort: Balanced
Findings: None

Resolved since last review (1)

Install pinned, SHA-256-verified sccache 0.18.0 binaries during Linux
and Windows runner setup. Configure runner services with a protected,
persistent 10 GiB cache under _work/_sccache, use sccache as
RUSTC_WRAPPER, and disable Cargo incremental compilation.

Validate the installed version, service environment, cache-directory
permissions, and Windows ACLs. Document the new cache model and extend
CI configuration tests to enforce sccache lifecycle integration and
removal of cached OpenVMM target trees.
Extend the cross-platform runner validation action to require sccache
and verify that jobs inherit the expected compiler-cache environment.

Confirm that RUSTC_WRAPPER selects sccache, Cargo incremental
compilation is disabled, the cache size is limited to 10 GiB, and
SCCACHE_DIR references an existing directory. On Linux, also require the
cache directory to be writable so provisioning errors fail before build
jobs begin.
Add a cross-platform composite action to manage the runner-local sccache
lifecycle. Validate requested operations, initialize the cache with
per-job statistics, keep the server active for the job, and always
report statistics before stopping it.

Wrap OpenVMM build and test jobs with the new action and remove GitHub
Actions restore/save steps for compiled target directories. Retain
shared Cargo and OpenVMM input caches while avoiding large,
platform-specific target archives.

Expand CI path detection to include the sccache action and runner setup
scripts so cache infrastructure changes trigger the appropriate OpenVMM
tests.
@ppenna
Pedro Henrique Penna (ppenna) merged commit bdce9d6 into dev Sep 19, 2026
19 checks passed
@ppenna
Pedro Henrique Penna (ppenna) deleted the ci/clean-target-sccache branch September 19, 2026 14:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants