Skip to content

ci: improve Windows benchmark provenance and storage diagnostics #119

Description

Context

Two Windows/WHP CI runs produced bimodal snapshot-generation measurements isolated to mapped_memory_flush:

  • run 35382585623: five fast and five slow samples
  • run 35393916034: two fast and eight slow samples

The benchmark in the latter run executed on the same azure-windows-3 / VMSSVAOAN000002 instance that produced most of the approximately 758 ms baseline history. This rules out runner identity mixing as the primary cause. #118 extends the stability detector so both distributions trigger the existing one-time remeasurement.

Additional observability is needed to identify the host storage condition and preserve evidence when the retry succeeds.

Improvements

  • Preserve every lifecycle measurement attempt instead of overwriting the first unstable acceptance.json.
    • Retain attempt-specific JSON files in the benchmark artifact.
    • Clearly identify which attempt supplied the published CSV.
  • Record safe runner provenance in benchmark metadata and summaries.
    • Runner and machine names
    • CPU model and logical processor count
    • Azure VM size
    • OS disk SKU, size, and caching policy
    • Workspace and temporary-directory volume
  • Capture bounded storage diagnostics around snapshot generation.
    • Disk latency and queue depth
    • Read/write throughput and IOPS
    • Azure managed-disk throttling indicators when available
    • Windows writeback and Defender activity relevant to the runner workspace
  • Correlate the diagnostics with the profiled mapped_memory_flush phase.
  • Evaluate moving benchmark scratch data to an identical dedicated data disk on every Windows runner if telemetry confirms OS-disk contention.
    • Treat this as a performance-platform change.
    • Apply the same storage policy to all pooled Windows runners.
    • Reset the Windows performance baseline after migration.
  • Keep runner identity diagnostic-only unless evidence shows a persistent runner-specific offset that exceeds the topology-wide gate tolerance.

Acceptance criteria

  • A successful retry retains both the rejected and accepted lifecycle artifacts.
  • Artifacts identify the runner and storage configuration without exposing credentials or host inventory.
  • A future mapped_memory_flush split contains enough host telemetry to distinguish disk throttling, scanning, writeback, and unrelated host load.
  • Documentation states when a storage change requires a performance-history reset.

Related: #113, #118.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions