Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
98 changes: 98 additions & 0 deletions .github/workflows/bench.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
name: Bench

# Runs only when a maintainer adds the `bench` label to a PR. Remove and re-add
# the label to run it again after new commits.
on:
pull_request:
types: [labeled]

permissions:
contents: read

# The label name is in the group so adding some other label to the PR can't
# cancel a running bench.
concurrency:
group: bench-${{ github.event.pull_request.number }}-${{ github.event.label.name }}
cancel-in-progress: true
Comment on lines +14 to +16

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Other labels stop benchmarks

Adding an unrelated label while a benchmark runs can cancel it. Every labeled event shares this workflow-level concurrency group. Cancellation happens before the job-level bench check skips the new run, leaving the PR without benchmark results.

Move concurrency into the gated job, or include the label in the group.


jobs:
bench:
name: Output and memory check
if: github.event.label.name == 'bench'
runs-on: blacksmith-6vcpu-macos-latest
timeout-minutes: 45
steps:
- name: Checkout
uses: actions/checkout@v4
with:
fetch-depth: 0
persist-credentials: false

- name: Install Rust
uses: dtolnay/rust-toolchain@stable

- name: Install Bun
uses: oven-sh/setup-bun@v2

- name: Check out the base branch
run: |
echo "BASE_DIR=$RUNNER_TEMP/wrec-base" >> "$GITHUB_ENV"
git worktree add --detach "$RUNNER_TEMP/wrec-base" "${{ github.event.pull_request.base.sha }}"

- name: Cache Cargo
uses: actions/cache@v4
with:
path: |
~/.cargo/registry
~/.cargo/git
target
${{ env.BASE_DIR }}/target
key: ${{ runner.os }}-cargo-bench-${{ hashFiles('Cargo.lock') }}
restore-keys: |
${{ runner.os }}-cargo-bench-

- name: Build base
working-directory: ${{ env.BASE_DIR }}
run: cargo build --release -p cli -p daemon

- name: Build PR
run: cargo build --release -p cli -p daemon

# The VM has SIP off, so TCC.db is writable. macOS checks the runner
# process that launched the recorder, so the runner itself needs the grant.
- name: Grant Screen Recording
run: |
now=$(date +%s)
for c in \
"$PWD/target/release/wrec" "$PWD/target/release/daemon" \
"$BASE_DIR/target/release/wrec" "$BASE_DIR/target/release/daemon" \
$(find "$PWD/target/release/build" "$BASE_DIR/target/release/build" -name capture-engine -type f) \
"$(command -v bun)" "$HOME/bin/Runner.Listener" "$HOME/bin/Runner.Worker"; do
for db in "$HOME/Library/Application Support/com.apple.TCC/TCC.db" "/Library/Application Support/com.apple.TCC/TCC.db"; do
sudo sqlite3 "$db" "insert or replace into access (service,client,client_type,auth_value,auth_reason,auth_version,indirect_object_identifier,flags,last_modified) values ('kTCCServiceScreenCapture','$c',1,2,4,1,'UNUSED',0,$now);"
done
done
sudo killall tccd || true
sleep 2

- name: Bench PR against base
run: |
set -o pipefail
bun run bench ci --wrec target/release/wrec --against "$BASE_DIR/target/release/wrec" 2>&1 | tee /tmp/bench.log

- name: Summary
if: always()
run: |
{
echo '```'
sed -n '/^[a-z0-9-]*: \(pass\|fail\|inconclusive\)$/,$p' /tmp/bench.log
echo '```'
} >> "$GITHUB_STEP_SUMMARY"

- name: Upload results
if: always()
uses: actions/upload-artifact@v4
with:
name: bench-${{ github.event.pull_request.number }}
path: benchmarks/runs/
if-no-files-found: ignore
16 changes: 16 additions & 0 deletions benchmarks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,22 @@ cd docs && bun run dev # then open localhost:4321/benchmarks
The smoke suite runs one balanced HEVC recording against the stimulus window and
does not apply release gates. It is meant for local sanity checks and CI plumbing.

## PR Check

Add the `bench` label to a PR to run `.github/workflows/bench.yml`. It builds
the PR and its base branch on a Blacksmith macOS runner and runs:

```sh
bun run bench ci --wrec target/release/wrec --against /path/to/base/wrec
```

The `ci` suite records balanced 1080p HEVC and H.264 for 5 s, three times per
build. It fails if a recording breaks (exit code, decode, codec, dimensions,
timestamp order) or if peak memory grows more than 2% and 1 MB over the base.
It skips fps, CPU, and latency gates, because the runner is a VM and those
numbers only measure the VM. To run it again after new commits, remove the
label and add it back.

## Release Gate

Before creating a version tag, run the local release check from a clean checkout
Expand Down
5 changes: 5 additions & 0 deletions benchmarks/src/benchmark-values.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,11 @@ describe("benchmarkExitCode", () => {
expect(benchmarkExitCode("release", "inconclusive")).toBe(1);
});

test("fails the PR check when it does not pass", () => {
expect(benchmarkExitCode("ci", "pass")).toBe(0);
expect(benchmarkExitCode("ci", "fail")).toBe(1);
});

test("keeps smoke runs informational", () => {
expect(benchmarkExitCode("smoke", "pass")).toBe(0);
expect(benchmarkExitCode("smoke", "fail")).toBe(0);
Expand Down
2 changes: 1 addition & 1 deletion benchmarks/src/benchmark-values.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,4 @@ export const nullableNumber = (value: unknown) => {
};

export const benchmarkExitCode = (suite: SuiteName, status: OverallStatus) =>
suite === "release" && status !== "pass" ? 1 : 0;
suite !== "smoke" && status !== "pass" ? 1 : 0;
70 changes: 70 additions & 0 deletions benchmarks/src/gates.test.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import { describe, expect, test } from "bun:test";
import {
evaluateCiGates,
evaluateProfileGates,
type ProfileSpec,
type RunResult,
Expand Down Expand Up @@ -160,3 +161,72 @@ describe("release gate hardening", () => {
expect(status("regression_observed_fps")).toBe("pass");
});
});

describe("ci gates", () => {
const mb = 1024 * 1024;
const withRss = (
rep: number,
variant: "candidate" | "reference",
rssBytes: number,
) => {
const base = run(rep, 500, 100);
return {
...base,
id: `${variant}-${rep}`,
variant,
processSummary: { ...base.processSummary, maxTotalRssBytes: rssBytes },
} satisfies RunResult;
};
const reference = [1, 2, 3].map((rep) => withRss(rep, "reference", 46 * mb));
const status = (gates: ReturnType<typeof evaluateCiGates>, name: string) =>
gates.find((gate) => gate.name === name)?.status;

test("passes memory within VM noise and fails a real increase", () => {
const same = [1, 2, 3].map((rep) => withRss(rep, "candidate", 46.1 * mb));
const grown = [1, 2, 3].map((rep) => withRss(rep, "candidate", 48 * mb));
const dims = { width: 1920, height: 1080 };

expect(
status(
evaluateCiGates(profile, dims, same, reference),
"regression_max_rss",
),
).toBe("pass");
expect(
status(
evaluateCiGates(profile, dims, grown, reference),
"regression_max_rss",
),
).toBe("fail");
});

test("fails a recording with the wrong codec", () => {
const h264 = [1, 2, 3].map((rep) => {
const base = withRss(rep, "candidate", 46 * mb);
return { ...base, decode: { ...base.decode!, codec: "h264" } };
});
const gates = evaluateCiGates(
profile,
{ width: 1920, height: 1080 },
h264,
reference,
);

expect(status(gates, "codec_match")).toBe("fail");
});

test("fails a recording with no frames", () => {
const empty = [1, 2, 3].map((rep) => {
const base = withRss(rep, "candidate", 46 * mb);
return { ...base, decode: { ...base.decode!, frames: [] } };
});
const gates = evaluateCiGates(
profile,
{ width: 1920, height: 1080 },
empty,
reference,
);

expect(status(gates, "frames_decoded")).toBe("fail");
});
});
103 changes: 76 additions & 27 deletions benchmarks/src/gates.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
export type SuiteName = "smoke" | "release";
export type SuiteName = "smoke" | "release" | "ci";
export type VariantName = "candidate" | "reference";
export type Quality = "efficient" | "balanced" | "high";
export type Codec = "hevc" | "h264";
Expand Down Expand Up @@ -223,10 +223,22 @@ export const releaseProfiles: ProfileSpec[] = [
},
];

// The PR check runs on a CI VM, where capture fps and CPU only measure the VM.
// It keeps the gates a VM can't distort: output correctness and memory.
export const ciProfiles: ProfileSpec[] = releaseProfiles.filter(
(profile) =>
profile.name === "balanced-1080p30-hevc" ||
profile.name === "balanced-1080p30-h264",
);

const cpuNoiseFloor = 3;
const rssNoiseFloor = 20 * 1024 * 1024;
const startLatencyNoiseFloor = 500;
const finalizeLatencyNoiseFloor = 150;
// Max RSS moved under 0.25% between two copies of one binary on the CI VM,
// so the PR check can flag a memory change far smaller than the release gate.
const ciRssNoiseFloor = 1024 * 1024;
const ciRssPercentTolerance = 2;

export const expectedOutputDimensions = (
native: Dimensions,
Expand Down Expand Up @@ -442,32 +454,7 @@ export const evaluateProfileGates = (
),
}),
);
gates.push({
name: "pts_monotonic",
threshold: "true",
measured: candidate.observed.ptsMonotonic,
status: candidate.observed.ptsMonotonic ? "pass" : "fail",
});
gates.push({
name: "codec_match",
threshold: profile.codec,
measured: candidate.observed.codec,
status: candidateRuns.every((run) => run.decode?.codec === profile.codec)
? "pass"
: "fail",
});
gates.push({
name: "dimensions_match_request",
threshold: `${expectedDimensions.width}x${expectedDimensions.height}`,
measured: candidate.observed.dimensions
? `${candidate.observed.dimensions.width}x${candidate.observed.dimensions.height}`
: null,
status: candidateRuns.every((run) =>
dimensionsEqual(run.decode?.dimensions, expectedDimensions),
)
? "pass"
: "fail",
});
gates.push(...outputGates(profile, expectedDimensions, candidateRuns));
gates.push(
numericBudgetGate({
name: "self_report_disagreement",
Expand Down Expand Up @@ -528,6 +515,68 @@ export const evaluateProfileGates = (
);
};

export const evaluateCiGates = (
profile: ProfileSpec,
expectedDimensions: Dimensions,
candidateRuns: RunResult[],
referenceRuns?: RunResult[],
): Gate[] => [
...outputGates(profile, expectedDimensions, candidateRuns),
{
name: "frames_decoded",
threshold: "> 0 in every run",
measured: summarizeRuns(candidateRuns).observed.decodedFrames,
status: candidateRuns.every((run) => (run.decode?.frames.length ?? 0) > 0)
? "pass"
: "fail",
},
regressionGate(
Comment on lines +523 to +533

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Empty recordings pass

evaluateCiGates can pass a movie with zero decoded frames. decode.swift can return an empty frames array with the track's codec and dimensions, including when reading stops before yielding a frame. deriveObserved treats the empty timestamp list as ordered and reports no stimulus rate, so retries do not reject it. The CI gates then pass if memory is unchanged, even though no frame decoded.

Require at least one decoded frame and make the decoder reject a failed read.

"regression_max_rss",
"bytes",
candidateRuns,
referenceRuns,
(run) => run.processSummary.maxTotalRssBytes,
ciRssNoiseFloor,
ciRssPercentTolerance,
),
];

const outputGates = (
profile: ProfileSpec,
expectedDimensions: Dimensions,
candidateRuns: RunResult[],
): Gate[] => {
const candidate = summarizeRuns(candidateRuns);
return [
{
name: "pts_monotonic",
threshold: "true",
measured: candidate.observed.ptsMonotonic,
status: candidate.observed.ptsMonotonic ? "pass" : "fail",
},
{
name: "codec_match",
threshold: profile.codec,
measured: candidate.observed.codec,
status: candidateRuns.every((run) => run.decode?.codec === profile.codec)
? "pass"
: "fail",
},
{
name: "dimensions_match_request",
threshold: `${expectedDimensions.width}x${expectedDimensions.height}`,
measured: candidate.observed.dimensions
? `${candidate.observed.dimensions.width}x${candidate.observed.dimensions.height}`
: null,
status: candidateRuns.every((run) =>
dimensionsEqual(run.decode?.dimensions, expectedDimensions),
)
? "pass"
: "fail",
},
];
};

export const statusFromGates = (
gates: Gate[],
runs: RunResult[],
Expand Down
Loading
Loading