Benchmarking build size and compilation speed

[ok-nick]

Hey, if it's in scope of this project, it'd be amazing if we can integrate a way to measure and track build size and compilation speed into the interface.

[GuillaumeLagrange]

Hello! Thank you for your feedback, we have been asked for similar features already, they definitely are very relevant.

Compilation speed: we are currently working on support for benchmarking arbitrary commands, not just benchmarks written in the context of benchmarking frameworks, this could be a very good use case for it.

Build size: we view this as arbitrary custom metrics associated to runs, which we do plan to support, but we're not sure yet how to best incorporate it nor do we have concrete visibility on when we will, so stay tuned!

[ok-nick]

@GuillaumeLagrange Is it possible to benchmark compilation speed now with the new CodSpeed CLI?

I gave it a shot using this GitHub Actions step:

- name: Run compile time benchmarks on debug build
  uses: CodSpeedHQ/action@v4
  with:
    mode: simulation
    run: codspeed exec --warmup-time 0 --max-rounds 1 -- cargo build -p c2pa --no-default-features --features ${{needs.get-features.outputs.rust-native-features}}
    token: ${{ secrets.CODSPEED_TOKEN }}

and received this error:

Running the benchmarks
  Installing packages: /tmp/valgrind-codspeed.deb
  Error: Sudo is not available to run the command: apt-get update
  Error: failed to execute the benchmark process, exit code: 1
  Error: Process completed with exit code 1.

If this is a bug let me know and I can open up a separate issue.

[GuillaumeLagrange]

@ok-nick

We are close to being able to do so, we currently have one main limitation: simulation mode does not support subprocess spawning, and any cargo command is extremely likely to spawn dozens of rustc processes.

Until we do support this, you can try the walltime instruments, although we do not yet have hooks to run before/after each iteration so you would have to include the clean in the benched command for now unfortunately.

At the root of your repo, or wherever you want, you should add a codspeed.yml file

# Tweak these accordingly, or remove them to use the defaults.
options:
  warmup-time: 500ms
  max-time: 5s
  max-rounds: 10

benchmarks:
  - name: "Compilation test"
    exec: sh -c 'cargo clean && cargo build -p c2pa {...}'

Then your CI workflow becomes

- name: Run compile time benchmarks on debug build
  uses: CodSpeedHQ/action@v4
  with:
    mode: walltime
    # Omit the `run` input to make CodSpeed use commands from your `codspeed.yaml config file
    token: ${{ secrets.CODSPEED_TOKEN }} # Note: you can get rid of this and use OIDC auth if you wish: https://codspeed.io/docs/integrations/ci/github-actions/configuration#oidc-recommended

If you're on a Ubuntu machine, you can try it out outside of the CI action by following these steps: https://codspeed.io/docs/cli#running-benchmarks

If you try this in simulation mode, you will get an error saying we do not support measuring processes spawning subprocesses yet. I'll make sure to let you know once we do.
Similarly, I'll let you know once we've implemented before/after each iteration hooks to get rid of the cargo clean invocation in the benched command.

[ok-nick]

@GuillaumeLagrange Thank you for the info and updates! I appreciate the transparency.