Benchmark CPU seconds

[Dreamsorcerer]

We have a need to benchmark the total load for one of our tests, rather than the total time it takes. It would be good if there was an option in codspeed to set a benchmark to measure CPU-seconds instead.

[art049]

Hi, in instrumentation mode (what aiohttp runs in CI), we already measure something much closer to CPU-seconds than to wall time. The benchmark runs under a CPU simulation (Valgrind based), and the reported time is computed from the instructions executed and the memory accesses of the process, including work done on other threads. Time spent blocked on I/O, epoll waits or sleeps costs essentially nothing. So a benchmark whose wall time is dominated by waiting will report only the actual compute it did.

Walltime mode is the one measuring real elapsed time, and there the clock is currently hard-coded to perf_counter. Adding a CPU clock option there (something like time.process_time, which also aggregates across threads) is doable if that's what you need.

Can you share the test you have in mind and what "total load" means for it? If the goal is catching regressions in CPU consumed while the test waits on I/O, instrumentation mode should already give you that.

[Dreamsorcerer]

Yeah, this test is for https://github.com/dimensionalOS/dimos
We basically want to run a full blueprint (so basically running the entire application in simulation mode). Have been trying to figure out a good way to benchmark with codspeed, but seems there are a number of issues which limit this particular application.

I think we could probably use simulation mode if #131 was resolved. It would also be ideal if we could benchmark a number of other things like peak memory use and threads. Currently our best solution so far is dimensionalOS/dimos#3524 (the measurements are in https://github.com/dimensionalOS/dimos/actions/runs/32504870223/job/96843636673?pr=3524).