Add reservation benchmark#151
Conversation
7592f8f to
da93859
Compare
ReinierMaas
left a comment
ReinierMaas
left a comment
There was a problem hiding this comment.
I didn't review opsqueue/benches/chunks_select.rs yet, only the top block comment. But I think I already have some good advice that can be helpful.
| source "./.setup_local_venv.sh" | ||
|
|
||
| timeout {{pytest_timeout_seconds}} pytest --color=yes {{TEST_ARGS}} | ||
| pytest --timeout {{pytest_timeout_seconds}} --color=yes {{TEST_ARGS}} |
There was a problem hiding this comment.
I don't think this covers what the previous timeout covered. The previous timeout covers the problem that the GIL isn't ever released by the OpsQueue code. This also only mentions that it is the timeout before dumping the stacks, does it also exit pytest at that point?
XRef: #154 for dumping the transitive xdist-worker stacks via signal handlers on top over killing the process on hanging on the GIL lock.
There was a problem hiding this comment.
We also have in python per test timeouts defined in pyproject.toml if that didn''t trigger it is probably a lock-up:
opsqueue/libs/opsqueue_python/pyproject.toml
Lines 54 to 57 in 01b6b00
| pythonEnv | ||
| ]; | ||
| # Fixes: libstdc++.so.6: cannot open shared object file: No such file or directory | ||
| LD_LIBRARY_PATH = pkgs.lib.makeLibraryPath [ pkgs.stdenv.cc.cc.lib ]; |
There was a problem hiding this comment.
Which dependency needed this? And at what time of it lifecycle build or runtime?
Just enabling the benchmarks probably caused it to show-up for you to add it here. What I am thinking is should we also ensure that our packaged opsqueue server has this library path available.
| """Plot the chunks_select benchmark: one subplot per backlog shape, a line per strategy. | ||
|
|
||
| Reads target/chunks_select_bench.csv (written by the bench) and writes a PNG next to | ||
| this script so it can be committed alongside the code it measures. | ||
|
|
||
| """ |
There was a problem hiding this comment.
The committed file is a SVG. Not a problem but a documentation mismatch.
| /// FOR EACH shape: | ||
| /// FOR EACH amount of chunks: | ||
| /// FOR EACH strategy: | ||
| /// | ||
| /// // Setup | ||
| /// Create fresh temporary SQLite database | ||
| /// Insert chunks | ||
| /// | ||
| /// // MEASURE | ||
| /// FOR _ in (SAMPLES + WARMUP): | ||
| /// Start Timer | ||
| /// Execute SQL Query -> Fetch EXACTLY ONE chunk | ||
| /// Stop Timer | ||
| /// IF (not warmup): save duration | ||
| /// | ||
| /// // REPORT & CLEANUP | ||
| /// Calculate stats | ||
| /// Write result to terminal and CSV | ||
| /// Delete temporary database files |
There was a problem hiding this comment.
If you flip FOR EACH amount of chunks and FOR EACH strategy you can extend chunks/submissions instead of generating an entirely new dataset for each measurement. This can greatly speed-up your test runs.
2 participants
This PR adds a benchmark which can be run with
just bench-chunks-select. It produces the SVG contained in this PR. Output from the Rust bench is written to a CSV (not committed) which is then read by Python to produce a graph. The Rust bench also prints the information to stdout, which looks like:just bench-chunks-select Compiling opsqueue v0.36.0 (/home/jeremy-barisch-rooney/channable/opsqueue/opsqueue) Finished `bench` profile [optimized] target(s) in 17.32s Running benches/chunks_select.rs (target/release/deps/chunks_select-d5aca04d748d2c69) SHAPE STRATEGY SIZE MEDIAN P10 / P90 --------------------------------------------------------------------------------------------------------- Companies: 10 Submissions per company: 1 Chunks per submission: 10 FewSubmissionsManyChunks Random 100 55.1 50.5 / 60.6 FewSubmissionsManyChunks PreferDistinct(company_id, Oldest) 100 107.3 99.0 / 117.5 Companies: 10 Submissions per company: 1 Chunks per submission: 50 FewSubmissionsManyChunks Random 500 52.4 48.9 / 69.4 FewSubmissionsManyChunks PreferDistinct(company_id, Oldest) 500 253.0 184.0 / 272.5 Companies: 10 Submissions per company: 1 Chunks per submission: 100 FewSubmissionsManyChunks Random 1000 53.2 49.3 / 59.1 FewSubmissionsManyChunks PreferDistinct(company_id, Oldest) 1000 281.0 246.5 / 377.9 Companies: 10 Submissions per company: 1 Chunks per submission: 200 FewSubmissionsManyChunks Random 2000 54.5 49.2 / 62.3 FewSubmissionsManyChunks PreferDistinct(company_id, Oldest) 2000 480.2 424.4 / 721.8 Companies: 10 Submissions per company: 1 Chunks per submission: 400 FewSubmissionsManyChunks Random 4000 56.1 53.5 / 62.9 FewSubmissionsManyChunks PreferDistinct(company_id, Oldest) 4000 1100.2 799.4 / 1247.0