Repository navigation
Add AI generated test suite - #96
Draft
olafmersmann wants to merge 10 commits into
Draft
olafmersmann wants to merge 10 commits into
olafmersmann wants to merge 10 commits into
Conversation
conftest.py isolates the global state of cocopp (module globals, testbed settings, rcParams, numpy RNG and error settings, cwd) around each test, seeds numpy, runs each test in a fresh working directory, and redirects package-relative files into a temporary copy of the package such that tests never write into the source tree (the session fails otherwise). Tests needing the online archives are marked 'network' and only run with --run-network, tests running the full pipeline are marked 'slow'. synthdata.py writes deterministic synthetic data (bbob in the current and pre-2016 format, bbob-noisy, bbob-constrained with g-evaluations, bbob-largescale, bbob-mixint, bbob-boxed and bbob-biobj) and a tiny hand-crafted data set with hand-computable runtimes. pipeline.py runs cocopp.main in-process and defines the scenarios shared by integration and regression tests, snapshots.py takes and compares normalized golden-file snapshots of the text output.
Unit tests for the statistics and formatting helpers (toolsstats, toolsdivers, pptex), data reading and alignment (readalign, DataSet on a hand-crafted data set with hand-computed runtimes, DataSetList and the dictAlg helpers), target values, reference algorithm generation, the settings modules (testbedsettings, genericsettings, config, dataformatsettings), findfiles, captions, ppfig helpers, local data archives, reduce_size, sanitycheck and the package interface. Many expected values are characterization values pinning the current behavior. Known bugs are documented as strict xfail tests.
test_modules.py calls the figure and table modules directly on small synthetic data, test_pipeline.py runs cocopp.main on many scenarios (single, two and three algorithms, expensive and convergence settings, background algorithms, --include-single, output options, bbob-biobj, bbob-noisy, bbob-constrained, bbob-largescale, bbob-mixint, bbob-boxed, old data format) and checks the produced files, the validity of figures, and that table entries agree with the data.
Run python -m cocopp, cocopp.main from python -c, cocopp.reduce_size and findfiles in subprocesses that import a temporary copy of the package.
Port the scenarios of the legacy python -m cocopp.test runner, including its doctests and pickle regression test. Skipped unless --run-network.
Snapshots of the normalized text output (all LaTeX and html files, the list of produced files and of broken html links) of 15 pipeline scenarios on deterministic synthetic data. Rewrite them after intended changes with pytest tests/regression --update-snapshots. Verified to be reproducible across processes with different PYTHONHASHSEED values.
How to run the tests, their layout, the isolation of global state, how to update and review snapshots, and the bugs and quirks found while writing the tests.
Python 3.9 reached its end of life in October 2025 and current versions of numpy and matplotlib do not support it anymore.
Remove src/cocopp/test.py. All its scenarios are covered by the pytest suite in tests/network (opt-in with --run-network) and, offline, on synthetic data in tests/integration and tests/regression. The doctests, formerly run with python -m cocopp.test doctest, are run by tests/run_doctests.py, which tests/network calls on a throw-away copy of the package. CI now tests Python 3.10 to 3.14, builds the wheel with each version and tests the installed wheel, lints the tests too, runs the offline suite with pytest tests, and runs the network tests that correspond to the former default CI tests (archives, doctests, subtests 1 and 2, constrained data, pickle regression). setup-python is updated to v5 to provide the new Python versions. As cocopp is pure python, the release workflow now publishes the single py3-none-any wheel next to the sdist. This commit is meant to stay on top of the test stack.
olafmersmann
force-pushed
the
feat/auto-generated-test-suite
branch
from
November 4, 2026 16:33
f15575a to
69f4d95
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.