Repository navigation
ci: harden the GitHub Actions suite (priority items 1–8) - #1023
Merged
Merged
Conversation
The lint job ran with continue-on-error, so Ruff failures surfaced as a red job inside a green workflow. The baseline is clean on dev with the pinned ruff==0.15.18, so the job can fail the workflow without any code changes. Co-authored-by: csmangum <csmangum@gmail.com>
The workflow only triggered on main, but all contributor PRs target dev, so the threshold checks never ran on the PR that could introduce a regression. Adds dev to both triggers, covers farm/core/decision and farm/config (transitive imports of the regression test), dependency files, and a manual trigger. Co-authored-by: csmangum <csmangum@gmail.com>
The wheel previously shipped with no runtime dependencies, no requires-python, no YAML config files, and bundled tests/, scripts/ and benchmarks/ as top-level packages. pyproject.toml is now the single source of truth: [project.dependencies] mirrors the former requirements.txt, a dev extra holds the test tooling, packages.find is restricted to farm*, and farm.config package data is declared. requirements.txt is reduced to '-e .[dev]'. Moves scripts/analysis_config.py into farm/analysis/ because three package modules imported it, so an installed wheel could not import farm.api.server. Adds scripts/ci/package_smoke.py and a 'package' CI job that builds sdist+wheel, runs twine check, installs the wheel into a clean venv, and runs a 5-step simulation from outside the checkout. Release workflow also runs twine check. Python floor documented as 3.10. Co-authored-by: csmangum <csmangum@gmail.com>
… canary Adds requirements-lock.txt, generated by 'make lock' (uv pip compile, universal across platforms for Python >= 3.10, no --upgrade so pins only move when pyproject.toml changes or 'make lock-upgrade' is run). All Python CI jobs now install from the lock with 'pip install -e . --no-deps', and the lint job runs 'make lock-check' so a stale lock fails CI. New latest-deps workflow installs the unpinned [project.dependencies] weekly and runs the fast suite, so upstream breakage is detected and attributed separately from code changes. Co-authored-by: csmangum <csmangum@gmail.com>
Replaces the single ubuntu/3.10 job with a matrix: ubuntu 3.10 and 3.12 are required legs that enforce the coverage gate; ubuntu 3.13, Windows 3.12 and macOS 3.12 are informational legs (continue-on-error) that run the same suite without the gate. pip caching moves to setup-python so it works on every OS. A new 'python' job aggregates the required pytest legs plus lint, docs-links and package, keeping the status-check name the branch ruleset already requires while making those jobs effectively blocking. Co-authored-by: csmangum <csmangum@gmail.com>
Co-authored-by: csmangum <csmangum@gmail.com>
Dependabot covers pip, the farm/editor npm toolchain and GitHub Actions with weekly grouped minor/patch bumps. A new security workflow runs pip-audit (blocking, 3.10 and 3.12) against requirements-lock.txt with explicitly justified ignores for the protobuf advisories pinned by tianshou 0.4.11 and the Pillow advisories that need Python 3.11+, plus an informational npm audit. CodeQL analyzes Python and JavaScript on push, PR and a weekly schedule. To get the lock off Pillow 11.3 (73 open advisories): moviepy moves to a new 'media' extra because moviepy 2.x caps Pillow below 12, and the Pillow requirement forks so 3.10 gets 12.x and 3.11+ gets 13. Co-authored-by: csmangum <csmangum@gmail.com>
Every PR-triggered workflow now cancels superseded runs for the same ref, and every job has a timeout-minutes instead of the 6-hour default. The nightly workflow no longer runs tests/test_deterministic_pytest.py a second time (-m "" already selects it). A composite action opens or comments on a 'ci-failure' tracking issue when nightly-heavy-tests, latest-deps or the scheduled evolution-regression run fails. reproduce-consensus.yml actions are SHA-pinned like the rest. Co-authored-by: csmangum <csmangum@gmail.com>
… benchmark smokes - typecheck job runs mypy (scoped via [tool.mypy] to farm/core, farm/api, farm/config); informational until the ~400-error baseline is cleared. - Required pytest legs write coverage.xml and upload it as an artifact. - scripts/ci/api_smoke.sh starts uvicorn as a real process and probes read-only endpoints; runs on the required pytest legs. - pages.yml builds Jekyll on pull requests touching docs/ and deploys only on push/workflow_dispatch. - Nightly benchmark-smoke job runs benchmarks/specs/ci_smoke.yaml (observation_flow, a few seconds) and uploads the results JSON. Co-authored-by: csmangum <csmangum@gmail.com>
Reflects the matrix + gate job, packaging and API smokes, lock file, security scanning, scheduled-failure issues, and the remaining gaps (ruleset additions, mypy baseline, ruff format, benchmark trending). Co-authored-by: csmangum <csmangum@gmail.com>
…2 CI leg Axes.boxplot() removed the 'labels' keyword in matplotlib 3.11; use tick_labels (available since 3.9) and raise the matplotlib floor to match. The genetics CI-bounds test used a perfectly linear fitness, so with scipy 1.18 the slope standard error is exactly zero and the strict ordering assertion fails; add deterministic noise so the interval is non-degenerate. Co-authored-by: csmangum <csmangum@gmail.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: Nightly reporter can open duplicates
- Replaced the per-job failure() reporter steps in python-heavy and benchmark-smoke with a single report-failure job that needs both and runs once on failure(), so only one list-then-create executes per workflow run.
Or push these changes by commenting:
@cursor push 75787e9d95
Preview (75787e9d95)
diff --git a/.github/workflows/nightly-heavy-tests.yml b/.github/workflows/nightly-heavy-tests.yml
--- a/.github/workflows/nightly-heavy-tests.yml
+++ b/.github/workflows/nightly-heavy-tests.yml
@@ -15,9 +15,6 @@
python-heavy:
runs-on: ubuntu-latest
timeout-minutes: 90
- permissions:
- contents: read
- issues: write
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
@@ -38,20 +35,12 @@
- name: Run full heavy pytest suite
run: |
pytest -q -vv --durations=50 -m "" --cov=farm --cov-config=.coveragerc --cov-report=term-missing --cov-fail-under=90
- - name: Report failure
- if: failure()
- uses: ./.github/actions/report-scheduled-failure
- with:
- token: ${{ github.token }}
benchmark-smoke:
# Exercises the benchmark runner end to end on a tiny spec and keeps the
# results JSON as an artifact so timings can be trended over time.
runs-on: ubuntu-latest
timeout-minutes: 30
- permissions:
- contents: read
- issues: write
env:
PYTHONHASHSEED: "0"
steps:
@@ -77,8 +66,19 @@
name: benchmark-smoke-results
path: benchmarks/results/ci_smoke/
if-no-files-found: error
- - name: Report failure
- if: failure()
- uses: ./.github/actions/report-scheduled-failure
+
+ report-failure:
+ # Single reporter for the whole workflow so concurrent job failures cannot
+ # race the action's list-then-create and open duplicate issues.
+ needs: [python-heavy, benchmark-smoke]
+ if: failure()
+ runs-on: ubuntu-latest
+ timeout-minutes: 5
+ permissions:
+ contents: read
+ issues: write
+ steps:
+ - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
+ - uses: ./.github/actions/report-scheduled-failure
with:
token: ${{ github.token }}You can send follow-ups to the cloud agent here.
Reviewed by Cursor Bugbot for commit 42ab4f2. Configure here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


Description
Implements the eight priority items from the CI assessment in
docs/guides/continuous-integration.md, one commit each, stacked on #1022 (which adds that guide).ci: make the Ruff lint job blockingcontinue-on-errorfromlint. Baseline is already clean ondevwith the pinnedruff==0.15.18.ci: run evolution regression on dev and widen its path filterdevto both triggers (PRs never targetedmain), coversfarm/core/decision/**,farm/config/**, dependency files; addsworkflow_dispatch.build: declare package metadata and add a packaging smoke jobpyproject.tomlnow hasrequires-python = ">=3.10",[project.dependencies], adevextra,packages.find include = ["farm*"], andfarm.configpackage data. The wheel previously had no dependencies, no YAML configs, and shippedtests/,scripts/,benchmarks/as top-level packages. Newpackagejob:build→twine check→ install wheel into a clean venv →scripts/ci/package_smoke.pyfrom outside the checkout (5-step simulation). That smoke immediately foundfarm.api.serverimportingscripts.analysis_config; the module moved tofarm/analysis/analysis_config.py.ci: install from a universal lock file and add a weekly unpinned-deps canaryrequirements-lock.txtviamake lock(uv pip compile --universal, Python ≥3.10). All jobs install from it;make lock-checkinlintfails on a stale lock.latest-deps.ymlruns the fast suite weekly against unpinned deps.ci: test on a Python/OS matrix behind a single 'python' gate jobpythongate jobneedsthe matrix +lint+docs-links+package, so those block merges without changing the ruleset (which requires a check literally namedpython).ci: add Dependabot, pip-audit and CodeQL scanningdependabot.yml(pip, npm, actions; grouped minor/patch).security.yml:pip-auditblocking on 3.10 + 3.12 with explicitly justified ignores (protobuf pinned by tianshou 0.4.11; Pillow 12.x on 3.10 since Pillow 13 needs 3.11+),npm auditinformational.codeql.ymlfor Python + JS. To get off Pillow 11.3 (73 advisories), moviepy moved to amediaextra (it caps Pillow <12) and Pillow forks by Python version.ci: add concurrency groups, job timeouts and scheduled-failure issuesconcurrencyon every PR workflow,timeout-minuteson every job, nightly no longer runstest_deterministic_pytest.pytwice,reproduce-consensus.ymlactions SHA-pinned. Composite action.github/actions/report-scheduled-failureopens/comments aci-failureissue for nightly, latest-deps, and scheduled evolution-regression failures.ci: add type checking, coverage artifacts, docs build on PRs, API and benchmark smokestypecheckjob (mypy, informational, scoped via[tool.mypy]),coverage.xmlartifact per required leg,scripts/ci/api_smoke.sh(realuvicornprocess, 4 endpoints) on required legs,pages.ymlbuilds Jekyll on docs PRs and deploys only on push, nightlybenchmark-smokerunsbenchmarks/specs/ci_smoke.yamland uploads results.fix: matplotlib 3.11 and scipy 1.18 compatibility surfaced by the 3.12 CI legdevCI never sees (it only runs 3.10):Axes.boxplot(labels=)was removed in matplotlib 3.11 →tick_labels=; a genetics test used a perfectly linear fitness so the slope CI collapsed to a point under scipy 1.18.docs: update the CI guide …ruff format, benchmark trending).Still needs an admin
The ruleset still requires only
pythonandjs-ui.deterministic-testandpip-auditdo not block yet; the guide's Merge gating section has thegh apicommand to add them.Expectations on first run
The informational legs (3.13, Windows, macOS) have never run and may be red; that is what
continue-on-erroris for. Thepackagejob installs torch into a second venv, so it is a few minutes longer than the others.Related Issues
Stacked on #1022.
Checklist
devbranch (notmain) — based on the docs: add continuous-integration guide for the GitHub Actions suite #1022 branch; retarget todevafter that mergesdevpytest) and it passes — full fast suite with--cov-fail-under=90locally on Python 3.12 with the lock versions: coverage 90.28%, 5 failures found and fixed, the affected analysis tests re-run: 432 passedruff check .) — clean withruff==0.15.18;actionlintclean on all workflowstests/analysis/test_genetics.py(degenerate CI fixture); CI smoke scripts underscripts/ci/Note
Medium Risk
Broad CI and packaging changes affect every merge and how wheels install; the Python 3.10 floor and analysis_config import move can break downstream setups that relied on older Python or
scripts.analysis_config.Reviewed by Cursor Bugbot for commit 42ab4f2. Configure here.