Skip to content

ci: harden the GitHub Actions suite (priority items 1–8) - #1023

Merged
csmangum merged 11 commits into
cursor/ci-workflows-doc-940ffrom
cursor/ci-hardening-940f
Nov 1, 2026
Merged

csmangum merged 11 commits into
cursor/ci-workflows-doc-940ffrom
cursor/ci-hardening-940f

Conversation

@csmangum

@csmangum csmangum commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Description

Implements the eight priority items from the CI assessment in docs/guides/continuous-integration.md, one commit each, stacked on #1022 (which adds that guide).

# Commit What changed
1 ci: make the Ruff lint job blocking Drops continue-on-error from lint. Baseline is already clean on dev with the pinned ruff==0.15.18.
2 ci: run evolution regression on dev and widen its path filter Adds dev to both triggers (PRs never targeted main), covers farm/core/decision/**, farm/config/**, dependency files; adds workflow_dispatch.
3 build: declare package metadata and add a packaging smoke job pyproject.toml now has requires-python = ">=3.10", [project.dependencies], a dev extra, packages.find include = ["farm*"], and farm.config package data. The wheel previously had no dependencies, no YAML configs, and shipped tests/, scripts/, benchmarks/ as top-level packages. New package job: build → twine check → install wheel into a clean venv → scripts/ci/package_smoke.py from outside the checkout (5-step simulation). That smoke immediately found farm.api.server importing scripts.analysis_config; the module moved to farm/analysis/analysis_config.py.
4 ci: install from a universal lock file and add a weekly unpinned-deps canary requirements-lock.txt via make lock (uv pip compile --universal, Python ≥3.10). All jobs install from it; make lock-check in lint fails on a stale lock. latest-deps.yml runs the fast suite weekly against unpinned deps.
5 ci: test on a Python/OS matrix behind a single 'python' gate job Required legs: ubuntu 3.10 + 3.12 (coverage gate). Informational: ubuntu 3.13, Windows 3.12, macOS 3.12. A python gate job needs the matrix + lint + docs-links + package, so those block merges without changing the ruleset (which requires a check literally named python).
6 ci: add Dependabot, pip-audit and CodeQL scanning dependabot.yml (pip, npm, actions; grouped minor/patch). security.yml: pip-audit blocking on 3.10 + 3.12 with explicitly justified ignores (protobuf pinned by tianshou 0.4.11; Pillow 12.x on 3.10 since Pillow 13 needs 3.11+), npm audit informational. codeql.yml for Python + JS. To get off Pillow 11.3 (73 advisories), moviepy moved to a media extra (it caps Pillow <12) and Pillow forks by Python version.
7 ci: add concurrency groups, job timeouts and scheduled-failure issues concurrency on every PR workflow, timeout-minutes on every job, nightly no longer runs test_deterministic_pytest.py twice, reproduce-consensus.yml actions SHA-pinned. Composite action .github/actions/report-scheduled-failure opens/comments a ci-failure issue for nightly, latest-deps, and scheduled evolution-regression failures.
8 ci: add type checking, coverage artifacts, docs build on PRs, API and benchmark smokes typecheck job (mypy, informational, scoped via [tool.mypy]), coverage.xml artifact per required leg, scripts/ci/api_smoke.sh (real uvicorn process, 4 endpoints) on required legs, pages.yml builds Jekyll on docs PRs and deploys only on push, nightly benchmark-smoke runs benchmarks/specs/ci_smoke.yaml and uploads results.
— fix: matplotlib 3.11 and scipy 1.18 compatibility surfaced by the 3.12 CI leg Running the suite under the lock's 3.12 versions found 5 failures dev CI never sees (it only runs 3.10): Axes.boxplot(labels=) was removed in matplotlib 3.11 → tick_labels=; a genetics test used a perfectly linear fitness so the slope CI collapsed to a point under scipy 1.18.
— docs: update the CI guide … Guide now describes the new state and the remaining gaps (ruleset additions needing admin, mypy baseline, ruff format, benchmark trending).

Still needs an admin

The ruleset still requires only python and js-ui. deterministic-test and pip-audit do not block yet; the guide's Merge gating section has the gh api command to add them.

Expectations on first run

The informational legs (3.13, Windows, macOS) have never run and may be red; that is what continue-on-error is for. The package job installs torch into a second venv, so it is a few minutes longer than the others.

Related Issues

Stacked on #1022.

Checklist

  • This PR targets the dev branch (not main) — based on the docs: add continuous-integration guide for the GitHub Actions suite #1022 branch; retarget to dev after that merges
  • My branch is up to date with the latest dev
  • I ran the test suite (pytest) and it passes — full fast suite with --cov-fail-under=90 locally on Python 3.12 with the lock versions: coverage 90.28%, 5 failures found and fixed, the affected analysis tests re-run: 432 passed
  • I linted my changes (ruff check .) — clean with ruff==0.15.18; actionlint clean on all workflows
  • I added or updated tests for behavior changes — tests/analysis/test_genetics.py (degenerate CI fixture); CI smoke scripts under scripts/ci/
  • I updated documentation as necessary
Open in Web Open in Cursor 

Note

Medium Risk
Broad CI and packaging changes affect every merge and how wheels install; the Python 3.10 floor and analysis_config import move can break downstream setups that relied on older Python or scripts.analysis_config.

Reviewed by Cursor Bugbot for commit 42ab4f2. Configure here.

cursoragent and others added 11 commits October 31, 2026 23:54
The lint job ran with continue-on-error, so Ruff failures surfaced as a
red job inside a green workflow. The baseline is clean on dev with the
pinned ruff==0.15.18, so the job can fail the workflow without any
code changes.

Co-authored-by: csmangum <csmangum@gmail.com>
The workflow only triggered on main, but all contributor PRs target dev,
so the threshold checks never ran on the PR that could introduce a
regression. Adds dev to both triggers, covers farm/core/decision and
farm/config (transitive imports of the regression test), dependency
files, and a manual trigger.

Co-authored-by: csmangum <csmangum@gmail.com>
The wheel previously shipped with no runtime dependencies, no
requires-python, no YAML config files, and bundled tests/, scripts/ and
benchmarks/ as top-level packages. pyproject.toml is now the single
source of truth: [project.dependencies] mirrors the former
requirements.txt, a dev extra holds the test tooling, packages.find is
restricted to farm*, and farm.config package data is declared.
requirements.txt is reduced to '-e .[dev]'.

Moves scripts/analysis_config.py into farm/analysis/ because three
package modules imported it, so an installed wheel could not import
farm.api.server. Adds scripts/ci/package_smoke.py and a 'package' CI
job that builds sdist+wheel, runs twine check, installs the wheel into
a clean venv, and runs a 5-step simulation from outside the checkout.
Release workflow also runs twine check. Python floor documented as
3.10.

Co-authored-by: csmangum <csmangum@gmail.com>
… canary

Adds requirements-lock.txt, generated by 'make lock' (uv pip compile,
universal across platforms for Python >= 3.10, no --upgrade so pins only
move when pyproject.toml changes or 'make lock-upgrade' is run). All
Python CI jobs now install from the lock with 'pip install -e . --no-deps',
and the lint job runs 'make lock-check' so a stale lock fails CI.

New latest-deps workflow installs the unpinned [project.dependencies]
weekly and runs the fast suite, so upstream breakage is detected and
attributed separately from code changes.

Co-authored-by: csmangum <csmangum@gmail.com>
Replaces the single ubuntu/3.10 job with a matrix: ubuntu 3.10 and 3.12
are required legs that enforce the coverage gate; ubuntu 3.13, Windows
3.12 and macOS 3.12 are informational legs (continue-on-error) that run
the same suite without the gate. pip caching moves to setup-python so it
works on every OS.

A new 'python' job aggregates the required pytest legs plus lint,
docs-links and package, keeping the status-check name the branch ruleset
already requires while making those jobs effectively blocking.

Co-authored-by: csmangum <csmangum@gmail.com>
Co-authored-by: csmangum <csmangum@gmail.com>
Dependabot covers pip, the farm/editor npm toolchain and GitHub Actions
with weekly grouped minor/patch bumps. A new security workflow runs
pip-audit (blocking, 3.10 and 3.12) against requirements-lock.txt with
explicitly justified ignores for the protobuf advisories pinned by
tianshou 0.4.11 and the Pillow advisories that need Python 3.11+, plus an
informational npm audit. CodeQL analyzes Python and JavaScript on push,
PR and a weekly schedule.

To get the lock off Pillow 11.3 (73 open advisories): moviepy moves to a
new 'media' extra because moviepy 2.x caps Pillow below 12, and the
Pillow requirement forks so 3.10 gets 12.x and 3.11+ gets 13.

Co-authored-by: csmangum <csmangum@gmail.com>
Every PR-triggered workflow now cancels superseded runs for the same
ref, and every job has a timeout-minutes instead of the 6-hour default.
The nightly workflow no longer runs tests/test_deterministic_pytest.py a
second time (-m "" already selects it). A composite action opens or
comments on a 'ci-failure' tracking issue when nightly-heavy-tests,
latest-deps or the scheduled evolution-regression run fails.
reproduce-consensus.yml actions are SHA-pinned like the rest.

Co-authored-by: csmangum <csmangum@gmail.com>
… benchmark smokes

- typecheck job runs mypy (scoped via [tool.mypy] to farm/core, farm/api,
  farm/config); informational until the ~400-error baseline is cleared.
- Required pytest legs write coverage.xml and upload it as an artifact.
- scripts/ci/api_smoke.sh starts uvicorn as a real process and probes
  read-only endpoints; runs on the required pytest legs.
- pages.yml builds Jekyll on pull requests touching docs/ and deploys
  only on push/workflow_dispatch.
- Nightly benchmark-smoke job runs benchmarks/specs/ci_smoke.yaml
  (observation_flow, a few seconds) and uploads the results JSON.

Co-authored-by: csmangum <csmangum@gmail.com>
Reflects the matrix + gate job, packaging and API smokes, lock file,
security scanning, scheduled-failure issues, and the remaining gaps
(ruleset additions, mypy baseline, ruff format, benchmark trending).

Co-authored-by: csmangum <csmangum@gmail.com>
…2 CI leg

Axes.boxplot() removed the 'labels' keyword in matplotlib 3.11; use
tick_labels (available since 3.9) and raise the matplotlib floor to
match. The genetics CI-bounds test used a perfectly linear fitness, so
with scipy 1.18 the slope standard error is exactly zero and the strict
ordering assertion fails; add deterministic noise so the interval is
non-degenerate.

Co-authored-by: csmangum <csmangum@gmail.com>

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

Bugbot Autofix prepared a fix for the issue found in the latest run.

  • ✅ Fixed: Nightly reporter can open duplicates
    • Replaced the per-job failure() reporter steps in python-heavy and benchmark-smoke with a single report-failure job that needs both and runs once on failure(), so only one list-then-create executes per workflow run.

Create PR

Or push these changes by commenting:

@cursor push 75787e9d95
Preview (75787e9d95)
diff --git a/.github/workflows/nightly-heavy-tests.yml b/.github/workflows/nightly-heavy-tests.yml
--- a/.github/workflows/nightly-heavy-tests.yml
+++ b/.github/workflows/nightly-heavy-tests.yml
@@ -15,9 +15,6 @@
   python-heavy:
     runs-on: ubuntu-latest
     timeout-minutes: 90
-    permissions:
-      contents: read
-      issues: write
     steps:
       - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
       - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
@@ -38,20 +35,12 @@
       - name: Run full heavy pytest suite
         run: |
           pytest -q -vv --durations=50 -m "" --cov=farm --cov-config=.coveragerc --cov-report=term-missing --cov-fail-under=90
-      - name: Report failure
-        if: failure()
-        uses: ./.github/actions/report-scheduled-failure
-        with:
-          token: ${{ github.token }}
 
   benchmark-smoke:
     # Exercises the benchmark runner end to end on a tiny spec and keeps the
     # results JSON as an artifact so timings can be trended over time.
     runs-on: ubuntu-latest
     timeout-minutes: 30
-    permissions:
-      contents: read
-      issues: write
     env:
       PYTHONHASHSEED: "0"
     steps:
@@ -77,8 +66,19 @@
           name: benchmark-smoke-results
           path: benchmarks/results/ci_smoke/
           if-no-files-found: error
-      - name: Report failure
-        if: failure()
-        uses: ./.github/actions/report-scheduled-failure
+
+  report-failure:
+    # Single reporter for the whole workflow so concurrent job failures cannot
+    # race the action's list-then-create and open duplicate issues.
+    needs: [python-heavy, benchmark-smoke]
+    if: failure()
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    permissions:
+      contents: read
+      issues: write
+    steps:
+      - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
+      - uses: ./.github/actions/report-scheduled-failure
         with:
           token: ${{ github.token }}

You can send follow-ups to the cloud agent here.

Reviewed by Cursor Bugbot for commit 42ab4f2. Configure here.

@csmangum
csmangum merged commit 8d0970c into cursor/ci-workflows-doc-940f Nov 1, 2026
4 checks passed
@csmangum
csmangum deleted the cursor/ci-hardening-940f branch November 1, 2026 01:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants