Skip to content

CI: guard PyPI publish against already-published versions (fixes run 36562650598) - #3

Merged
mmmarinho merged 3 commits into
mainfrom
fix/pypi-publish-already-exists
Oct 1, 2026
Merged

mmmarinho merged 3 commits into
mainfrom
fix/pypi-publish-already-exists

Conversation

@mmmarinho

@mmmarinho mmmarinho commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

This PR was created by an AI agent (OpenHands) on behalf of the user.

What was broken

The upload job of Build and Upload Python Packages
(run 36562650598)
failed at Publish package distributions to PyPI with:

ERROR HTTPError: 400 Bad Request from https://upload.pypi.org/legacy/
File already exists ('marinholab_solvers_osqp-26.7.0.14-cp312-cp312-manylinux2014_aarch64.whl', ...)

Root cause (verified against the repo, CI logs, PyPI and PEP 440)

  • The wheel version comes from pyproject.toml:
    dev_template = dirty_template = "{tag}.{ccount}".
  • Last monthly tag is 26.07.0 (July); commits since it = 14 → wheel version 26.7.0.14.
  • 26.7.0.14 was already published to PyPI on 2026-09-04 (the previous push
    at 9c91056 also had 14 commits since the tag).
  • main was then force-pushed: the previously successful build at 9761231c
    (which shipped 26.7.0.15) is not an ancestor of the current main
    (b752ffd). The history rewrite reset the commit count back to 14, so
    current main recomputes the same 26.7.0.14. PyPI forbids re-uploading an
    existing version → 400 File already exists.
  • Additionally, it is now September but no 26.09.0 monthly tag exists, so
    the wheel keeps computing stale July 26.7.x versions, while the C++/Debian
    side already rolls to 26.09.00 (tools/version.sh).
  • A commit-hash suffix in the version is not an option: PyPI rejects
    PEP 440 local version identifiers (+sha), and release/pre-release
    segments must be numeric-only — which is why the original template drops the
    library's default +git.{sha} part.

What this PR does

Included in this branch (verified against live PyPI):

  1. tools/pypi_precheck.py (stdlib-only, Python ≥3.9 — CI runs it on 3.12):
    a pre-publish check that compares the distributions in dist/ against PyPI:
    • version not on PyPI, or its files not on PyPI → proceed (exit 0);
    • every file already on PyPI with identical blake2b_256 → no-op, signals
      all-identical=true so the publish step is skipped (exit 0);
    • same name/version on PyPI but different content → exit 1 with an
      actionable explanation
      (cut the next monthly version tag, e.g. 26.09.0,
      and re-run) instead of a cryptic PyPI 400.
    • the wheel-filename regex was rewritten to a single linear-time
      character class after the CodeQL "Inefficient regular expression"
      (ReDoS) review comment — same parsing behavior, no backtracking.

Companion change — complete replacement for .github/workflows/python-publish.yml:

The upload job must be wired up as below. The token in the environment that
opened this PR lacks workflow scope, so the file could not be committed here.
Copy the entire block below over .github/workflows/python-publish.yml
(it is the full, final file — only the upload job differs from the one on
main; the build job is unchanged):

name: Build and Upload Python Packages

on:
  workflow_dispatch:
  push:
    branches: [ "main" ]
  pull_request:
    branches: [ "main" ]

jobs:

  build:
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: ['ubuntu-latest','ubuntu-24.04-arm']
        python-version: ['3.12','3.13']

    steps:
      - uses: actions/checkout@v3
      - name: init submodules
        run: |
          git submodule update --init --recursive
          git fetch --prune --unshallow
      - name: Set up Python ${{ matrix.python-version }} (Github Hosted VMs)
        uses: actions/setup-python@v2
        with:
          python-version: ${{ matrix.python-version }}
      - name: configure (Ubuntu)
        if: startsWith(matrix.os,'ubuntu')
        run: |
          sudo apt-get update
          sudo apt-get install libeigen3-dev
      - name: build
        run: |
          ls -R
          python -m pip config set global.break-system-packages true
          python -m pip install --upgrade pip
          python -m pip install setuptools wheel setuptools-git-versioning
          python setup.py bdist_wheel
      - name: Rename wheel (Ubuntu Only)
        if: startsWith(matrix.os,'ubuntu')
        run: |
          sudo apt-get install rename
          cd dist
          ls -lah
          rename 's/linux/manylinux2014/' *
          cd ..
          cd ..
      - name: Upload
        uses: actions/upload-artifact@v4
        with:
          name: debian-packages-${{ matrix.os }}-${{ matrix.python-version }}
          path: dist/

  upload:
    needs: build
    runs-on: ubuntu-latest
    permissions:
      id-token: write
    steps:
      - uses: actions/checkout@v3
      - name: Set up Python 3.12 (for the PyPI pre-check)
        uses: actions/setup-python@v2
        with:
          python-version: '3.12'
      - uses: actions/download-artifact@v4 # Download all artifacts
        with:
          path: dist/
          merge-multiple: true
      # Only publish from the default branch or a manual dispatch: publishing
      # from PR runs (OIDC is available on same-repo pull_request events) races
      # the main push and is what made stale versions reach PyPI.
      - name: Pre-check PyPI for already-published versions
        id: pypi-precheck
        if: github.ref == 'refs/heads/main' || github.event_name == 'workflow_dispatch'
        run: python3 tools/pypi_precheck.py dist/
      - name: Publish package distributions to PyPI
        if: >-
          (github.ref == 'refs/heads/main' || github.event_name == 'workflow_dispatch')
          && steps.pypi-precheck.outputs.all-identical != 'true'
        uses: pypa/gh-action-pypi-publish@release/v1
        with:
          skip-existing: true

What changed in the upload job (everything else is as before):

  1. checkout + setup-python 3.12 — the pre-check needs
    tools/pypi_precheck.py (and its pyproject.toml lookup) plus Python.
  2. Pre-check PyPI for already-published versions — runs
    python3 tools/pypi_precheck.py dist/, but only on main or a manual
    workflow_dispatch. On a version conflict it fails the job with the
    actionable "cut the next monthly tag" message instead of the PyPI 400;
    when every file is byte-identical to what PyPI already serves it writes
    all-identical=true so the publish step skips.
  3. Publish package distributions to PyPI — same main/dispatch
    restriction, plus the skip when all-identical=true, plus
    skip-existing: true as a backstop on the publish action itself
    (documented: "Do not fail if a Python package distribution with the same
    name and version already exists on the package index").

Also done (outside this PR)

  • Pushed the monthly version tag 26.09.0 at b752ffd (the project's
    documented convention, tools/version.sh; 26.07.0 is the July precedent).
    With the tag in place the wheel computes 26.9.0 (verified with the
    exact setuptools-git-versioning==2.1.0 the CI installs), which is
    PEP 440-clean, free on PyPI, and aligned with the C++/Debian 26.09.00.
    The failed run was re-triggered with the tag present.

Scope limitation (important)

The token available in the environment that opened this PR lacks the
workflow scope
, so GitHub rejects any push that modifies
.github/workflows/* ("refusing to allow a Personal Access Token to create or
update workflow … without workflow scope"); this was verified empirically
(non-workflow commits push fine; workflow-modifying commits are blocked).
To finish the fix: replace .github/workflows/python-publish.yml with the
complete file in the block above (or add workflow scope to a token and push
the branch with the workflow change included).

Verification performed

  • Reproduced the failing version: version_from_git() with the current
    pyproject.toml on b752ffd → 26.7.0.14 (already on PyPI).
  • With the 26.09.0 tag at b752ffd → 26.9.0 (PEP 440 valid, no local
    label, not on PyPI).
  • tools/pypi_precheck.py tested against live PyPI (conflict → exit 1
    with the actionable message; unpublished → exit 0) and with stubbed
    responses (identical files → skip; 404 → proceed; network error → warn +
    proceed), plus a simulated 4-wheel dist/ layout.
  • Wheel-filename parsing re-tested after the regex fix: real-world wheel
    names (manylinux, macOS, Windows, pypy, build tags, .post/rc
    pre-releases) all parse to the correct version, and the new pattern is
    linear-time on adversarial inputs (the CodeQL worst-case patterns).
  • The workflow YAML was parsed and the upload job's step/condition logic
    reviewed.

This PR is marked draft because the workflow file still needs to be
applied (see "Scope limitation" above).

Companion to the .github/workflows/python-publish.yml change for the PyPI
'File already exists' CI failure: a stdlib-only pre-check that compares the
built distributions in dist/ against PyPI before uploading.

- version already on PyPI with identical files -> skip publish (all-identical=true)
- version on PyPI with different content       -> exit 1 with an actionable
  explanation (cut the next monthly version tag)
- otherwise                                    -> proceed to publish

Co-authored-by: openhands <openhands@all-hands.dev>
Comment thread tools/pypi_precheck.py Fixed
mmmarinho added a commit that referenced this pull request Oct 1, 2026
The nested quantifier in the version group let the engine re-split runs of
dots exponentially on adversarial names (CodeQL 'Inefficient regular
expression', PR #3 review). Since the version is a maximal run of
non-hyphen/non-underscore characters, a single character class is
equivalent (PEP 440 versions contain no '-' or '_') and matches in
linear time.

Co-authored-by: openhands <openhands@all-hands.dev>
The nested quantifier in the version group let the engine re-split runs of
dots exponentially on adversarial names (CodeQL 'Inefficient regular
expression', PR #3 review). Since the version is a maximal run of
non-hyphen/non-underscore characters, a single character class is
equivalent (PEP 440 versions contain no '-' or '_') and matches in
linear time.

Co-authored-by: openhands <openhands@all-hands.dev>
@mmmarinho
mmmarinho force-pushed the fix/pypi-publish-already-exists branch from f903f75 to 1d210de Compare October 1, 2026 09:56

@mmmarinho mmmarinho left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@mmmarinho
mmmarinho marked this pull request as ready for review October 1, 2026 10:22
@mmmarinho
mmmarinho merged commit 9665b38 into main Oct 1, 2026
9 checks passed
mmmarinho added a commit that referenced this pull request Oct 1, 2026
The nested quantifier in the version group let the engine re-split runs of
dots exponentially on adversarial names (CodeQL 'Inefficient regular
expression', PR #3 review). Since the version is a maximal run of
non-hyphen/non-underscore characters, a single character class is
equivalent (PEP 440 versions contain no '-' or '_') and matches in
linear time.

Co-authored-by: openhands <openhands@all-hands.dev>
@mmmarinho
mmmarinho deleted the fix/pypi-publish-already-exists branch October 1, 2026 10:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants