Skip to content

[FEATURE] Branching strategy Phase 3 - PyPI and CodeArtifact publish workflows - #638

Merged
John McCall (lowlydba) merged 25 commits into
mainfrom
509-devops-branching-strategy---phase-3
Aug 24, 2026
Merged

[FEATURE] Branching strategy Phase 3 - PyPI and CodeArtifact publish workflows#638
John McCall (lowlydba) merged 25 commits into
mainfrom
509-devops-branching-strategy---phase-3

Conversation

@lowlydba

@lowlydba John McCall (lowlydba) commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Summary

Phase 3 of the branching strategy (#509): the publish side of the pipeline Phase 2.B (#557) computed versions for. Internal builds go to CodeArtifact on every push to main; human-cut releases go to public PyPI.

What lands

Internal builds → CodeArtifact

main-publish.yaml, on push to main: detect-affected-packages (new composite action, using tj-actions/changed-files pinned by commit SHA rather than a hand-rolled git diff --name-only, for correct filename quoting and its dir_names-to-package-name folding) diffs the push against packages/** to find packages touched without a version bump (bumped packages release via the path below instead). Each affected package gets a .postN+main.<sha> version stamped in via compute-version, built, and published to CodeArtifact.

The same workflow doubles as a build-only smoke test on PRs touching the composite actions or itself: it discovers every package unconditionally instead of diffing (a PR touching .github/actions/** never touches packages/**, so the diff would always return zero and skip the job it's meant to exercise), builds, and stops short of publishing. Replaces compute-versions-dry-run.yaml.

Releases → PyPI

release-publish.yaml, on release: published: parses <package>-v<version> off the tag, builds, verifies the wheel matches (and, for overture-schema-pyspark, that the generated expressions are in it), and publishes via pypa/gh-action-pypi-publish with OIDC trusted publishing and attestations. The version-bump PR review is the approval gate; no separate environment reviewer step, one human approval per release is enough.

It also takes a workflow_dispatch (package, target: test-pypi|pypi) that always publishes a synthetic <on-disk-version>.dev0, never the real version. This exists to prime a package's PyPI Trusted Publisher from "pending" to "normal" ahead of its real release, working around PyPI's pending-publisher rate limit (~3 at a time, independently on pypi.org and test.pypi.org, see #653). Real-PyPI dispatch reuses the same pypi-release-<package> environment as the automated path; workflow_dispatch already requires repo write access to trigger, which is gate enough for a disposable .dev0. Temporary: #688 tracks removing the pypi target once every package is primed for the v2.0 launch.

release-trigger.yaml creates releases with an overture-release-publisher app installation token instead of GITHUB_TOKEN: GitHub doesn't fire release: published for releases created with the default token, so release-publish.yaml would never see them. Token scoped to contents: write only. App provisioning was tracked on #637 (now done).

A package's first version counts as a bump too: detect_version_bumps.py no longer skips before: null entries, so a brand-new package flows through release-trigger to PyPI on its first version, same as any other bump. detect_affected_packages.py's bumped set was widened to match (any package with a non-null after version), so the same push doesn't also queue it for an internal .postN build.

Mechanical follow-through: diff-package-versions action, deleted workflows, docs

package_versions.py diff was inlined at three call sites (detect-version-bumps, the new detect-affected-packages, and the pre-existing reusable-check-python-package-versions.yaml), each reimplementing the temp-file-then-$GITHUB_OUTPUT plumbing slightly differently. Pulled it into its own diff-package-versions action exposing count/diff outputs; the three consumers now just pipe diff into their own filter logic instead of each knowing the script's path and output convention.

package_versions.py itself moved from .github/workflows/scripts/ into diff-package-versions/, its only remaining caller, so every action in the repo now carries its own script(s) instead of reaching outside its directory.

Deletes compute-versions-dry-run.yaml and publish-python-packages.yaml, both superseded by the two workflows above.

docs/versioning.md gets a workflow-name reference table and the manual-dispatch priming mechanics under the release walkthrough. CONTRIBUTING.md drops the "rolling out in phases" banner (this is the last phase) and corrects two release-flow descriptions that predated the approval gate.

Testing

detect-affected-packages's exclusion logic is tested locally against real repo commit ranges plus synthetic bumped/removed/new-package cases, including through the diff-package-versions wiring and after the script relocation. The uv version <post-release-string> --package <pkg> --frozen --dry-run stamping technique is confirmed to accept PEP 440 local-label and dev-release strings. zizmor is clean on every touched/new workflow and action file. The workflows themselves can't run end to end until the external prerequisites below land.

Still open (tracked on #653, all external/manual)

  • PyPI Trusted Publisher config per package, batched to work around the pending-publisher rate limit
  • pypi-release-<package>/test-pypi-<package> GitHub Environments
  • a resolver smoke test against real CodeArtifact and a real end-to-end PyPI release, once the above are live

vnext dev builds to CodeArtifact moved out of this phase entirely, tracked on Phase 4 (#510) instead: it's blocked on a CodeArtifact repo migration (ops-team#299) with no committed timeline, so Phase 3 ships without waiting on it.

Closes #509
Contributes to #653, #688

…ish workflows

Adds the publish side of the versioning pipeline built in Phase 2.B:

- main-publish.yaml: on push to main, diffs changed packages/** files
  (via the new detect-affected-packages action) to find packages touched
  without a version bump, stamps a .postN+main.<sha> build via
  compute-version, and publishes to CodeArtifact. Also runs as a
  build-only smoke test on PRs touching the composite actions or itself,
  replacing compute-versions-dry-run.yaml.
- 
elease-publish.yaml: on 
elease: published, parses <package>-v<version>
  from the tag, builds, and publishes to PyPI via
  `pypa/gh-action-pypi-publish` (OIDC trusted publishing + attestations),
  gated by the pypi-release environment's required reviewers.
- 
elease-trigger.yaml: releases now get created with an
  overture-release-publisher app installation token instead of
  GITHUB_TOKEN, since GITHUB_TOKEN-created releases don't fire
  
elease: published for other workflows to pick up. App provisioning is
  tracked separately in #637 and has to happen before this path works.

Deletes compute-versions-dry-run.yaml and publish-python-packages.yaml,
both superseded by the two workflows above.

docs/versioning.md gets a workflow-name reference table and the
pypi-release approval-gate mechanics.

Still open, all external/manual, tracked on #509:
- provisioning overture-release-publisher (#637)
- PyPI Trusted Publisher config per package
- pypi-release environment + reviewers
- p3-dev-builds-ca, blocked on ops-team#299

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Signed-off-by: John McCall <john@overturemaps.org>
@github-actions

github-actions Bot commented Aug 5, 2026

Copy link
Copy Markdown

🗺️ Schema reference docs preview is live!

🌍 Preview https://staging.overturemaps.org/schema/pr/638/schema/index.html
🕐 Updated Aug 24, 2026 19:29 UTC
📝 Commit 5022321
🔧 env SCHEMA_PREVIEW true

Note

♻️ This preview updates automatically with each push to this PR.

Replaces the hand-rolled `git diff --name-only` + path-splitting with
tj-actions/changed-files' dir_names output, pinned by commit SHA. Fixes a
real gap in the old parser: git diff --name-only quotes non-ASCII/unusual
filenames, which naive line.split("/") didn't account for. Uses
�ll_modified_files (ACMRD) rather than �ll_changed_files (ACMR) so file
deletions still count as a package change, matching the old diff's behavior.

detect_affected_packages.py drops its subprocess/git plumbing entirely and
just reads the directory list from CHANGED_DIRS; the bump/removed
exclusion logic (a three-way set difference) stays in Python rather than
jq/bash, that's a different complexity class than this repo's existing
jq usage in enforce-change-type-label.yaml.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Signed-off-by: John McCall <john@overturemaps.org>
…scripts run back to back

The action wasn't wrong, just awkward to read: python, jump to a JS
action, jump back to python. Steps 1 (version diff) and 2 (changed
dirs) don't depend on each other, only step 3 does, so nothing stops
them running adjacent.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Signed-off-by: John McCall <john@overturemaps.org>
package_versions.py diff was inlined at three call sites
(detect-version-bumps, detect-affected-packages, and

eusable-check-python-package-versions.yaml), each reimplementing the
temp-file-then-$GITHUB_OUTPUT plumbing slightly differently (one even
wrote to a different temp path and used heredoc-style output).

diff-package-versions is now the one place that knows how to run the
script and expose it as count/diff outputs. The three consumers just
pipe steps.diff.outputs.diff into their own filter logic:
detect_version_bumps.py and detect_affected_packages.py are unchanged,
they already read JSON from stdin. 
eusable-check-python-package-versions.yaml's
CodeArtifact existence check now reads the diff from an env var instead of
a temp file; its changed_packages output is compact JSON now instead of
pretty-printed, no consumer depends on the formatting.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Signed-off-by: John McCall <john@overturemaps.org>
@lowlydba John McCall (lowlydba) added the automation 🦾 Change/enhance automation of a process label Aug 5, 2026
Last remaining dangling script: package_versions.py lived in
.github/workflows/scripts/ and was reached into by relative path.
Every other action already keeps its script(s) alongside its own
action.yml (detect-version-bumps, detect-affected-packages,
create-package-release); this was the one holdout, and after the last
refactor diff-package-versions is its only remaining caller anyway.

Moved it in, switched the reference to ${GITHUB_ACTION_PATH}, and
removed the now-empty scripts/ directory. No behavior change: git
commands inside the script still run with the repo root as CWD, only
the path used to invoke it changed.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Signed-off-by: John McCall <john@overturemaps.org>
…lease flow

Phase 3 was the last phase (#509); Phase 4 got folded into it instead of
staying a separate doc-polish pass (see #490). Drops the "rolling out in
phases" banner pointing at the tracking issue, since the flow it describes
is now fully implemented rather than in progress.

Also corrects two release-flow descriptions that predated the pypi-release
approval gate this phase added: a release no longer lands on PyPI
"immediately", it starts a maintainer-gated publish first.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Signed-off-by: John McCall <john@overturemaps.org>
push never fires for fork PRs (only same-repo branch pushes), so a
required check relying solely on push would never run for external
contributors, a required status check that can never be satisfied by
the PRs GitHub Actions actually needs to gate.

Add pull_request with the same path filters; push stays for post-merge
validation on main.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Signed-off-by: John McCall <john@overturemaps.org>
John McCall (lowlydba) added a commit that referenced this pull request Aug 6, 2026
push never fires for fork PRs, so a required-status-check rule depending
on test-schema.yaml would deadlock every fork PR. Adds a pull_request
trigger alongside push, and switches the concurrency group's PR fallback
to github.event.pull_request.number so per-PR runs don't collide.

Cherry-picked from #638's test-schema.yaml hunk, that PR's scope is the
unrelated Phase 3 publish workflows and won't merge soon, this piece
can't wait on it since it's needed to verify the Build job actually
triggers and reports on a fork PR per #647's acceptance criteria.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Signed-off-by: John McCall <john@overturemaps.org>
release-trigger.yaml assumes the narrow gha-releaser-secrets-reader OIDC
role and fetches the PEM from omf-github-terraform/releaser/pem instead
of reading GHA repo secrets, matching the safe-settings and
project-manager app pattern. Client ID is inlined (not sensitive) instead
of a secret. Companion Terraform PR wires the role and Secrets Manager
entry in omf-github-terraform.

Fixes #637.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Signed-off-by: John McCall <john@overturemaps.org>
John McCall (lowlydba) added a commit that referenced this pull request Aug 11, 2026
…ion (#648)

* [CHORE](ci) Add all-Python-checks gate, resolve Build job-name collision

check-python-code.yaml's check matrix produces one status per cell with no
aggregator, so a required-status-check rule has to track six names and
misses any new Python version until updated. test-schema.yaml and
schema-pr-preview.yml both name their job Build, ambiguous once
test-schema.yaml gets a pull_request trigger and needs pinning itself.

Adds an all-python-checks-pass job to check-python-code.yaml (needs:
[check], if: always()) using lowlydba/are-we-good to roll the matrix's
per-cell results into one pass/fail. Renames schema-pr-preview.yml's build
job from Build to Build preview, leaving test-schema.yaml's Build as the
stable name #646 already plans to pin.

Fixes #647

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Signed-off-by: John McCall <john@overturemaps.org>

* [CHORE](ci) Add pull_request trigger to test-schema.yaml

push never fires for fork PRs, so a required-status-check rule depending
on test-schema.yaml would deadlock every fork PR. Adds a pull_request
trigger alongside push, and switches the concurrency group's PR fallback
to github.event.pull_request.number so per-PR runs don't collide.

Cherry-picked from #638's test-schema.yaml hunk, that PR's scope is the
unrelated Phase 3 publish workflows and won't merge soon, this piece
can't wait on it since it's needed to verify the Build job actually
triggers and reports on a fork PR per #647's acceptance criteria.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Signed-off-by: John McCall <john@overturemaps.org>

---------

Signed-off-by: John McCall <john@overturemaps.org>
detect_version_bumps.py no longer skips before:null entries; a
brand-new package now flows through release-trigger to PyPI on its
first version instead of sitting on CodeArtifact-only .postN builds.

detect_affected_packages.py's bumped set is widened to match (any
package with a non-null after version) so the same push doesn't also
queue an internal build for a package that just got its first release.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Signed-off-by: John McCall <john@overturemaps.org>
Signed-off-by: John McCall <john@overturemaps.org>
@lowlydba
John McCall (lowlydba) marked this pull request as ready for review August 12, 2026 15:25
@lowlydba

Copy link
Copy Markdown
Contributor Author

Copilot resolve the merge conflicts in this pull request

Copilot AI and others added 2 commits August 24, 2026 14:59
Co-authored-by: lowlydba <16843041+lowlydba@users.noreply.github.com>
Co-authored-by: lowlydba <16843041+lowlydba@users.noreply.github.com>

Copilot AI commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Copilot resolve the merge conflicts in this pull request

Resolved by merging origin/main and fixing the conflict state in commit 65bb3bb6 (with a follow-up guard fix in c83e62cc).

… workflow_dispatch

PyPI rate-limits pending Trusted Publisher registrations to ~3 at a time, on
both pypi.org and test.pypi.org independently (see #653). Priming a publisher
past "pending" requires an actual publish, and workflow_dispatch previously
only supported Test PyPI, so there was no way to prime real PyPI ahead of a
package's first release.

workflow_dispatch now takes a target input (test-pypi default, or pypi) and
always publishes a synthetic <on-disk-version>.dev0 instead of the real
version: valid PEP 440, ignored by default resolvers, and never collides with
the eventual human-owned release. Dispatching to real PyPI uses a new
pypi-dispatch-<package> environment with its own required-reviewer gate,
since it's the one path that publishes to production PyPI without a
version-bump PR behind it; pypi-release-<package> (the automated release
path) and test-pypi-<package> are unchanged.

This is a bootstrapping tool, not a permanent fixture. Once all 12 packages
are primed for the v2.0 launch, the pypi target and its dispatch environments
should come out.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Signed-off-by: John McCall <john@overturemaps.org>
…phase-3' into 509-devops-branching-strategy---phase-3

Signed-off-by: John McCall <john@overturemaps.org>
…ackage>, flag temp blocks for #688

Drops the separate pypi-dispatch-<package> environment: workflow_dispatch
already requires repo write access to trigger at all, which is gate enough
for a disposable .dev0 priming publish, so a real-PyPI priming dispatch now
reuses the same pypi-release-<package> environment the automated release
path uses. One environment set per package instead of a third variant.

Also marks every pypi-target-specific block with TODO(#688) so the temporary
priming path (vs. the permanent test-pypi dispatch) is obvious at the code
site, not just in the tracking issue.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Signed-off-by: John McCall <john@overturemaps.org>
@vcschapp

Victor Schappert (vcschapp) commented Aug 24, 2026

Copy link
Copy Markdown
Collaborator

@copilot resolve the merge conflicts in this pull request.

Edit: Copilot can't hear me.

@lowlydba
John McCall (lowlydba) merged commit 5dc32f3 into main Aug 24, 2026
39 checks passed
@lowlydba
John McCall (lowlydba) deleted the 509-devops-branching-strategy---phase-3 branch August 24, 2026 19:34
Seth Fitzsimmons (sethfitz) added a commit that referenced this pull request Aug 26, 2026
Phase 3 (#638) replaced publish-python-packages.yaml with
main-publish.yaml and release-publish.yaml, and moved the PySpark
codegen step out of the workflow into a package-owned
packages/<package>/scripts/prebuild.sh. Describe that convention and the
two publish targets (CodeArtifact for interim .postN builds, public PyPI
via Trusted Publishing for version bumps) instead of the workflow that
no longer exists.

Correct the README package count: packages/ holds thirteen packages, as
GLOSSARY.md and SCHEMA_GUIDE.md already said. extensions-operating-hours
has never been on main.

Run the markdown code blocks through ruff format, which make check has
been failing on independently of this rebase.

Signed-off-by: Seth Fitzsimmons <seth@mojodna.net>
Seth Fitzsimmons (sethfitz) added a commit that referenced this pull request Aug 26, 2026
Phase 3 (#638) replaced publish-python-packages.yaml with
main-publish.yaml and release-publish.yaml, and moved the PySpark
codegen step out of the workflow into a package-owned
packages/<package>/scripts/prebuild.sh. Describe that convention and the
two publish targets (CodeArtifact for interim .postN builds, public PyPI
via Trusted Publishing for version bumps) instead of the workflow that
no longer exists.

Correct the README package count: packages/ holds thirteen packages, as
GLOSSARY.md and SCHEMA_GUIDE.md already said. extensions-operating-hours
has never been on main.

Run the markdown code blocks through ruff format, which make check has
been failing on independently of this rebase.

Signed-off-by: Seth Fitzsimmons <seth@mojodna.net>
Victor Schappert (vcschapp) pushed a commit that referenced this pull request Aug 26, 2026
* update revised and consolidated docs

Signed-off-by: Dana Bauer <dana.bauer@gmail.com>
Signed-off-by: Seth Fitzsimmons <seth@mojodna.net>

* docs: carry the corrections into SCHEMA_GUIDE.md

The consolidation was written against main before the doc fixes landed, so
it brought the stale content forward and catalogued the rest.

Ported from the corrected PYDANTIC_GUIDE.md, extracted from that file
rather than retyped: the module-organization section, the
import-organization example, and the container-to-mixin example built on
the real `Named` / `Appearance` mixins. Section 3.2 is reframed too --
Overture publishes one shape, and GeoJSON is the representation the models
support for compatibility and that the generated JSON Schema describes;
the renamed heading's anchor is chased through GLOSSARY.md and two
in-guide links.

"Docs in the repo that are currently wrong" listed eight items; seven are
fixed, so only `pip install` survives, pointing at the section that
explains it. GLOSSARY.md's Tag entry said there is no plain `overture`
tag, which was true when written and is what prompted adding one; it now
reads eight tags, with a note on what the tag does and does not assert.
The transcribed `list-types` output was regenerated by running the
command.

The imported guide also exposed two holes in the drift detector. A `>>>`
REPL block failed to parse and dropped out of the sweep whole, taking its
imports with it. And the guide deliberately shows `from overture.schema
import Building` as a counter-example, which the detector read as a
defect; a line carrying the counter-example glyph is exempt. Both have unit
fixtures and a mutation that fails them. With those handled and the
validation README's block fixed, no block in the repo is excused.

Signed-off-by: Seth Fitzsimmons <seth@mojodna.net>

* docs: split SCHEMA_GUIDE.md into four audience-scoped pages

The guide had grown to 4,803 lines across four parts, mixing rationale,
consumer instructions, contributor reference, and troubleshooting. The
consolidation issue asked for consumer and contributor material to be
separated rather than interleaved; this promotes that separation from
parts within one file to pages.

  SCHEMA_GUIDE.md   1,743  using the schema (was Part I)
  AUTHORING.md      1,687  extending and authoring it (was Part II + templates)
  CONCEPTS.md         803  why the schema is built this way (was Part 0 + digressions)
  TROUBLESHOOTING.md  370  symptom-indexed errors and model gotchas (was section 12)

Guide sections 4-6 now delegate per-command reference to the package
READMEs under packages/, which are versioned with the code they document,
rather than restating it. SDK and CLI construction moved from Part II into
the guide as section 8 -- that is consumer work, not authoring.

Section 2.5 rewritten around the queries people run rather than an
explanation of JSON Schema: 454 lines to 194. The required-field
derivation, the tag model, and the example-file walkthrough moved to
CONCEPTS.md.

All `uv run python <<'PY'` heredocs converted to plain python blocks, which
also brings them under tests/test_documented_imports.py -- that raised the
checked-block count from 227 to 241.

REVIEW STATE: guide sections 1 through 2.6 have had a full pass. Sections
2.6-8, CONCEPTS.md, and AUTHORING.md are not yet reviewed as prose;
AUTHORING.md is a near-verbatim move of Part II. All code blocks are
parsed, import-checked, and enum-checked by the test suite, and all
internal links and anchors resolve.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Seth Fitzsimmons <seth@mojodna.net>

* expand README to cover the whole repo, record the tenets and doctrine

Signed-off-by: Dana Bauer <dana.bauer@gmail.com>

* docs: reconcile the docs with the Phase 3 publish pipeline

Phase 3 (#638) replaced publish-python-packages.yaml with
main-publish.yaml and release-publish.yaml, and moved the PySpark
codegen step out of the workflow into a package-owned
packages/<package>/scripts/prebuild.sh. Describe that convention and the
two publish targets (CodeArtifact for interim .postN builds, public PyPI
via Trusted Publishing for version bumps) instead of the workflow that
no longer exists.

Correct the README package count: packages/ holds thirteen packages, as
GLOSSARY.md and SCHEMA_GUIDE.md already said. extensions-operating-hours
has never been on main.

Run the markdown code blocks through ruff format, which make check has
been failing on independently of this rebase.

Signed-off-by: Seth Fitzsimmons <seth@mojodna.net>

---------

Signed-off-by: Dana Bauer <dana.bauer@gmail.com>
Signed-off-by: Seth Fitzsimmons <seth@mojodna.net>
Co-authored-by: Dana Bauer <dana.bauer@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

automation 🦾 Change/enhance automation of a process

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Devops] Branching Strategy - Phase 3

5 participants