Skip to content

ci: make development release uploads resumable - #103

Merged
Pedro Henrique Penna (ppenna) merged 1 commit into
devfrom
feat-ci-reliability
Sep 18, 2026
Merged

Pedro Henrique Penna (ppenna) merged 1 commit into
devfrom
feat-ci-reliability

Conversation

@ppenna

@ppenna Pedro Henrique Penna (ppenna) commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Summary

  • create or reuse the development release as an explicit draft before attaching assets
  • upload and verify each platform package independently, preserving valid assets across reruns
  • bound uploads to three-minute attempts with transient-failure retries, exponential backoff, and post-error release queries
  • replace only a specific mismatched asset and publish only after the exact three-package set verifies
  • increase the release job timeout from 10 to 90 minutes so the complete bounded retry and partial-asset cleanup sequence has artifact-handling headroom
  • run the Python publisher unbuffered so package-level progress remains visible during slow uploads and cancellation
  • add focused regression coverage for retry success and exhaustion, partial draft resume, mismatched assets, server-side completion, and publication gating

Root cause

CI run 35270530890, attempt 2, at head 3644cbc18c57b900c61bdcd0ce8b6a64586f7b4b spent the full 10-minute job budget inside one multi-asset gh release create operation and was canceled on September 17, 2026. The command left a partial draft release. The previous rerun path used a single gh release upload --clobber, which retransferred valid assets, could delete completed work before a replacement succeeded, and never published a draft left by a failed create.

The new publisher resumes that draft safely, identifies the individual package that failed, and keeps the release unpublished until all assets are verified. Each upload attempt is capped at three minutes, while the 90-minute job timeout covers the configured upload, verification, and partial-asset cleanup budget across all three assets with artifact-handling headroom. Unbuffered Python output ensures attempt and elapsed-time messages are emitted promptly.

Validation

  • python3 scripts/nvx.py verify
  • python3 -m unittest scripts/test_development_release.py -v
  • full Linux command sequence from .github/actions/validate-nvx/action.yml
  • changed workflow and composite-action YAML parsed successfully
  • live read-only parsing of the incomplete draft release and its asset digests

The existing same-run workflow artifact names and producer/consumer dependencies are unchanged; no github.run_attempt scoping was added.

Fixes #74

Copilot AI balanced review requested due to automatic review settings September 17, 2026 23:47

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The configured timeout cannot cover the retry budget, and buffered output can hide upload progress during cancellation.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced (auto)
Findings: 1 Medium severity · 1 Low severity

Note

Copilot is running an experiment and ran this review at Balanced.

Open (2)
What changed in this PR

Makes development release publishing resumable and resilient to transient upload failures.

Changes:

  • Adds per-asset verification, retries, backoff, and draft publication gating.
  • Replaces shell publishing with a tested Python implementation.
  • Extends CI validation and release timeout.
File Description
scripts/​nvx_tools/​development_release.py Implements resumable release publishing.
scripts/​publish_development_release.py Adds the CI entry point.
scripts/​test_development_release.py Adds focused regression tests.
.github/​actions/​publish-development-release/​action.yml Invokes the new publisher.
.github/​actions/​validate-nvx/​action.yml Runs the new tests.
.github/​workflows/​ci.yml Increases the release-job timeout.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread .github/workflows/ci.yml Outdated
Comment thread .github/actions/publish-development-release/action.yml Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

The production release path depends on external GitHub API and CLI state transitions that require final human validation.

Review effort: Balanced (auto)
Findings: None

Note

Copilot is running an experiment and ran this review at Balanced.

Resolved since last review (2)

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Production release mutation, deletion, retries, and publication involve external state transitions that warrant final human review.

Review effort: Balanced (auto)
Findings: None

Note

Copilot is running an experiment and ran this review at Balanced.

Create or reuse the development release as an explicit draft before
transferring assets. Upload each platform package independently, verify
its remote size and SHA-256 when available, and preserve already-valid
assets across reruns.

Bound every upload attempt to three minutes and retry transient GitHub
5xx, timeout, connection, and duplicate-asset failures with exponential
backoff. Re-query the release after every upload result so a request that
completed server-side is accepted without retransmission, while only a
specific mismatched asset is removed before retrying.

Require the exact three-package asset set before publishing the draft as
a prerelease, and leave the release unpublished on any verification or
retry failure. Allow 90 minutes for the release job so the configured
upload, verification, and partial-asset cleanup budget can complete with
artifact-handling headroom, and run Python unbuffered so package-level
progress remains visible during slow uploads and cancellation.

Keep release metadata parsing fully type-safe under the pinned Pyright
configuration without redundant casts, and format the publisher with the
pinned Ruff version. Add focused regression coverage for retry success
and exhaustion, partial draft resume, mismatched assets, server-side
completion, and publication gating.

Validation:
- python3 scripts/nvx.py verify
- pinned Ruff check and format checks for scripts
- pinned Pyright checks for Linux and Windows
- python3 -m unittest scripts/test_development_release.py -v
- full Linux validate-nvx command sequence
- changed workflow and composite-action YAML parsed successfully

Refs #74

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: fdacb24e-53c2-4118-9196-d4c194a33187

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The implementation satisfies the stated recovery and publication requirements with focused regression coverage.

Review effort: Balanced
Findings: None

@ppenna
Pedro Henrique Penna (ppenna) merged commit f58f423 into dev Sep 18, 2026
20 checks passed
@ppenna
Pedro Henrique Penna (ppenna) deleted the feat-ci-reliability branch September 18, 2026 22:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make development release publishing resilient to transient asset upload failures

2 participants