Skip to content

ci: bound every job, and stop apt waiting forever on a stalled mirror - #4

Merged
Ryanmello07 merged 1 commit into
urnetwork:mainfrom
Ryanmello07:upstream/ci-job-timeouts
Aug 21, 2026
Merged

ci: bound every job, and stop apt waiting forever on a stalled mirror#4
Ryanmello07 merged 1 commit into
urnetwork:mainfrom
Ryanmello07:upstream/ci-job-timeouts

Conversation

@Ryanmello07

@Ryanmello07 Ryanmello07 commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Two runs of this workflow burned to GitHub's 6-hour default without producing a
result. Neither failed — both hung inside apt, which by default waits
indefinitely on a mirror connection that has gone quiet. One lost an hour on
apt-get install zip; the other lost six hours split across dependency
installs. Different packages, same cause, so this is fixed at the transport
rather than at each call site.

Two layers, because either alone leaves a hole:

  1. timeout-minutes on every job. Without it, a stall is only noticed six
    hours later, by which point the run is worthless and the queue has been held.
    The values are roughly 3× each job's observed wall time, so a slow-but-healthy
    run is never killed.

  2. UR_APT_OPTS in the workflow env, threaded through every apt invocation:

    • Acquire::Retries=3 — covers the transient 503s the Azure mirrors serve
      under load.
    • Acquire::http::Timeout=30 / https::Timeout=30 — the options that
      actually convert a silent stall into a fast, legible failure.
    • Acquire::http::No-Cache=true — keeps a proxy from serving a stale index
      that then 404s on download.

No dependency versions change, and no step is added or removed. This is entirely
about how the existing steps fail.

Why this is split out on its own

It is the only change in this series that touches nothing but CI mechanics: no
source file, no packaging file, no product decision. It is independent of the
app-ID rename that PRs 2–6 build on, and it is worth landing before them so
their runs fail in minutes instead of hours.

Note for whoever merges second

PR 4 (upstream/appstream-metainfo) also edits the gui job — it appends
appstream to the same apt package list and adds a validation step at the end.
The two were written against main independently and merge cleanly in testing,
but if git ever disagrees, the resolution is "keep both": this PR's
$UR_APT_OPTS on the apt-get lines, and PR 4's extra package plus extra step.

Two runs of this workflow burned to GitHub's 6-hour default timeout without
producing a result. Neither failed: both hung inside apt, which by default
waits indefinitely on a mirror connection that has gone quiet. One lost an
hour on `apt-get install zip`, the other six hours split across dependency
installs. Different packages, same cause, so the fix is applied at the
transport rather than at each call site.

Two layers, because either alone leaves a hole:

1. `timeout-minutes` on every job. Without it a stall is only noticed six
   hours later, by which point the run is worthless and the queue has been
   held. The values are roughly 3x the observed wall time of each job, so a
   slow-but-healthy run is never killed.

2. `UR_APT_OPTS` in the workflow env, threaded through every apt invocation:
   `Acquire::Retries=3` covers the transient 503s the Azure mirrors serve
   under load, and the http/https `Timeout=30` options are what actually
   convert a silent stall into a fast, legible failure. `No-Cache=true`
   keeps a proxy from serving a stale index that then 404s on download.

No dependency versions change and no step is added or removed; this is
entirely about how the existing steps fail.
@Ryanmello07 Ryanmello07 changed the title PR 1 — upstream/ci-job-timeouts ci: bound every job, and stop apt waiting forever on a stalled mirror Aug 21, 2026
@Ryanmello07
Ryanmello07 marked this pull request as ready for review August 21, 2026 15:39
@Ryanmello07
Ryanmello07 merged commit fc44b1e into urnetwork:main Aug 21, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant