Skip to content

Let CI give up on a dnf mirror that is slow or hangs - #22

Merged
stefan-azentrius merged 1 commit into
mainfrom
ci-dnf-limits
Nov 1, 2026
Merged

stefan-azentrius merged 1 commit into
mainfrom
ci-dnf-limits

Conversation

@stefan-azentrius

Copy link
Copy Markdown
Contributor

The first dnf install in an AlmaLinux container is sometimes very slow, and under -q it prints nothing until it is done. In the upgrade job of Image Update Monitor:

  • run 41252349101: upgrade (almalinux:9) spent 815 seconds in that one call (almost 14 minutes).
  • run 41260581451: upgrade (almalinux:9) sat silent in it until the job's 20-minute limit cancelled it. The rerun took 649 seconds there, and upgrade (almalinux:10) in the same run 391.

The same steps normally take 11 to 80 seconds, and Rocky Linux stayed under a minute in the same runs. The logs don't show which part was slow, the repository data or the packages.

tools/dnf.sh runs dnf as root in a CI container:

  • It writes minrate=100000 and timeout=20 under [main] in /etc/dnf/dnf.conf, so dnf leaves a mirror that sends less than 100 kB/s for 20 seconds and takes the next one. dnf's defaults are 1 kB/s and 30 seconds. The settings stay in place for every later dnf in that container.
  • It runs dnf makecache first, without -q, so the log shows each repository's size and speed. Then it runs the command.
  • Each of the two gets 180 seconds per try and up to three tries, 15 seconds apart. Each failed try gets a warning (exit code, or no end within the limit); after the third the script fails with an error.
  • sh /dnf.sh makecache fetches only the repository data, for a dnf call that has to stay as it is (a refused install, whose failure is the expected result).

It is the same file in all seven repositories that run dnf in CI containers (blob ad912ed, executable): Cron Jobs, Image Update Monitor, Status Board, Host Settings, Update Gaps, Security Monitor and packages.

Now in .github/workflows/test.yml:

  • The repository end-to-end step installs azentrius-repository through /dnf.sh on Fedora 43, Rocky Linux 9 and 10 and AlmaLinux 9 and 10.
  • The README names the script next to tools/apt-update.sh.

Unchanged: dnf remove, which works from the repository data that is already there.

Checks:

  • A smoke test with a fake dnf under dash: the settings land right after [main], and only once; makecache runs before the command; two failed tries and then a working one give exit 0; three failed tries give exit 1 and the error; a hung try ends after the limit with exit 124; without [main] the script stops before any dnf; without arguments it exits 64.
  • The workflows parse with no duplicate keys, and every changed run block and container script passes bash -n and dash -n.
  • CI on this pull request runs every changed step.

tools/dnf.sh runs dnf in a CI container: the repository data first (dnf makecache, without
-q), then the command, each with 180 seconds per try and up to three tries, 15 seconds apart.
It sets minrate=100000 and timeout=20 in /etc/dnf/dnf.conf, so dnf leaves a mirror slower
than 100 kB/s. On AlmaLinux the first dnf install of a job once took almost 14 minutes and
once ran until the job's 20-minute limit.

The repository end-to-end step installs azentrius-repository through it on Fedora, Rocky Linux
and AlmaLinux.
@stefan-azentrius
stefan-azentrius merged commit f585451 into main Nov 1, 2026
@stefan-azentrius
stefan-azentrius deleted the ci-dnf-limits branch November 1, 2026 09:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant