test: fix some flakes / wait for conmon to exit

[kolyshkin]

attach: container finishes immediately without --stdin is flaky: it waits for
the container to reach the stopped state and then reads the log, but the
container being stopped does not mean conmon is done writing the log, so the
test can read an empty one.

Seen in runc CI (the runc validate / conmon job runs this suite against a
runc build):

not ok 92 attach: container finishes immediately without --stdin
#   `assert "${output}" =~ "Container stopped!"  "'Container stopped!' found in the log"' failed
#|     FAIL: 'Container stopped!' found in the log
#| expected: =~ Container stopped\!
#|   actual:    ''

A rerun of the exact same commit passed.

conmon having exited does mean the log is complete, so this waits for that.

Waiting for a specific conmon needs its pid, and $CONMON_PID_FILE is a single
path per test that every conmon a test starts overwrites, so the first commit
gives each conmon started by start_conmon_with_default_args its own pidfile
and exposes the pid as $CONMON_PID. The rest builds on that: fix the flaky
test, then convert the remaining "wait for stopped, then read a log" places,
which have not been seen to fail but share the race.

Testing: the full suite passes locally with -j $(nproc). Note that I could
not reproduce the race itself locally -- instrumenting the new helper showed
conmon had always already exited by the time it looked -- so this is verified
as "correct barrier, no regressions" rather than "observed to fix the flake".

The last commit fixes an unrelated but similar thing spotted on the way:
04-runtime.bats did wait $CONMON_PID_FILE 2>/dev/null || true, i.e. it
waited on a filename and swallowed the error, so it waited for nothing.

---

Update: rebased onto main, and the barrier is now in run_conmon_with_default_args itself (amended into "test: wait for conmon in the remaining log checks" rather than added as a separate commit).

That helper is the most common instance of the pattern -- 13 call sites across 02-ctr-logs.bats, 04-runtime.bats and 10-ctrl.bats, every one of which reads a log right after -- so one barrier there covers all of them.

It also turns out this one is not merely theoretical: the same race fails the coverage job, where conmon is built with --coverage and is thus slow enough to lose it more often. Seen on an unrelated PR's run:

not ok 45 ctr logs: k8s partial message
#   `assert "${output}" =~ "stdout P"' failed
#| expected: =~ stdout P
#|   actual:    ''

Same signature as the 07-attach.bats one: the runtime reports stopped, the log is read, and it is still empty. So this PR fixes two observed flakes, not one.

Re-tested after the rebase: 02-ctr-logs, 04-runtime, 07-attach, 08-exec and 10-ctrl all pass locally.

---

Update 2: four more commits, all about CI not hanging for hours.

Today's runs kept getting stuck -- one job sat for 2.5 hours before being cancelled -- and it was never the same place twice: hack/github-actions-setup hung for 49 minutes on one run, the test step for 35 minutes on another. Both consist of network access with no timeout on it, and the GitHub Actions default job timeout is 6 hours, so nothing ever gave up.

5. ci: set job timeouts -- timeout-minutes on the jobs (conmon 20, cri-o 90, coverage 20, lint 10), a few times their normal runtime. A hang now fails in minutes instead of squatting a runner for the rest of the day.

6. ci: time out and retry network access in the setup script -- the script clones five repositories, downloads a tarball and a repo key, and runs apt. git has no timeout at all; curl has no total timeout and no retries; wget has a 900 second read timeout and 20 tries by default; apt retries nothing. All of them get a bound now, the clones via GIT_HTTP_LOW_SPEED_LIMIT/GIT_HTTP_LOW_SPEED_TIME.

7. test: time out the image pull -- setup_suite pulls the image the test rootfs comes from. It is the only network access in the suite, and a hang there stops everything before a single test runs, silently: bats shows setup_suite output only once it completes.

8. test: bump the wait_for_runtime_status timeout -- 5 seconds turned out to be too little on a loaded runner:

not ok 51 runtime: container execution with multiple log drivers
not ok 101 ctrl: resize the terminal, negative width and height
#| FAIL: timed out waiting for 'stopped' from conmon-test-...

The helper polls, so a longer limit costs nothing when things are healthy; 30 seconds now.

Commits 1-4 fix races (read a log before conmon wrote it); 5-8 stop CI hanging on stalled I/O. Happy to split the latter into their own PR if that reads better.

[kolyshkin]

This is the order in which we need to merge test PRs: #673 (this one, fixes conmon job) → #674 (fixes cri-o jobs) → #670 (fixes meson which was recently broken again).

[kolyshkin]

OK this one is ready but I'll wait for review from @jnovy first