Skip to content

Fix CI: golangci-lint tmpfs quota, KNE topology deploy failures, presubmit apt lock - #775

Draft
bstoll wants to merge 8 commits into
openconfig:mainfrom
bstoll:fix-lint-gotmpdir
Draft

bstoll wants to merge 8 commits into
openconfig:mainfrom
bstoll:fix-lint-gotmpdir

Conversation

@bstoll

@bstoll bstoll commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

This PR fixes three separate CI failures on main and open PRs.

1. golangci-lint: disk quota exceeded on Ubuntu 26.04 runners

Failure: golangci-lint on main (job 122443639602)

sysrib/connected.go:1: : write /tmp/go-build3334798570/b1714/importcfg: disk quota exceeded (typecheck)

Cause: ubuntu-latest is now Ubuntu 26.04, which mounts /tmp as a RAM-backed tmpfs with a user quota (tmpfs 7.9G ... usrquota). golangci-lint runs go builds (including cgo) whose work directory defaults to /tmp, and those builds use up the quota. The root disk still has ~88G free.

Fix: Set GOTMPDIR: ${{ runner.temp }} on the golangci-lint step. This puts the Go build work dir on the runner's main disk.

2. Presubmit (Cloud Build): KNE topology deploy failures, then leaked topology state

Failures:

The same failures show up on the other open PRs (#777, #779, #780).

Cause: These are two failures, and the first one causes the second:

  1. Several integration tests deploy topologies in parallel. The KNE topology Create watchers were not scoped to the topology's namespace, so a test could see pods from another topology and fail with:
    failed to create topology: Pod lemming failed to deploy
    Deploy() failed: ... node "otg" ... could not get ixia CRD: context canceled
    
    This is fixed upstream in topo, pods, events: scope Create watchers to the topology namespace kne#797 (released in kne v0.3.4). That code lives in the github.com/openconfig/kne/topo library, which internal/binding compiles into the test binaries, so it has to be picked up through go.mod. Rebuilding the KNE VM image does not include it.
  2. When Create failed, LemmingBind left the partially created topology behind and didn't mark it as created, so nothing ever deleted it. The Bazel retry, and every later test sharing that topology (e.g. onedut-oneotg), reused the same lemming instance. vlan_subintf leaves subinterface 1 configured on eth1, so the later tests that replace eth1 then fail on every attempt:
    invalid SetRequest: field name Subinterface value 1 (uint32 ptr) schema path
    /root/network-instances/network-instance/interfaces/interface/state/subinterface has leafref path
    /interfaces/oc-if:interface[oc-if:name=current()/../interface]/subinterfaces/subinterface/index not equal to any target nodes
    

Fix:

  • Bump github.com/openconfig/kne v0.3.2 → v0.3.4 (includes topo, pods, events: scope Create watchers to the topology namespace kne#797). The other module bumps in go.mod/go.sum are minimum versions that kne v0.3.4 requires. repositories.bzl was regenerated with make buildfile (gazelle update-repos -prune), which also drops repos kne no longer depends on.
  • internal/binding: if tm.Create fails, delete the partially created topology (tm.Delete waits for the namespace to be removed) before returning the error. A retry then starts from a clean topology instead of inheriting broken or leftover state.

3. Presubmit (Cloud Build): apt-get dpkg lock race on the fresh VM

Failure: Presubmit (build 8bdfed18)

E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 1746 (unattended-upgr)

Cause: unattended-upgrades often runs right after the presubmit VM boots and holds the dpkg lock, so apt-get install in cloudbuild/presubmit.sh fails right away.

Fix: Run apt-get with -o DPkg::Lock::Timeout=600 so it waits for the lock instead of failing.

Touching cloudbuild/presubmit.sh makes super-linter lint the whole file, so this also fixes the findings that were already in it: shfmt formatting, shellcheck SC2329 (dumpinfo is only called through trap), and a codespell typo.

Ubuntu 26.04 GitHub runners mount /tmp as a quota-limited tmpfs. golangci-lint
compiles packages (including cgo) through the go command, whose work directory
defaults to /tmp, and now fails with 'disk quota exceeded'. Point GOTMPDIR at
the runner temp directory, which lives on the main disk.
@bstoll
bstoll marked this pull request as ready for review October 28, 2026 16:54
@bstoll
bstoll requested a review from a team as a code owner October 28, 2026 16:54
kne v0.3.4 includes openconfig/kne#797, which scopes the topology Create
watchers to the topology namespace. Without it, integration tests that
deploy topologies in parallel can see pods from other topologies and fail
with 'Pod lemming failed to deploy'. The topo package is compiled into the
integration test binaries via internal/binding, so it must be picked up
through go.mod.

Other module bumps are the minimum versions required by kne v0.3.4.
repositories.bzl regenerated with 'make buildfile'.
Previously a failed tm.Create left the topology in place without marking
it as created, so it was never deleted. Bazel retries and later tests
sharing the topology then reused the same, already-configured lemming
instance (e.g. vlan_subintf's subinterface 1 on eth1), causing leafref
validation failures on unrelated tests. Delete the topology on failure so
retries start from a clean deployment.
unattended-upgrades frequently holds the dpkg lock right after the
presubmit VM boots, causing apt-get install to fail immediately. Wait up
to 10 minutes for the lock instead.
@bstoll
bstoll marked this pull request as draft October 30, 2026 17:04
@bstoll bstoll changed the title Fix golangci-lint 'disk quota exceeded' on Ubuntu 26.04 runners Fix CI: golangci-lint tmpfs quota, KNE topology deploy failures, presubmit apt lock Oct 30, 2026
@gavinz-ship-it

Copy link
Copy Markdown
Contributor

For the version upgrade, can you merge the following PR instead? and leave this PR for other changes.

#748

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants