Skip to content

Fix lemming startup crash with 1000+ interfaces - #768

Open
kraney wants to merge 9 commits into
openconfig:mainfrom
kraney:fix-1500-intf-startup
Open

kraney wants to merge 9 commits into
openconfig:mainfrom
kraney:fix-1500-intf-startup

Conversation

@kraney

@kraney kraney commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

Problem

Lemming crashes on startup when the device has 1000 or more interfaces (e.g. a KNE topology with eth1..eth1500):

Failed to start lemming: failed to stop handler "inferface": reconciler "inferface" start errs: failed to setup ports: failed to create host interface "eth1000-internal": rpc error: code = Internal desc = Link not found

For each front-panel interface, the interface reconciler creates a hostif TAP named <name>-internal. For eth1000 and above, that name is 16+ characters. Linux interface names are limited to 15 characters (IFNAMSIZ - 1), so creating the TAP fails and the reconciler stops startup.

The fwdcontext: unable to send notification in context ... nil queue warning that also shows up in the logs isn't related. It's logged on every startup, because ports are created before anything subscribes to port state notifications.

Fix

Pick a hostif name that always fits:

  1. <name>-internal if it fits. This is unchanged for eth1..eth999, so existing topologies behave the same.
  2. Otherwise <name>-int (e.g. eth1000-int).
  3. Otherwise if<ifindex>-int. The ifindex is unique within the network namespace, so this name is unique too.

setupPorts now recognizes both suffixes when skipping existing hostif devices. Also:

  • Fixed a log.Info call that should be log.Infof.
  • Fixed two existing typos that codespell flags in the touched file.

Testing

  • Added unit tests for hostifName / isHostifName, covering the name length limit, uniqueness, and suffix detection.
  • Reproduced locally by running lemming with the dataplane enabled in an unprivileged network namespace with 1500 dummy ethN interfaces:
    • Before: crashes on eth1000-internal.
    • After: all 1500 interfaces get ports, hostifs, and router interfaces (501 use the -int form), and lemming initialization completes.

Hostif TAP interfaces were named "<name>-internal". For eth1000 and above this is 16+ characters, exceeding the Linux IFNAMSIZ limit (15 chars), so TAP creation failed with "Link not found" and the interface reconciler aborted startup.

Use "<name>-internal" when it fits, fall back to "<name>-int", and finally to "if<ifindex>-int" for very long names. Also fix a log.Info call that should be log.Infof.

Fixes b/540538596
@kraney
kraney requested a review from a team as a code owner October 21, 2026 19:40
@kraney
kraney requested review from bstoll and manizzzz October 26, 2026 20:41

@manizzzz manizzzz left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Kris, LGTM! Left a few non-blocking comments and follow-up ideas below.

Separate from this PR, we should probably clean up the error chain from the bug report in a follow-up (dataplane/server.go logs a reconciler start error as "failed to stop handler", the reconciler ID is "inferface", and the actual error comes back as "Link not found" instead of checking the name length up front in CreateHostif or tap.New).

Comment thread dataplane/dplanerc/interface.go Resolved
Comment thread dataplane/dplanerc/interface.go Resolved
Comment thread dataplane/dplanerc/interface.go Resolved
Comment thread dataplane/dplanerc/interface.go Resolved
setupPorts used the interface's position in the name-sorted link list as the port's hardware lane, but saiserver binds a port to the kernel interface eth<lane>. With more than ten ports eth10 sorts before eth2, so ports were bound to the wrong kernel interfaces. Parse the lane from the ethN name instead, falling back to the list index for other names.
@kraney

kraney commented Oct 29, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the review! Agreed on cleaning up the error chain (the misleading "failed to stop handler" log, the inferface reconciler ID, and checking the name length up front in CreateHostif/tap.New rather than surfacing "Link not found") — I'll pick that up in a separate PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants