mgmtd panics on peer disconnect - peer_addr().unwrap() in Stream::addr()

[scaleoutsean]

• 8.4.1 x64 on RHEL/Rocky 9.8
• Mgmt VM on TCP network, MTU 1500
• Two bare metal servers with RDMA (storage/BeeGFS) and TCP (Mgmt link), MTU 1500, 0 hopes to Mgmt IP
• One bare metal client with RDMA (BeeGFS) and TCP, MTU 1500, 0 hops to Mgmt IP

In summary, the network seems fine, system time is correct, no complex routing is in place.
Metadata, storage and clients all use the TCP network to connect to management service.

beegfs health check reports erratic results. Sometimes all storage services are Offline, minutes later, they're all Online, but all metadata are now Offline, then MD go Possibly-offline, etc.

Mgmt server seems to be the problem. It appears to be working well...

[2026-09-15T15:21:17Z INFO ] BeeGFS version: v8.4.1
[2026-09-15T15:21:17Z INFO ] Loaded config file from "/etc/beegfs/beegfs-mgmtd.toml"
[2026-09-15T15:21:17Z INFO ] IPv6 is disabled by the configuration, falling back to IPv4 sockets
[2026-09-15T15:21:17Z INFO ] Successfully initialized certificate verification library.
[2026-09-15T15:21:17Z INFO ] Opened database at "/var/lib/beegfs/mgmtd.sqlite"
[2026-09-15T15:21:17Z INFO ] Successfully loaded license certificate: TMP-433214891
[2026-09-15T15:21:17Z INFO ] Listening for BeeGFS connections on 0.0.0.0:8008
[2026-09-15T15:21:17Z INFO ] Receiving BeeGFS datagrams on 0.0.0.0:8008
[2026-09-15T15:21:17Z WARN ] gRPC server running with TLS disabled
[2026-09-15T15:21:17Z INFO ] Serving gRPC requests on 0.0.0.0:8010
[2026-09-15T15:21:17Z WARN ] Waiting for shutdown signal ...

Minutes later...

[2026-09-15T15:36:49Z DEBUG] Updated node c11B75-6AA92E66-se-sv-r670-04[client:5, uid:25] node
[2026-09-15T15:36:51Z DEBUG] Updated target states for Meta targets [33], 0 consistency states and 1 reachability states changed
[2026-09-15T15:36:51Z DEBUG] Updated Meta target info
[2026-09-15T15:36:52Z DEBUG] Accepted incoming stream from 10.12.222.62:57295
[2026-09-15T15:36:52Z DEBUG] Updated target states for Storage targets [103, 104, 107, 108], 0 consistency states and 4 reachability states changed
[2026-09-15T15:36:52Z DEBUG] Updated Storage target info
[2026-09-15T15:36:52Z DEBUG] Accepted incoming stream from 10.12.222.61:47785
[2026-09-15T15:36:52Z DEBUG] Updated target states for Storage targets [100, 102, 105, 106], 0 consistency states and 4 reachability states changed
[2026-09-15T15:36:52Z DEBUG] Updated Storage target info
[2026-09-15T15:36:54Z DEBUG] Updated target states for Meta targets [33], 0 consistency states and 0 reachability states changed
[2026-09-15T15:37:14Z ERROR] PANIC: panicked at shared/src/conn/stream.rs:124:54:
    called `Result::unwrap()` on an `Err` value: Os { code: 107, kind: NotConnected, message: "Transport endpoint is not connected" }

    BACKTRACE:
       0: <unknown>
       1: <unknown>
       2: <unknown>
       3: <unknown>
       4: <unknown>
       5: <unknown>
       6: <unknown>
       7: <unknown>
       8: <unknown>
       9: <unknown>
      10: <unknown>
      11: <unknown>
      12: <unknown>
      13: <unknown>
      14: <unknown>
      15: <unknown>
      16: <unknown>
      17: <unknown>
      18: <unknown>
      19: <unknown>
      20: <unknown>
      21: <unknown>
      22: <unknown>
      23: start_thread
      24: __clone3

[2026-09-15T15:37:14Z ERROR] PANIC: panicked at shared/src/conn/stream.rs:124:54:
    called `Result::unwrap()` on an `Err` value: Os { code: 107, kind: NotConnected, message: "Transport endpoint is not connected" }

    BACKTRACE:
       0: <unknown>
       1: <unknown>
       2: <unknown>
       3: <unknown>
       4: <unknown>
       5: <unknown>
       6: <unknown>
       7: <unknown>
       8: <unknown>
       9: <unknown>
      10: <unknown>
      11: <unknown>
      12: <unknown>
      13: <unknown>
      14: <unknown>
      15: <unknown>
      16: <unknown>
      17: <unknown>
      18: <unknown>
      19: <unknown>
      20: <unknown>
      21: <unknown>
      22: <unknown>
      23: start_thread
      24: __clone3

[2026-09-15T15:37:14Z ERROR] PANIC: panicked at shared/src/conn/stream.rs:124:54:
    called `Result::unwrap()` on an `Err` value: Os { code: 107, kind: NotConnected, message: "Transport endpoint is not connected" }

    BACKTRACE:
       0: <unknown>
       1: <unknown>
       2: <unknown>
       3: <unknown>
       4: <unknown>
       5: <unknown>
       6: <unknown>
       7: <unknown>
       8: <unknown>
       9: <unknown>
      10: <unknown>
      11: <unknown>
      12: <unknown>
      13: <unknown>
      14: <unknown>
      15: <unknown>
      16: <unknown>
      17: <unknown>
      18: <unknown>
      19: <unknown>
      20: <unknown>
      21: <unknown>
      22: <unknown>
      23: start_thread
      24: __clone3

[2026-09-15T15:37:17Z DEBUG] Running switchover check
[2026-09-15T15:37:21Z DEBUG] Updated Meta target info
[2026-09-15T15:37:24Z DEBUG] Updated target states for Meta targets [33], 0 consistency states and 0 reachability states changed
[2026-09-15T15:37:35Z ERROR] PANIC: panicked at shared/src/conn/stream.rs:124:54:

The panic unwinds the connection-handling task but the process survives (PID is still around, "it's fine"), so mgmtd appears healthy. The in-flight request from service is never answered ("soft disconnect" reported), so the peer blocks for the full CONN_LONG_TIMEOUT (600000 ms) before retrying.

  mgmtd 15:36:52Z  Updated target states for Storage targets [...], 4 reachability states changed
  mgmtd 15:37:14Z  PANIC ...
  meta  15:26:47Z  Communication error: Receive timed out from: <mgmtd>:8008
  meta  15:28:18Z  Soft disconnect from <mgmtd>:52120

That goes on and off. At times everything may be Online but it's not long enough to do anything on the filesystem (which can be mounted, but nothing more than that).

The source (./beegfs-rust/shared/src/conn/stream.rs) already carries a // TODO unwrap ? comment at that line.

The debuginfo package for Mgmt is useless (stripped, etc.) and some BeeGFS debug RPMs are missing, making self-help harder.

[rustybee42]

Hey,
thanks for reporting this.

What likely happens here is that the connection breaks/gets disconnected and management tries to log that on the next operation, which includes the peer address. The peer address is resolved internally via getpeeraddr() every time, which doesn't work anymore on broken connections, causing the panic. Which should not happen. I just made a fix by caching the address on connect/accept (and the TODO is also gone).

The debug info is actually broken, the binary is missing the pointer to the file with the debug symbols. Should also be fixed now.

Both fixes should be in the next release (8.4.2 or 8.5).

[scaleoutsean]

Thanks!
I think gdb also reported some missing or mismatched symbols related to libbeegfs-ib-debuginfo and libbeegfs-license, but at that point it was clear the debug was missing, so I didn't look at that message carefully.
We moved the mgmt server from a hypervisor to a bare metal server which helped, but it's good to know this has been addressed.