Skip to content

Make NLB listener TCP idle timeout configurable and set 1800s for prod HTTPS - #261

Merged
ryanjjung merged 1 commit into
mainfrom
nlb-tcp-idle-timeout
Sep 25, 2026
Merged

ryanjjung merged 1 commit into
mainfrom
nlb-tcp-idle-timeout

Conversation

@Sancus

@Sancus Sancus commented Sep 25, 2026

Copy link
Copy Markdown
Member

What changed?

Adds an optional tcp_idle_timeout_seconds setting (service name → seconds, 60–6000) to public_load_balancer and private_load_balancers, passed to each service's aws.lb.Listener. Services not listed keep the AWS default of 350 s. Prod sets https: 1800.

Code and description were written by an AI agent (Cursor) at my direction.

Why?

An NLB stops tracking a TCP flow once it has been idle for the listener's idle timeout (350 s by default), without sending FIN or RST to either side. If the client closes the connection after that point, the NLB drops the client's FIN/RST and the target never receives it.

Stalwart does not close idle HTTP connections itself: its HTTP server has no idle/header read timeout and it does not enable SO_KEEPALIVE on accepted sockets. A connection whose close is dropped therefore stays ESTABLISHED on the node indefinitely and keeps one of the HTTPS listener's concurrent connection slots (8192 by default). When all slots are held, Stalwart refuses new HTTPS connections.

Clients that hold idle connections longer than 350 s before closing trigger this on every such connection; for example, Cloudflare closes idle origin connections after 900 s (docs). With an 1800 s idle timeout the NLB still tracks the flow when these clients close, so the close reaches Stalwart and the socket is released.

Limitations and Notes

  • Clients that disappear without closing (idle longer than 1800 s, then gone) still leave sockets open on the node. Covering those needs an idle timeout or TCP keepalive in Stalwart, or a keepalive-capable proxy in front of it.
  • The prod listener is already set to 1800 s through the AWS API. pulumi preview --stack prod shows tcpIdleTimeoutSeconds: 350 => 1800 on mailstrom-prod-pub-listener-https (from stored state) and no other change from this PR.

Applicable Issues

None.

@ryanjjung ryanjjung self-assigned this Sep 25, 2026
@ryanjjung
ryanjjung merged commit b39b6d8 into main Sep 25, 2026
1 check passed
@ryanjjung
ryanjjung deleted the nlb-tcp-idle-timeout branch September 25, 2026 16:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants