others-How to solve too many time_wait network connections in linux server?

Jan 9, 2022

1. Purpose

In this post, I would demonstrate how to solve too many time_wait network connections problem in linux server?

2. The problem and solution

2.1 The problem

netstat -n | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'  

TIME_WAIT 4356
CLOSE_WAIT 75
FIN_WAIT1 15
FIN_WAIT2 72
ESTABLISHED 429
CLOSING 1
LAST_ACK 3

2.2 The solution

2.2.1 What is TIME_WAIT status of TCP connection

This problem is caused by the incorrectly set network configurations, The TIME_WAIT state of TCP, also known as the 2MSL waiting state: When one end of TCP initiates an active shutdown (receives a FIN request), after the last ACK response is sent, that is, after the third handshake is completed, the fourth handshake ACK is sent After the package, it enters the TIME_WAIT state

2.2.2 How to solve too many TIME_WAIT problem?

Change your network configuration file ** /etc/sysctl.conf**, add following configurations:

net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_fin_timeout = 60
net.ipv4.ip_local_port_range = 10000 65000
net.ipv4.tcp_max_syn_backlog = 8192
net.ipv4.tcp_max_tw_buckets = 10000

When a TCP connection is closed, a delay of 2*MSL in TIME-WAIT state ties up the socket pair for 4 minutes (see Section 3.5 of [Postel81]. Applications built upon TCP that close one connection and open a new one (e.g., an FTP data transfer connection using Stream mode) must choose a new socket pair each time

According to Linux documentation, you should use the TCP_TW_REUSE flag to allow reusing sockets in TIME_WAIT state for new connections.

It seems to be a good option when dealing with a web server that have to handle many short TCP connections left in a TIME_WAIT state.

As described here, The TCP_TW_RECYCLE could cause some problems when using load balancers.

warning: When recycle is enabled, the server can’t distinguish new incoming connections from different clients behind the same NAT device.

Apply the changes:

sysctl -p

After apply the changes, we got this:

[[email protected] etc]# /opt/bin/print_tcp_by_status.sh
TIME_WAIT 2268
CLOSE_WAIT 21
FIN_WAIT1 8
FIN_WAIT2 70
ESTABLISHED 461
SYN_RECV 2
LAST_ACK 38
[[email protected] etc]# /opt/bin/print_tcp_by_status.sh
TIME_WAIT 2270
CLOSE_WAIT 22
FIN_WAIT1 8
FIN_WAIT2 69
ESTABLISHED 462
LAST_ACK 39
[[email protected] etc]# /opt/bin/print_tcp_by_status.sh
TIME_WAIT 2268
CLOSE_WAIT 24
FIN_WAIT1 6
FIN_WAIT2 84
ESTABLISHED 451
SYN_RECV 1
LAST_ACK 38
[[email protected] etc]# /opt/bin/print_tcp_by_status.sh
TIME_WAIT 60
CLOSE_WAIT 6
FIN_WAIT1 17
FIN_WAIT2 33
ESTABLISHED 426
SYN_RECV 1
LAST_ACK 5

3. Summary

In this post, I demonstrated how to solve the too-many-time_wait-problems in linux servers, the key point is to change your network settings to allow reuse/recycle time_wait connections. That’s it, thanks for your reading.