How many TCP connections can a Linux server support at most? How many TCP connections can be established on a Linux machine at most?
The concurrency problem that puzzles many people
I found that many students never fully understand a basic problem. How many network connections can a server support? I think it is necessary for me to send a separate article to talk about this problem.
Many students' first reaction to this question was 65535. The reason is: "I heard that the maximum number of ports is 65535, and the maximum number of long connections is 65535". Is that right? Others said, "It should be limited by the size of the quads in the TCP connection, which is more than 200 trillion!"
If you don't understand this problem thoroughly enough, let me tell you a story today!
A chat about server side concurrency
"TCP connection quads are Source IP address, source port, destination IP address and destination port 。 If any element changes, it represents a completely different connection. Take my Nginx for example. Its port is fixed at 80. In addition, my IP is also fixed, so the destination IP address and destination port are fixed. The remaining source IP address and source port are variable. So theoretically, my Nginx can build 32 power of 2 (number of ip) × 16 power of 2 (number of ports) connections 。 This is a big number of more than 200 trillion!! "
"Every time a process opens a file (everything under Linux, including sockets), it consumes a certain amount of memory resources. If someone who is not kind starts a process to create and open new files indefinitely, the server will crash. For security reasons, the Linux system limits the number of file descriptors that can be opened in multiple locations, including system level, user level, and process level. The meaning and modification of these three restrictions are as follows:“
-
System level: the maximum number that the current system can open, which can be modified through the fs.file-max parameter
-
User level: specify the maximum number that users can open, and modify/etc/security/limits.conf
-
Process level: the maximum number that a single process can open, which can be modified through the fs.nr_open parameter
"My receive buffer size can be configured and can be viewed through the sysctl command."
sysctl -a | grep rmem net.ipv4.tcp_rmem = 4096 87380 8388608 net.core.rmem_default = 212992 net.core.rmem_max = 8388608
The first value in "tcp_rmem" is the minimum number of bytes allocated for your TCP connection. The default value is 4K, and the maximum value is 8MB. That is to say, when you have data to send, I need to allocate at least 4K more memory for the corresponding socket, or even more. "
"The size of the TCP allocation send buffer is affected by the configuration of the parameter net.ipv4.tcp_wmem."
sysctl -a | grep wmem net.ipv4.tcp_wmem = 4096 65536 8388608 net.core.wmem_default = 212992 net.core.wmem_max = 8388608
The first value in "net. ipv4. tcp_wmem" is the minimum value of the transmit buffer, which is also 4K by default. Of course, if the data is large, the actual allocation of the cache will be larger than the default value. "
Record of millions of connections on the server
"What are you going to do? I remember that Linux has a limit on the maximum number of file objects, so if you want to complete this experiment, you need to increase the upper limit at the user level, system level, process level, etc. Our experiment goal is 100W, and we set it to 110W here, which is very important! Because we need to ensure that other basic commands such as ps, vi, etc. are available when doing experiments. “
The number of active connections has indeed reached 100W:
ss -n | grep ESTAB | wc -l one million and twenty-four
Currently, the total machine memory is 3.9GB, of which the kernel Slab occupies 3.2GB. MemFree and Buffers add up to just over 100 MB:
cat /proc/meminfo MemTotal: 3922956 kB MemFree: 96652 kB MemAvailable: 6448 kB Buffers: 44396 kB ...... Slab: 3241244KB kB
Through the slabtop command, you can see that there are 100W kernel objects in each of the four kernel objects: deny, flip, sock_inode_cache, and TCP:
epilogue
One of the business characteristics of the Internet backend is high concurrency However, the question of how many TCP connections a server can support seems to perplex many students. I hope that after today, you can step on this problem and rub it under your feet!
Reprinted on https://mp.weixin.qq.com/s/Lkyj42NtvqEj63DoCY5btQ