The way to solve bugs - nginx 502 bad gateway

Undamaged Huguang al 2020-11-07 20:15:13
way solve bugs nginx bad

Explain Bug The way -Nginx 502 Bad Gateway


The fact proved that , have read Linux Kernel source code does have great benefits , Especially when dealing with problems . When you see the moment when you report an error , We can put the phenomenon / reason / And solutions flash in your head . Even the phenomenon of some corners can quickly reflect why . I've read a few Linux TCP Protocol stack source code , There is a very smooth feeling in solving the following problem .

Bug The scene

First , This problem is not hard to solve , But the phenomenon caused by this problem is quite interesting . First describe the phenomenon ,
The author wants to study the self-study of dubbo Pressure test of protocol tunnel gateway ( The gateway design is also interesting , Ready to put it in the blog later ). Let's first look at the topology of pressure measurement :

In order to pressure test the author gateway The single machine performance of , Only one gateway is reserved at each end , namely gateway1 and gateway2. Press to a certain extent and start to report errors , Causing the pressure test to stop . It's natural to think of , The gateway can't handle it .

The gateway situation

Go to Gateway2 I took a look at it on my machine , There is no mistake . and Gateway1 There are a lot of 502 Report errors .502 yes Bad Gateway,Nginx The classic report error of , The first thing that comes to mind Gateway2 Overburdened by Nginx stay Upstream To kick out .

that , Just take a look Gateway2 The load condition of the load will be , Check the surveillance , Find out Gateway2 stay 4 nucleus 8G There's only one core on the machine , I can't see the bottleneck at all , Is it IO There is a problem ? Looked at the small poor network card traffic to eliminate this conjecture .

Nginx Location machine CPU The utilization rate is close to 100%

Now , Find an interesting phenomenon ,Nginx It's really full CPU!

Pressure test again , Go to Nginx On the machine top For a moment , Find out Nginx Of 4 individual Worker Each occupies a nuclear handle CPU Full -_-!

what , It's said to be powerful Nginx So weak , Say good event driven \epoll edge-triggered \ pure C It's made ? It must be in the wrong posture !

Get rid of Nginx Direct communication has no pressure

Since the guess is Nginx Bottleneck , Just put Nginx Get rid of it .Gateway1 and Gateway2 Direct connection , Pressure measurement TPS It soared inside , and Gateway2 Of CPU I'll eat at most 2 A nuclear , No pressure .

Go to Nginx Take a look at the log

because Nginx Machine permissions are not in the hands of the author , So I didn't pay attention to the log at first , Now contact the corresponding operation and maintenance to have a look . stay accesslog A lot of 502 Report errors , Is, indeed, Nginx Of . Look at the error log again , A large number of

Cannot assign requested address

Because I have read TCP Source code , In a flash, it's like , Is the port number exhausted ! because Nginx upstream Back end Backend The default is short connection , So when a lot of request traffic comes in, a lot of TIME_WAIT The connection of .

And these TIME_WAIT It takes up the port number , And basically 1 It takes about minutes to be Kernel Recycling .

cat /proc/sys/net/ipv4/ip_local_port_range
32768 61000

in other words , In a minute 28232(61000-32768) individual TIME_WAIT Of socket Will cause the port number to run out , That is to say 470.5TPS(28232/60), It's just a pressure measurement that is easy to reach . In fact, the limit is Client Terminal ,Server There is no such restriction on the end , because Server There is only one port number 8080 Such a famous port number . And in the
upstream in Nginx It's about Client, and Gateway2 It's about playing Nginx

Why? Nginx Of CPU yes 100%

And the author soon figured out Nginx Why full of machines CPU, The problem is the search process of port number .

Let's take a look at the most performance consuming function :

int __inet_hash_connect(...)
// Be careful , This side is static Variable
static u32 hint;
// hint Help not to follow 0 Begin your search , Instead, search from the next port number to be assigned
u32 offset = hint + port_offset;
inet_get_local_port_range(&low, &high);
// here remaining Namely 61000 - 32768
remaining = (high - low) + 1
for (i = 1; i <= remaining; i++) {
port = low + (i + offset) % remaining;
/* port Occupied or not check */
goto ok;
hint += i;

Look at the code above , If there is no port number available , Cycle is required remaining Times before the port number is exhausted , That is to say 28232 Time . And if it's normal , Because there is hint The existence of , So each search starts with the next port number to be assigned , Search in single digits to find the port number . As shown in the figure below :

So when the port number runs out ,Nginx Of Worker The process is immersed in the above for You can't get out of the loop , hold CPU Full .

Why? Gateway1 call Nginx No problem

It's simple , Because the writer is in Gateway1 call Nginx When you set Keepalived, So it's a long connection , There is no limit to the exhaustion of the port number .

Nginx If there are more than one machine in the back

Because it's the port number search that leads to CPU 100%, And whenever there is an available port number , because hint Why , The number of searches might be 1 and 28232 The difference between .

Because the port number limit is for a particular remote end server:port Of .
therefore , as long as Nginx Of Backend There are multiple machines , Even multiple different port numbers on the same machine , As long as it doesn't go beyond the critical point ,Nginx There won't be any pressure .

Increase the port number range

The more brainless solution, of course, is to increase the port number range , So you can resist more TIME_WAIT. At the same time tcp_max_tw_bucket The small ,tcp_max_tw_bucket yes kernel Most of them exist in TIME_WAIT Number , as long as port Range - tcp_max_tw_bucket Greater than a certain value , Then there will always be port Ports available , In this way, we can avoid further breakdown of the critical point when the critical value is increased again .

cat /proc/sys/net/ipv4/ip_local_port_range
22768 61000
cat /proc/sys/net/ipv4/tcp_max_tw_buckets

Turn on tcp_tw_reuse

This problem Linux In fact, there is a solution for a long time , That's it tcp_tw_reuse This parameter .

echo '1' > /proc/sys/net/ipv4/tcp_tw_reuse

in fact TIME_WAIT The reason is that the recovery time needs to be 1min, This 1min It's actually TCP As stipulated in the agreement 2MSL Time , and Linux It's fixed to be 1min.

#define TCP_TIMEWAIT_LEN (60*HZ) /* how long to wait to destroy TIME-WAIT
* state, about 60 seconds */

2MSL The reason is to exclude the remaining packets on the network for the new same quintuple Socket An impact , That is to say 2MSL(1min) There is a risk in reusing the quintuple within . To solve this problem ,Linux A series of measures have been taken to prevent this situation , So that in most cases 1s Within TIME_WAIT Can be reused . Here's the code , It's about testing this TIME_WAIT Whether to reuse .

static int __inet_check_established(......)
/* Check TIME-WAIT sockets first. */
sk_nulls_for_each(sk2, node, &head->twchain) {
tw = inet_twsk(sk2);
// If in time_wait Find one of them match Of port, Judge whether it can be reused
if (INET_TW_MATCH(sk2, net, hash, acookie,
saddr, daddr, ports, dif)) {
if (twsk_unique(sk, sk2, twp))
goto unique;
goto not_unique;

And the core function is twsk_unique, Its judgment logic is as follows :

int tcp_twsk_unique(......)
if (tcptw->tw_ts_recent_stamp &&
(twp == NULL || (sysctl_tcp_tw_reuse &&
get_seconds() - tcptw->tw_ts_recent_stamp > 1))) {
// Yes write_seq Set to snd_nxt+65536+2
// This ensures that at the data transfer rate <=80Mbit/s It's not going to be rewound
tp->write_seq = tcptw->tw_snd_nxt + 65535 + 2
return 1;
return 0;

The logic of the above code is as follows :

In the open tcp_timestamp as well as tcp_tw_reuse Under the circumstances , stay Connect Search for port Just use this one more than before port Of TIME_WAIT State of Socket Last timestamp of the record >1s, You can reuse this port, Just before 1 Minutes to 1s. At the same time, in order to prevent potential serial number conflicts , Direct will write_seq Plus 65537, such , In the single Socket The transmission rate is less than 80Mbit/s Under the circumstances , No overlapping of serial numbers ( Conflict ).
At the same time the tw_ts_recent_stamp The setting time is shown in the figure below :

So if Socket Get into TIME_WAIT Post state , If there is always a corresponding package sent , That will affect this TIME_WAIT Corresponding port Whether available time .
When this parameter is turned on , As a result of 1min Shorten to 1s, that Nginx Single to single Upstream Affordable TPS From the original 470.5TPS(28232/60) It was promoted to 28232TPS, increased 60 times .
If the performance is not enough , You can add the above port number range and tcp_max_tw_bucket Turn down and continue to improve tps, however tcp_max_tw_bucket There may be a risk of overlapping serial numbers , After all Socket Not pass 2MSL Phases are reused .

Don't open tcp_tw_recycle

Turn on tcp_tw_recyle This parameter will be in NAT The environment has a big impact , It is not recommended to open .

Nginx upstream Change growth connection

in fact , All of the above problems are due to Nginx Yes Backend It's a short connection that leads to .
Nginx from 1.1.4 Start , The support function of long connection for back-end machine is realized . stay Upstream This configuration can open the function of long connection :

upstream backend {
# It should be particularly noted that the keepalive directive does not limit the total number of connections to upstream servers that an nginx worker process can open. The connections parameter should be set to a number small enough to let upstream servers process new incoming connections as well.
keepalive 32;
keepalive_timeout 30s; # Set the maximum size of the back-end connection idle Time is 30s

In this way, both the front end and the back end are long connections , We can play happily again .

The resulting risk points

Because for a single distal end ip:port Depletion can lead to CPU Full of this phenomenon . So in Nginx In the configuration Upstream You need to be very careful . Suppose a situation ,PE We have expanded the capacity of one Nginx, To prevent problems , We'll have one first Backend Look at the situation , At this time, if the quantity is relatively large, the breakdown critical point will cause a large number of errors ( And the application itself has no pressure , After all, the threshold is 470.5TPS(28232/60)), Even with Nginx Requests for non domain names on will also be due to CPU Exhausted and unresponsive . A few more Backend/ Turn on tcp_tw_reuse Maybe it's a good choice .


No matter how powerful the application is, it is still loaded on the kernel , Never escape Linux The cage of the kernel . So for Linux The tuning of kernel parameters is very meaningful . If you read some kernel source code , Undoubtedly, it has a great help for us to investigate online problems , At the same time, it can guide us to avoid some holes !

official account

I am concerned about the official account , Get more dry articles :

本文为[Undamaged Huguang al]所创,转载请带上原文链接,感谢

  1. [front end -- JavaScript] knowledge point (IV) -- memory leakage in the project (I)
  2. This mechanism in JS
  3. Vue 3.0 source code learning 1 --- rendering process of components
  4. Learning the realization of canvas and simple drawing
  5. gin里获取http请求过来的参数
  6. vue3的新特性
  7. Get the parameters from HTTP request in gin
  8. New features of vue3
  9. vue-cli 引入腾讯地图(最新 api,rocketmq原理面试
  10. Vue 学习笔记(3,免费Java高级工程师学习资源
  11. Vue 学习笔记(2,Java编程视频教程
  12. Vue cli introduces Tencent maps (the latest API, rocketmq)
  13. Vue learning notes (3, free Java senior engineer learning resources)
  14. Vue learning notes (2, Java programming video tutorial)
  15. 【Vue】—props属性
  16. 【Vue】—创建组件
  17. [Vue] - props attribute
  18. [Vue] - create component
  19. 浅谈vue响应式原理及发布订阅模式和观察者模式
  20. On Vue responsive principle, publish subscribe mode and observer mode
  21. 浅谈vue响应式原理及发布订阅模式和观察者模式
  22. On Vue responsive principle, publish subscribe mode and observer mode
  23. Xiaobai can understand it. It only takes 4 steps to solve the problem of Vue keep alive cache component
  24. Publish, subscribe and observer of design patterns
  25. Summary of common content added in ES6 + (II)
  26. No.8 Vue element admin learning (III) vuex learning and login method analysis
  27. Write a mini webpack project construction tool
  28. Shopping cart (front-end static page preparation)
  29. Introduction to the fluent platform
  30. Webpack5 cache
  31. The difference between drop-down box select option and datalist
  32. CSS review (III)
  33. Node.js学习笔记【七】
  34. Node.js learning notes [VII]
  35. Vue Router根据后台数据加载不同的组件(思考-&gt;实现-&gt;不止于实现)
  36. Vue router loads different components according to background data (thinking - & gt; Implementation - & gt; (more than implementation)
  37. 【JQuery框架,Java编程教程视频下载
  38. [jQuery framework, Java programming tutorial video download
  39. Vue Router根据后台数据加载不同的组件(思考-&gt;实现-&gt;不止于实现)
  40. Vue router loads different components according to background data (thinking - & gt; Implementation - & gt; (more than implementation)
  41. 【Vue,阿里P8大佬亲自教你
  42. 【Vue基础知识总结 5,字节跳动算法工程师面试经验
  43. [Vue, Ali P8 teaches you personally
  44. [Vue basic knowledge summary 5. Interview experience of byte beating Algorithm Engineer
  45. 【问题记录】- 谷歌浏览器 Html生成PDF
  46. [problem record] - PDF generated by Google browser HTML
  47. 【问题记录】- 谷歌浏览器 Html生成PDF
  48. [problem record] - PDF generated by Google browser HTML
  49. 【JavaScript】查漏补缺 —数组中reduce()方法
  50. [JavaScript] leak checking and defect filling - reduce() method in array
  51. 【重识 HTML (3),350道Java面试真题分享
  52. 【重识 HTML (2),Java并发编程必会的多线程你竟然还不会
  53. 【重识 HTML (1),二本Java小菜鸟4面字节跳动被秒成渣渣
  54. [re recognize HTML (3) and share 350 real Java interview questions
  55. [re recognize HTML (2). Multithreading is a must for Java Concurrent Programming. How dare you not
  56. [re recognize HTML (1), two Java rookies' 4-sided bytes beat and become slag in seconds
  57. 【重识 HTML ,nginx面试题阿里
  58. 【重识 HTML (4),ELK原来这么简单
  59. [re recognize HTML, nginx interview questions]
  60. [re recognize HTML (4). Elk is so simple