nginx 104 Connection reset by peer while reading upstream错误处理
故障现象
1.看日志发现正常日志和错误日志比例几乎1:1 2.错误日志全部是104: Connection reset by peer) while reading upstream 3.看访问日志也没有其他http错误状态码
1 | [root@VM_0_22_centos logs]# ls -lh |
分析问题
1.连续责任人咨询业务场景发现客户端请求基本上都是POST请求,开始以为是上传大文件连接超时了,后来开发确认为了安全使用POST请求,所以并没有大文件上传 2.由于upstream重置连接了,就是说后端主动断开了连接,然后发现连接里有很多TIME-WAIT,应该是qps比较大的情况下,连接处理比较快还在断开连接中就显得比较多了 3.nginx作为反向代理既然是客户端又是服务端,当和后端服务建立连接时并没有默认开启长连接,开启长连接后性能应该会提升很多 4.默认开启长连接不需要keeplive参数,如下是nginx官网查寻的keepalive参数,看的不是很明白,不过有个链接讲的很清楚,他可以激活连接缓存,应该属于长连接性能优化类 5.keepalive参数值应该与qps有关,默认不需要设置太大,如果访问日志里面有5XX错误还得根据实际情况调整,以达到最优效果
下面是官网keeplaive参数解释 Syntax: keepalive connections; Default: — Context: upstream This directive appeared in version 1.1.4.
Activates the cache for connections to upstream servers.
The connections parameter sets the maximum number of idle keepalive connections to upstream servers that are preserved in the cache of each worker process. When this number is exceeded, the least recently used connections are closed.
It should be particularly noted that the keepalive directive does not limit the total number of connections to upstream servers that an nginx worker process can open. The connections parameter should be set to a number small enough to let upstream servers process new incoming connections as well. When using load balancing methods other than the default round-robin method, it is necessary to activate them before the keepalive directive.
处理方案
1.修改nginx配置开启长连接及结合连接缓存 2.重启nginx服务 主要配置如下
1 | upstream gateway{ |
检查效果
1.查看错误日志 错误日志清空后没有增长过
1 | [root@VM_0_22_centos logs]# ls -lh |
2.查看连接数状态 长连接前TIME-WAIT比较多
1 | [root@VM_0_22_centos logs]# ss -an |awk '{print $2}'|sort |uniq -dc |sort -rn |
长连接后TSTAB比较多
1 | [root@VM_0_22_centos ~]# ss -an |awk '{print $2}'|sort |uniq -dc |sort -rn |
参考文档
http://nginx.org/en/docs/http/ngx\_http\_upstream\_module.html#keepalive https://www.cnblogs.com/sunsky303/p/10648861.html http://blog.51yip.com/apachenginx/2203.html