The "Out of socket memory" error

I recently did some work on some of our frontend machines (on which we run Varnish) at StumbleUpon and decided to track down some of the errors the Linux kernel was regularly throwing in kern.log such as:

Feb 25 08:23:42 foo kernel: [3077014.450011] Out of socket memory

Before we get started, let me tell you that you should NOT listen to any blog or forum post without doing your homework, especially when the post recommends that you tune up virtually every TCP related knob in the kernel. These people don't know what they're doing and most probably don't understand much to TCP/IP. Most importantly, their voodoo won't help you fix your problem and might actually make it worse.

Dive in the Linux kernel

In order to best understand what's going on, the best thing is to go read the code of the kernel. Unfortunately, the kernel's error messages or counters are often imprecise, confusing, or even misleading. But they're important. And reading the kernel's code isn't nearly as hard as what people say.

The "Out of socket memory" error

The only match for "Out of socket memory" in the kernel's code (as of v2.6.38) is in net/ipv4/tcp_timer.c:

66 static int tcp_out_of_resources(struct sock *sk, int do_reset)

67 {

68         struct tcp_sock *tp = tcp_sk(sk);

69         int shift = 0;

70

71         /* If peer does not open window for long time, or did not transmit

72          * anything for long time, penalize it. */

73         if ((s32)(tcp_time_stamp - tp->lsndtime) > 2*TCP_RTO_MAX || !do_reset)

74                 shift++;

75

76         /* If some dubious ICMP arrived, penalize even more. */

77         if (sk->sk_err_soft)

78                 shift++;

79

80         if (tcp_too_many_orphans(sk, shift)) {

81                 if (net_ratelimit())

82                         printk(KERN_INFO "Out of socket memory\n");

So the question is: when does tcp_too_many_orphans return true? Let's take a look in include/net/tcp.h:

268 static inline bool tcp_too_many_orphans(struct sock *sk, int shift)

269 {

270         struct percpu_counter *ocp = sk->sk_prot->orphan_count;

271         int orphans = percpu_counter_read_positive(ocp);

272

273         if (orphans << shift > sysctl_tcp_max_orphans) {

274                 orphans = percpu_counter_sum_positive(ocp);

275                 if (orphans << shift > sysctl_tcp_max_orphans)

276                         return true;

277         }

278

279         if (sk->sk_wmem_queued > SOCK_MIN_SNDBUF &&

280             atomic_long_read(&tcp_memory_allocated) > sysctl_tcp_mem[2])

281                 return true;

282         return false;

283 }

So two conditions that can trigger this "Out of socket memory" error:

1.There are "too many" orphan sockets (most common).

2.The socket already has the minimum amount of memory and we can't give it more because TCP is already using more than its limit.

In order to remedy to your problem, you need to figure out which case you fall into. The vast majority of the people (especially those dealing with frontend servers like Varnish) fall into case 1.

Are you running out of TCP memory?

Ruling out case 2 is easy. All you need is to see how much memory your kernel is configured to give to TCP vs how much is actually being used. If you're close to the limit (uncommon), then you're in case 2. Otherwise (most common) you're in case 1. The kernel keeps track of the memory allocated to TCP in multiple of pages, not in bytes. This is a first bit of confusion that a lot of people run into because some settings are in bytes and other are in pages (and most of the time 1 page = 4096 bytes).

Rule out case 2: find how much memory the kernel is willing to give to TCP:

$ cat /proc/sys/net/ipv4/tcp_mem

3093984 4125312 6187968

The values are in number of pages. They get automatically sized at boot time (values above are for a machine with 32GB of RAM). They mean:

When TCP uses less than 3093984 pages (11.8GB), the kernel will consider it below the "low threshold" and won't bother TCP about its memory consumption.

When TCP uses more than 4125312 pages (15.7GB), enter the "memory pressure" mode.

The maximum number of pages the kernel is willing to give to TCP is 6187968 (23.6GB). When we go above this, we'll start seeing the "Out of socket memory" error and Bad Things will happen.

Now let's find how much of that memory TCP actually uses.

$ cat /proc/net/sockstat

sockets: used 14565

TCP: inuse 35938 orphan 21564 tw 70529 alloc 35942 mem 1894

UDP: inuse 11 mem 3

UDPLITE: inuse 0

RAW: inuse 0

FRAG: inuse 0 memory 0

The last value on the second line (mem 1894) is the number of pages allocated to TCP. In this case we can see that 1894 is way below 6187968, so there's no way we can possibly be running out of TCP memory. So in this case, the "Out of socket memory" error was caused by the number of orphan sockets.

Do you have "too many" orphan sockets?

First of all: what's an orphan socket? It's simply a socket that isn't associated to a file descriptor. For instance, after you close() a socket, you no longer hold a file descriptor to reference it, but it still exists because the kernel has to keep it around for a bit more until TCP is done with it. Because orphan sockets aren't very useful to applications (since applications can't interact with them), the kernel is trying to limit the amount of memory consumed by orphans, and it does so by limiting the number of orphans that stick around. If you're running a frontend web server (or an HTTP load balancer), then you'll most likely have a sizeable number of orphans, and that's perfectly normal.

In order to find the limit on the number of orphan sockets, simply do:

$ cat /proc/sys/net/ipv4/tcp_max_orphans

65536

Here we see the default value, which is 64k. In order to find the number of orphan sockets in the system, look again in sockstat:

$ cat /proc/net/sockstat

sockets: used 14565

TCP: inuse 35938 orphan 21564 tw 70529 alloc 35942 mem 1894

[...]

So in this case we have 21564 orphans. That doesn't seem very close to 65536... Yet, if you look once more at the code above that prints the warning, you'll see that there is this shift variable that has a value between 0 and 2, and that the check is testing if (orphans << shift > sysctl_tcp_max_orphans). What this means is that in certain cases, the kernel decides to penalize some sockets more, and it does so by multiplying the number of orphans by 2x or 4x to artificially increase the "score" of the "bad socket" to penalize. The problem is that due to the way this is implemented, you can see a worrisome "Out of socket memory" error when in fact you're still 4x below the limit and you just had a couple "bad sockets" (which happens frequently when you have an Internet facing service). So unfortunately that means that you need to tune up the maximum number of orphan sockets even if you're 2x or 4x away from the threshold. What value is reasonable for you depends on your situation at hand. Observe how the count of orphans in /proc/net/sockstat is changing when your server is at peak traffic, multiply that value by 4, round it up a bit to have a nice value, and set it. You can set it by doing a echo of the new value in /proc/sys/net/ipv4/tcp_max_orphans, and don't forget to update the value of net.ipv4.tcp_max_orphans in /etc/sysctl.conf so that your change persists across reboots.

That's all you need to get rid of these "Out of socket memory" errors, most of which are "false alarms" due to the shift variable of the implementation.

原文

[本来准备翻译,时间不够,直接贴原文吧]

http://blog.tsunanet.net/2011/03/out-of-socket-memory.html

The "Out of socket memory" error的更多相关文章

  1. Myeclipse运行报错:an out of memory error has occurred的解决方法

    不知道怎么了,重装的myeclipse2013,里边就放了一个项目,启动myeclipse就报 an out of memory error has occurred....... 一点yes就退出 ...

  2. java.net.SocketException: Software caused connection abort: socket write error

    用Java客户端程序访问Java Web服务器时出错: java.net.SocketException: Software caused connection abort: socket write ...

  3. 解决:编译CM14.1 提示Jack “Out of memory error”错误

    Android 7.1编译到33%时出现JDK内存溢出的错误: Out of memory error (version f95d7bdecfceb327f9d201a1348397ed8a84384 ...

  4. Software caused connection abort: socket write error

    Exception in thread "main" java.net.SocketException: Software caused connection abort: soc ...

  5. Java socket 说明 以及web 出现java.net.SocketException:(Connection reset或者Connectreset by peer:Socket write error)的解释

    另外http://www.cnblogs.com/fengmk2/archive/2007/01/15/using-Socket.html可供参考   一Java socket 说明 所谓socket ...

  6. spring+ibatis问题1—— 程序报错:java.sql.SQLException: Io 异常: Connection reset by peer, socket write error; ”或“java.sql.SQLException 关闭的连接”异常

    转自:http://blog.sina.com.cn/s/blog_1549fb0710102whz2.html spring+ibatis程序测试时报错:java.sql.SQLException: ...

  7. testNG java.net.SocketException: Software caused connection abort: socket write error

    执行用例报错,提示 java.net.SocketException: Software caused connection abort: socket write error java.net.So ...

  8. Caused by: java.net.SocketException: Software caused connection abort: socket write error

    1.错误描述 [ERROR:]2015-05-06 10:54:18,967 [异常拦截] ClientAbortException: java.net.SocketException: Softwa ...

  9. 第一次JVM分析记录:Out of Memory Error (workgroup.cpp:96), pid=6196, tid=139999645685504

    tomcat的catalina.out日志报错如下: Exception in thread "http-bio-8081-Acceptor-0" java.lang.OutOfM ...

随机推荐

  1. 006.Zabbix添加监控主机

    一 配置步骤和流程 Zabbix完整的监控配置流程可以简单的描述为: Host groups(主机组)---->Hosts(主机)---->Applications(监控项组)----&g ...

  2. 【教程】使用gitee搭建免费的图床

    前几天七牛云的免费图床测试域名回收,导致我上传的图片都不能访问!要配置自定义域名,域名还要绑定主机.没有云主机的我开始想你们搞一个免费的图床,并且数据也不会丢失呢 ? ​ 想到之前自己在GitHub上 ...

  3. UI自动化测试(六)TestNG操作详解

    在编写TestNG代码的时候,若没有下载TestNG的jar包的话,代码会出错,下载jar包方法见该链接中java+selenium环境搭建的第二步即可:http://www.cnblogs.com/ ...

  4. UI自动化测试(五)TestNG简介与安装步骤

    简述 TestNG是一个设计用来简化广泛的测试需求的测试框架, 从单元测试(隔离测试一个类) 到集成测试(测试由有多个类多个包甚至多个外部框架组成的整个系统, 例如运用服务器) . testNG灵感来 ...

  5. odoo 工作流

    odoo工作流 介绍 新版本的odoo开始减少workflow的使用,推荐使用workflow-ish的方式来处理工作流过程 很多模块中还是使用到工作流,这里我记录一个简单的实例,欢迎大家给出建议. ...

  6. BZOJ2465: [中山市选2009]小球

    Description        给定n个不同颜色的球,每个球都有一个分数,同时有m个瓶子,每个瓶子都有固定的容量.现在,你必须把球放到瓶子里面.请编程计算最多能放多少个球到这些瓶子里. Inpu ...

  7. 使用gtest对DLL工程进行单元测试的实践

    前言 关于单元测试的重要性.gtest的优缺点等就不说了.之前项目是没有做单元测试的,在VS的解决方案中,只有一个可执行的工程,其他的工程都是以DLL库的形式提供.本文只针对使用VS对DLL库进行单元 ...

  8. Mac下配置Apache服务器

    有的时候,我们需要在内网工作组中分享一些文件或是后台接口没有及时给出,你又想要模拟真实数据,直接在项目里创建plist也可以做到这种需求,但难免让工程变得冗余且看起来比较Low.这个时候就看出配置本地 ...

  9. 解决Android LogCat 输出乱码的问题(转)

    Android日志系统提供了记录和查看系统调试信息的功能.日志都是从各种软件和一些系统的缓冲区中记录下来的. 可以使用adb的logcat 命令来查看系统日志缓冲区的内容,但是在实际操作时,会发现在C ...

  10. [Go] Http包 使用简介

    请求的结构 HTTP 的交互以请求和响应的应答模式.Go 的请求我们早就见过了,handler 函数的第二个参数 http.Requests.其结构为: type Request struct { M ...