We just wrote some new documentation on how Gitlab uses Unicorn and unicorn-worker-killer, available on doc.gitlab.com but also included below. We would love to hear from the community if you have other questions so we can improve this documentation resource!

Update 19:29 CEST: made link to doc.gitlab.com more specific.

Understanding Unicorn and unicorn-worker-killer

Unicorn

GitLab uses Unicorn, a pre-forking Ruby web server, to handle web requests (web browsers and Git HTTP clients). Unicorn is a daemon written in Ruby and C that can load and run a Ruby on Rails application; in our case the Rails application is GitLab Community Edition or GitLab Enterprise Edition.

Unicorn has a multi-process architecture to make better use of available CPU cores (processes can run on different cores) and to have stronger fault tolerance (most failures stay isolated in only one process and cannot take down GitLab entirely). On startup, the Unicorn 'master' process loads a clean Ruby environment with the GitLab application code, and then spawns 'workers' which inherit this clean initial environment. The 'master' never handles any requests, that is left to the workers. The operating system network stack queues incoming requests and distributes them among the workers.

In a perfect world, the master would spawn its pool of workers once, and then the workers handle incoming web requests one after another until the end of time. In reality, worker processes can crash or time out: if the master notices that a worker takes too long to handle a request it will terminate the worker process with SIGKILL ('kill -9'). No matter how the worker process ended, the master process will replace it with a new 'clean' process again. Unicorn is designed to be able to replace 'crashed' workers without dropping user requests.

This is what a Unicorn worker timeout looks like in unicorn_stderr.log. The master process has PID 56227 below.

[2015-06-05T10:58:08.660325 #56227] ERROR -- : worker=10 PID:53009 timeout (61s > 60s), killing
[2015-06-05T10:58:08.699360 #56227] ERROR -- : reaped #<Process::Status: pid 53009 SIGKILL (signal 9)> worker=10
[2015-06-05T10:58:08.708141 #62538] INFO -- : worker=10 spawned pid=62538
[2015-06-05T10:58:08.708824 #62538] INFO -- : worker=10 ready

Tunables

The main tunables for Unicorn are the number of worker processes and the request timeout after which the Unicorn master terminates a worker process. See the omnibus-gitlab Unicorn settings documentation if you want to adjust these settings.

unicorn-worker-killer

GitLab has memory leaks. These memory leaks manifest themselves in long-running processes, such as Unicorn workers. (The Unicorn master process is not known to leak memory, probably because it does not handle user requests.)

To make these memory leaks manageable, GitLab comes with the unicorn-worker-killer gem. This gem monkey-patches the Unicorn workers to do a memory self-check after every 16 requests. If the memory of the Unicorn worker exceeds a pre-set limit then the worker process exits. The Unicorn master then automatically replaces the worker process.

This is a robust way to handle memory leaks: Unicorn is designed to handle workers that 'crash' so no user requests will be dropped. The unicorn-worker-killer gem is designed to only terminate a worker process in between requests, so no user requests are affected.

This is what a Unicorn worker memory restart looks like in unicorn_stderr.log. You see that worker 4 (PID 125918) is inspecting itself and decides to exit. The threshold memory value was 254802235 bytes, about 250MB. With GitLab this threshold is a random value between 200 and 250 MB. The master process (PID 117565) then reaps the worker process and spawns a new 'worker 4' with PID 127549.

[2015-06-05T12:07:41.828374 #125918]  WARN -- : #<Unicorn::HttpServer:0x00000002734770>: worker (pid: 125918) exceeds memory limit (256413696 bytes > 254802235 bytes)
[2015-06-05T12:07:41.828472 #125918] WARN -- : Unicorn::WorkerKiller send SIGQUIT (pid: 125918) alive: 23 sec (trial 1)
[2015-06-05T12:07:42.025916 #117565] INFO -- : reaped #<Process::Status: pid 125918 exit 0> worker=4
[2015-06-05T12:07:42.034527 #127549] INFO -- : worker=4 spawned pid=127549
[2015-06-05T12:07:42.035217 #127549] INFO -- : worker=4 ready

One other thing that stands out in the log snippet above, taken from Gitlab.com, is that 'worker 4' was serving requests for only 23 seconds. This is a normal value for our current GitLab.com setup and traffic.

The high frequency of Unicorn memory restarts on some GitLab sites can be a source of confusion for administrators. Usually they are a red herring.

https://about.gitlab.com/2015/06/05/how-gitlab-uses-unicorn-and-unicorn-worker-killer/

Understanding Unicorn and unicorn-worker-killer Unicorn的更多相关文章

  1. different between unicorn / unicorn_rails

    $ unicorn_rails -h Usage: unicorn_rails [ruby options] [unicorn_rails options] [rackup config file] ...

  2. Nginx + unicorn 运行多个Rails应用程序

    PS:第一次写的很详细,可惜发布失败,然后全没了,这是第二次,表示只贴代码,剩下的自己领悟好了,这就是所谓的一鼓作气再而衰吧,希望没有第三次. 版本: ruby 2.1.0 rails 4.0.2 n ...

  3. puma vs passenger vs rainbows! vs unicorn vs thin 适用场景 及 performance

    ruby的几个web server,按照开发活跃度.并发方案及要点.适用场景等分析puma vs passenger vs rainbows! vs unicorn vs thin. 1. thin: ...

  4. Setting up Unicorn with Nginx

    gem install unicorn or gem 'unciron' 1 install Nginx yum install ... 2 Configuration vi /etc/nginx/n ...

  5. MacOS下安装unicorn这个库失败

    因为在Mac下安装pwntools,发现安装unicorn库的时候失败了,编译报错如下 make: *** [qemu/config-host.h-timestamp] Error 1 error: ...

  6. gitlab服务器搭建教程

    gitlab服务器搭建教程 ----2016年终总结 三 参考https://bbs.gitlab.cc/topic/35/gitlab-ce-8-7-%E6%BA%90%E7%A0%81%E5%AE ...

  7. gitlab10.0安装手记

    + +exec chpst -e /opt/gitlab/etc/gitlab-workhorse/env -P \ + -U git \ + -u git \ + /opt/gitlab/embed ...

  8. Jenkins + Ansible + Gitlab之gitlab篇

    前言 持续交付 版本控制器:Gitlab.GitHub 持续集成工具:jenkins 部署工具:ansible  课程安排 Gitlab搭建与流程使用 Ansible环境配置与Playbook编写规范 ...

  9. ror笔记2

    在rails app的 config 文件夹中新建unicorn.rb内容如下 worker_processes 2 working_directory "/home/mage/boleht ...

随机推荐

  1. 用jquery写的图片懒加载

    <%@ page language="java" contentType="text/html; charset=UTF-8" pageEncoding= ...

  2. ListView列表刷新方法的区别

    ListView列表刷新方法的区别 ListView对象的刷新方法: listNote.invalidate();重绘所有组件listNote.invalidateViews();重绘组件[包含所有的 ...

  3. js 判断变量是否为空

    js 判断变量是否为空 欢迎指正,补充! /** * 判断变量是否为空, * @param {[type]} param 变量 * @return {Boolean} 为空返回true,否则返回fal ...

  4. Django中使用表单

    使用表单 表单用 user 提交数据,是网站中比较重要的一个内容 GET 和 POST 方法 GET 和 POST 的区别 URL,全称是"统一资源定位符".用于对应互联网上的每一 ...

  5. 专访Nick McKeown:网络领域的游戏颠覆者

    如果要找到一个过去10年在网络领域最热的词汇,那么非SDN(软件定义网络)莫属.在过去的十年间无论是学术机构还是标准组织,无论是电信巨擘还是互联网大厂都成其拥趸. 然而几乎每一件SDN的重大事件都离不 ...

  6. rostopic pub

    rostopic pub -1 reinit_motor_wheel std_msgs/String -- "reinit_motor_wheel"rostopic pub -r ...

  7. 海量端口扫描工具masscan

    海量端口扫描工具masscan   masscan号称是互联网上最快的端口扫描工具,可以6分钟扫描整个互联网,每秒可以发送一百万个数据包.为了提高处理速度,masscan定制了TCP/IP栈,从而不影 ...

  8. mac mysql忘记密码解决办法

    http://www.jb51.net/article/87580.htm http://blog.csdn.net/soft2buy/article/details/50223373

  9. 南阳 oj 表达式求值 题目35 数据结构 NYO题目链接

     建议不会的看别人的代码自己在之上模拟一遍,仅仅要耐心模拟就会做出来 题目链接:http://acm.nyist.net/JudgeOnline/problem.php? pid=35 #incl ...

  10. Java序列化算法

    Serialization(序列化)是一种将对象以一连串的字节描述的过程:反序列化deserialization是一种将这些字节重建成一个对象的过程.java序列化API提供一种处理对象序列化的标准机 ...