linux 后台进程管理利器supervisor

Linux的后台进程运行有好几种方法，例如nohup，screen等，但是，如果是一个服务程序，要可靠地在后台运行，我们就需要把它做成daemon，最好还能监控进程状态，在意外结束时能自动重启。

 

supervisor就是用Python开发的一套通用的进程管理程序，能将一个普通的命令行进程变为后台daemon，并监控进程状态，异常退出时能自动重启。

ubuntu安装：

apt-get install supervisor

在/etc/supervisor 目录下有supervisord.conf 文件，内容如下：

; supervisor config file

[unix_http_server]
file=/var/run/supervisor.sock   ; (the path to the socket file)
chmod=0700                       ; sockef file mode (default 0700)

[supervisord]
logfile=/var/log/supervisor/supervisord.log ; (main log file;default $CWD/supervisord.log)
pidfile=/var/run/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
childlogdir=/var/log/supervisor            ; ('AUTO' child log dir, default $TEMP)

; the below section must remain in the config file for RPC
; (supervisorctl/web interface) to work, additional interfaces may be
; added by defining them in separate rpcinterface: sections
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl=unix:///var/run/supervisor.sock ; use a unix:// URL  for a unix socket

; The [include] section can just contain the "files" setting.  This
; setting can list multiple files (separated by whitespace or
; newlines).  It can also contain wildcards.  The filenames are
; interpreted as relative to this file.  Included files *cannot*
; include files themselves.

[include]
files = /etc/supervisor/conf.d/*.conf

然后，给我们自己开发的应用程序编写一个配置文件，让supervisor来管理它。每个进程的配置文件都可以单独分拆，放在/etc/supervisor/conf.d/目录下，以.conf作为扩展名，如果修改了 /etc/supervisord.conf ,,需要执行 supervisorctl reload （重启）来重新加载配置文件，否则会感觉没有生效，折腾到抓狂。。。

例如，app.conf定义了一个gunicorn的进程：

 

[html] 

[program:app]  

command=/usr/bin/gunicorn -w 1 wsgiapp:application  

directory=/srv/www  

user=www-data  

 

其中，进程app定义在[program:app]中，command是命令，directory是进程的当前目录，user是进程运行的用户身份。

 

重启supervisor，让配置文件生效，

 

# supervisorctl start app  

 

然后运行命令supervisorctl启动进程：

停止进程：

 

 

# supervisorctl stop app  

 

如果要在命令行中使用变量，就需要自己先编写一个shell脚本：

 

[html] 

#!/bin/sh  

/usr/bin/gunicorn -w `grep -c ^processor /proc/cpuinfo` wsgiapp:application  

 

然后，加上x权限，再把command指向该shell脚本即可。

 

先弄懂两个命令:

supervisord : supervisor的服务器端部分，启动supervisor就是运行这个命令

supervisorctl：启动supervisor的命令行窗口。

参考：http://www.2cto.com/os/201308/238166.html

一篇文章：使用supervisor监控进程：

在linux下监控进程，可以使用inittab，最近找到了supervisor，也很好用，记录一下：
1、系统要安装python，并安装与之对应的setuptools，下载地址在此
2、安装：
# sh setuptoolsxxxx.egg
3、安装supervisor，下载地址在此，解压缩后
# python setup.py install
这就ok了，然后执行
# echo_supervisord_conf > /etc/supervisord.conf
修改/etc/supervisord.conf文件，加入你要监控的进程，里面的注释很详细，举个简单的例子：
这是一段要监控的进程的描述信息，添加到这个文件的末尾就好了：
[program:meta.txn.recover.on.error]
command=/cas/bin/meta.txn.recover.on.error ; 被监控的进程路径
numprocs=1                    ; 启动几个进程
directory=/cas/bin                ; 执行前要不要先cd到目录去，一般不用
autostart=true                ; 随着supervisord的启动而启动
autorestart=true              ; 自动重启。。当然要选上了
startretries=10               ; 启动失败时的最多重试次数
exitcodes=0                 ; 正常退出代码（是说退出代码是这个时就不再重启了吗？待确定）
stopsignal=KILL               ; 用来杀死进程的信号
stopwaitsecs=10               ; 发送SIGKILL前的等待时间
redirect_stderr=true          ; 重定向stderr到stdout
为了节省空间，注释的内容就不贴出来了。
执行
# supervisord -n
能在控制台看到监控进程的输出：
2010-08-17 10:26:07,467 INFO supervisord started with pid 943
2010-08-17 10:26:08,469 INFO spawned: 'meta.txn.recover.on.error' with pid 1009
2010-08-17 10:26:09,876 INFO success: meta.txn.recover.on.error entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2010-08-17 10:26:48,442 INFO exited: meta.txn.recover.on.error (terminated by SIGKILL; not expected)
2010-08-17 10:26:49,444 INFO spawned: 'meta.txn.recover.on.error' with pid 2427
2010-08-17 10:26:50,487 INFO success: meta.txn.recover.on.error entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
黑体的地方是我用kill -9杀掉进程后出来的，看到supervisor检测到进程退出后又再次启动了进程。
不带参数运行supervisord是以daemon方式运行。
把supervisord加入到开机启动项里就可以完成监控进程的功能了。

【注意】：当supervisord以非daemon方式运行时，杀掉supervisord后，被监控的进程也退出了。
而以daemon方式运行，杀掉supervisord对被监控进程无影响。

python安装：

easy_install supervisor

测试是否安装成功：

echo_supervisord_conf

会显示以下内容，其实是一个配置模版:

; Sample supervisor config file.
;
; For more information on the config file, please see:
; http://supervisord.org/configuration.html
;
; Note: shell expansion ("~" or "$HOME") is not supported. Environment
; variables can be expanded using this syntax: "%(ENV_HOME)s".

[unix_http_server]
file=/tmp/supervisor.sock ; (the path to the socket file)
;chmod=0700 ; socket file mode (default 0700)
;chown=nobody:nogroup ; socket file uid:gid owner
;username=user ; (default is no username (open server))
;password=123 ; (default is no password (open server))

;[inet_http_server] ; inet (TCP) server disabled by default
;port=127.0.0.1:9001 ; (ip_address:port specifier, *:port for all iface)
;username=user ; (default is no username (open server))
;password=123 ; (default is no password (open server))

[supervisord]
logfile=/tmp/supervisord.log ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=50MB ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10 ; (num of main logfile rotation backups;default 10)
loglevel=info ; (log level;default info; others: debug,warn,trace)
pidfile=/tmp/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
……

; the below section must remain in the config file for RPC
; (supervisorctl/web interface) to work, additional interfaces may be
; added by defining them in separate rpcinterface: sections
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl=unix:///tmp/supervisor.sock ; use a unix:// URL for a unix socket
;serverurl=http://127.0.0.1:9001 ; use an http:// url to specify an inet socket
;username=chris ; should be same as http_username if set
;password=123 ; should be same as http_password if set
;prompt=mysupervisor ; cmd line prompt (default "supervisor")
;history_file=~/.sc_history ; use readline history if available

; The below sample program section shows all possible program subsection values,
; create one or more ‘real’ program: sections to be able to control them under
; supervisor.

;[program:theprogramname]
;command=/bin/cat ; the program (relative uses PATH, can take args)
;process_name=%(program_name)s ; process_name expr (default %(program_name)s)
;numprocs=1 ; number of processes copies to start (def 1)
;directory=/tmp ; directory to cwd to before exec (def no cwd)
;umask=022 ; umask for process (default None)
;priority=999 ; the relative start priority (default 999)
;autostart=true ; start at supervisord start (default: true)
;autorestart=unexpected ; whether/when to restart (default: unexpected)
;startsecs=1 ; number of secs prog must stay running (def. 1)
;startretries=3 ; max # of serial start failures (default 3)
;exitcodes=0,2 ; ‘expected’ exit codes for process (default 0,2)
;stopsignal=QUIT ; signal used to kill process (default TERM)
;stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10)
……

; The [include] section can just contain the "files" setting. This
; setting can list multiple files (separated by whitespace or
; newlines). It can also contain wildcards. The filenames are
; interpreted as relative to this file. Included files *cannot*
; include files themselves.

;[include]
;files = relative/directory/*.ini

创建配置文件：

echo_supervisord_conf > /etc/supervisord.conf

(easy_install安装的方式好像没有/etc/supervisord.conf,所以这里我们手动创建）

修改配置文件：

在supervisord.conf最后增加(分号后边的表示注释，可以不写)：

[program:open_search]
command=/home/s/www/fukun/gopath/src/open_search/open_search ;需要执行的命令
directory=/home/s/www/fukun/gopath/src/open_search/ ; directory to cwd to before exec (def no cwd)
autostart=true ; start at supervisord start (default: true)
autorestart=true ; whether/when to restart (default: unexpected)
startsecs=3 ; number of secs prog must stay running (def. 1)
redirect_stderr=true ; redirect proc stderr to stdout (default false) 错误输出重定向
stdout_logfile=/tmp/open_search_gorun.log ; stdout log path, NONE for none; default AUTO, log输出

（更多配置说明请参考：http://supervisord.org/configuration.html）

supervisor管理gearman

参考官方文档：

http://supervisord.org/running.html

Running supervisord automatically on startup

If you are using a distribution-packaged version of Supervisor, it should already be integrated into the service management infrastructure of your distribution.

There are user-contributed scripts for various operating systems at:https://github.com/Supervisor/initscripts

There are some answers at Serverfault in case you get stuck: How to automatically start supervisord on Linux (Ubuntu)

github的脚本下载下来：

#!/bin/bash
#
# supervisord   This scripts turns supervisord on
#
# Author:       Mike McGrath <mmcgrath@redhat.com> (based off yumupdatesd)
#               Jason Koppe <jkoppe@indeed.com> adjusted to read sysconfig,
#                   use supervisord tools to start/stop, conditionally wait
#                   for child processes to shutdown, and startup later
#               Mikhail Mingalev <mingalevme@gmail.com> Merged
#                   redhat-init-jkoppe and redhat-sysconfig-jkoppe, and
#                   made the script "simple customizable".
#               Brendan Maguire <maguire.brendan@gmail.com> Added OPTIONS to
#                   SUPERVISORCTL status call
#
# chkconfig:
#
# description:  supervisor is a process control utility.  It has a web based
#               xmlrpc interface as well as a few other nifty features.
#               Script was originally written by Jason Koppe <jkoppe@indeed.com>.
#

# source function library
. /etc/rc.d/init.d/functions

set -a

PREFIX=/usr

SUPERVISORD=$PREFIX/bin/supervisord
SUPERVISORCTL=$PREFIX/bin/supervisorctl

PIDFILE=/var/run/supervisord.pid
LOCKFILE=/var/lock/subsys/supervisord

OPTIONS="-c /etc/supervisord.conf"

# unset this variable if you don't care to wait for child processes to shutdown before removing the $LOCKFILE-lock
WAIT_FOR_SUBPROCESSES=yes

# remove this if you manage number of open files in some other fashion
ulimit -n 

RETVAL=

running_pid()
{
    # Check if a given process pid's cmdline matches a given name
    pid=$
    name=$
    [ -z "$pid" ] && return
    [ ! -d /proc/$pid ] && return
    (cat /proc/$pid/cmdline | tr "\000" "\n"|grep -q $name) || return
    return
}

running()
{
# Check if the process is running looking at /proc
# (works for all users)

    # No pidfile, probably no daemon present
    [ ! -f "$PIDFILE" ] && return
    # Obtain the pid and check it against the binary name
    pid=`cat $PIDFILE`
    running_pid $pid $SUPERVISORD || return
    return
}

start() {
        echo "Starting supervisord: "

        if [ -e $PIDFILE ]; then
        echo "ALREADY STARTED"
        return
    fi

    # start supervisord with options from sysconfig (stuff like -c)
        $SUPERVISORD $OPTIONS

    # show initial startup status
    $SUPERVISORCTL $OPTIONS status

    # only create the subsyslock if we created the PIDFILE
        [ -e $PIDFILE ] && touch $LOCKFILE
}

stop() {
        echo -n "Stopping supervisord: "
        $SUPERVISORCTL $OPTIONS shutdown
    if [ -n "$WAIT_FOR_SUBPROCESSES" ]; then
            echo "Waiting roughly 60 seconds for $PIDFILE to be removed after child processes exit"
            for sleep in              last; do
                if [ ! -e $PIDFILE ] ; then
                    echo "Supervisord exited as expected in under $total_sleep seconds"
                    break
                else
                    if [[ $sleep -eq "last" ]] ; then
                        echo "Supervisord still working on shutting down. We've waited roughly 60 seconds, we'll let it do its thing from here"
                        return
                    else
                        sleep $sleep
                        total_sleep=$(( $total_sleep + $sleep ))
                    fi

                fi
            done
        fi

        # always remove the subsys. We might have waited a while, but just remove it at this point.
        rm -f $LOCKFILE
}

restart() {
        stop
        start
}

case "$1" in
    start)
        start
        RETVAL=$?
        ;;
    stop)
        stop
        RETVAL=$?
        ;;
    restart|force-reload)
        restart
        RETVAL=$?
        ;;
    reload)
        $SUPERVISORCTL $OPTIONS reload
        RETVAL=$?
        ;;
    condrestart)
        [ -f $LOCKFILE ] && restart
        RETVAL=$?
        ;;
    status)
        $SUPERVISORCTL $OPTIONS status
        if running ; then
            RETVAL=
        else
            RETVAL=
        fi
        ;;
    *)
        echo $"Usage: $0 {start|stop|status|restart|reload|force-reload|condrestart}"
        exit
esac

exit $RETVAL

vim /etc/init.d/supervisord , 修改文件属性为可执行chmod 777 /etc/init.d/supervisordmy

加入gearman的任务，下面是百度云推送的任务

vim /etc/supervisord.conf

[program:APNS_PUSH]
command=/usr/bin/php /data2/www/web/apns/worker_origin.php
process_name=APNS_PUSH_%(process_num)s
numprocs=
autostart=true
autorestart=true
user=nginx
stdout_logfile=/data2/log/push/push_%(process_num)s.log
stderr_logfile=/data2/log/push/push.error.log

下面先启动gearman，启动supervisord，ps -ef |grep xxx 查看一下任务有没有执行

参考：http://www.cnblogs.com/trying/p/4038285.html

Supervisord是用Python实现的一款非常实用的进程管理工具，在批量服务化管理时特别有效。

1、安装
安装环境 centos python2.4
安装 wget http://pypi.python.org/packages/source/s/supervisor/supervisor-3.0a10.tar.gz#md5=99c6fbd45bade87301296b7a597fb68e 
tar zxvf supervisor-3.0a10.tar.gz
cd supervisor-3.0a10
python setup.py install
如果上述遇到问题，请按如下操作。
依赖setuptools安装 wget http://pypi.python.org/packages/2.4/s/setuptools/setuptools-0.6c11-py2.4.egg#md5=bd639f9b0eac4c42497034dec2ec0c2b 
sh setuptools-0.6c11-py2.4.egg

2、配置
产生基础配置文件 echo_supervisord_conf > /etc/supervisord.conf
配置 [inet_http_server]前面的分号去掉; 定义port=192.168.1.1:8888 启动 supervisord -c /etc/supervisord.conf 查看 浏览器打开 http://192.168.10.129:9001 
深入配置 [program:service] 
command=sh /opt/soft/service/run.sh
目前我们用supervisord 监控 memcached 进程
[program:memcached] ###定义服务 
command=memcached -m 2000 -u nobody -l 0.0.0.0 -c 10240 -p 11211  ###服务运行时的显示的command
numprocs=1                                                                                       ###控制memcached启动的进程数
user=root                                                                                            ###程序运行的用户
autostart=true                                                                                     ###supervisord启动memcached启动
autorestart=true                                                                                  ###supervisord重启memcache自动重启
stdout_logfile=/var/log/memcached.stdout.log                                       ###监控memcached日志文件
redirect_stderr=true                                                                            ###将stderr重定向到stdout（或者创建个err.log）
stopsignal=QUIT

这个程序最好不能是daemon后台守护程序。supervisord会作为这个程序的父来启动它

3、启动
supervisord (以daemon方式启动)
或 supervisord -c /etc/supervisord.conf （非daemon）

supervisord，初始启动Supervisord，启动、管理配置中设置的进程。
supervisorctl 简单的后台进程控制工具

supervisorctl stop programxxx，停止某一个进程(programxxx)，programxxx为[program:chatdemon]里配置的值
supervisorctl start programxxx，启动某个进程
supervisorctl restart programxxx，重启某个进程
supervisorctl stop all，停止全部进程，注：start、restart、stop都不会载入最新的配置文件。
supervisorctl reload，载入最新的配置文件，并按新的配置启动、管理所有进程。
supervisorctl reread，当一个服务由自动启动修改为手动启动时执行一下就ok

不带参数运行supervisord是以daemon方式运行
当supervisord以非daemon方式运行时，杀掉supervisord后，被监控的进程也退出了。
而以daemon方式运行，杀掉supervisord对被监控进程无影响

supervosrd配置文件：

command 字段设置的是后台守护应用的启动命令, 注意: 该命令必须是在前台执行的, 即会独占控制台, 否则会导致 supervisor 无法获得标准输出, 并失去进程的控制权.

更多：

http://digdeeply.org/archives/07102224.html