分布式存储之MogileFS基于Nginx实现负载均衡（Nginx+MogileFS）

MogileFS分布式文件系统特点：

　　1.具有raid的性能

　　2.不存在单点故障

　　3.简单的命名空间： 每个文件对应一个key：用于domain定义名称空间

4.不共享任何数据

5.传输中立，无特殊协议：可以通过NFS或HTTP进行通信

6.自动文件复制：复制的最小单位不是文件，而class

　　7.应用层： 用户空间文件系统，无须特殊的核心组件

Nginx+MogileFS的好处：

　　1、将请求代理至后端MogileFS服务器集群中，能实现负载均衡的效果。

　  2、能对后端的tracker节点进行健康检测。

　  3、将第三方模块“nginx_mogilefs_module”编译进Nginx中，能实现直接使用key访问对应的文件，如下：

　　　　　　　　使用nginx做代理之前：http://192.168.80.137:7500/dev2/0/000/000/0000000007.fid

　　　　　　　　使用nginx做代理之后：http://192.168.80.132/image/1.jpg

MogileFS是由三个组件组成的：

　　1、tracker：MogileFS的核心，是一个调度器，服务进程为mogilefsd，职责：删除数据、复制数据、监控、查询等。

　　2、database：为tracker存储元素据

　　3、数据存储的位置，通常是一个HTTP（webDAV）服务器，用来做数据的创建（put）、删除（delete）、获取（get），监听端口7500, storage节点使用http进行数据传输, 依赖于perbal, 进程为mogstored。

理想中模型：

实验架构：

实验中用到的rpm包：http://pan.baidu.com/s/1c1S0dJM

MariaDB节点配置：

对各mogilefs节点的主机名进行解析：

[root@www ~]# vim /etc/hosts
     192.168.80.136 mog1.daixiang.com
     192.168.80.137 mog2.daixiang.com
     192.168.80.138 mog3.daixiang.com
                           

二进制包安装MariaDB:

[root@mariadb mysql]# useradd -r -s /sbin/nologin mysql
[root@mariadb ~]# tar xf mariadb-10.1.14-linux-x86_64.tar.gz -C /usr/local/
[root@mariadb ~]# ln -sv /usr/local/mariadb-10.1.14-linux-x86_64 /usr/local/mysql
[root@mariadb mysql]# chown -R mysql.mysql /usr/local/mysql/
[root@mariadb mysql]# ./scripts/mysql_install_db --user=mysql --datadir=/data
[root@mariadb mysql]# cp support-files/mysql.server /etc/rc.d/init.d/mysqld
[root@mariadb mysql]# chmod +x /etc/rc.d/init.d/mysqld
[root@mariadb mysql]# cp support-files/my-large.cnf /etc/my.cnf
[root@mariadb mysql]# vim /etc/my.cnf
         datadir = /data
[root@mariadb mysql]# ln -sv /usr/local/mysql/include/ /usr/include/mysql/
[root@mariadb mysql]# vim /etc/ld.so.conf.d/mysql.conf
         /usr/local/mysql/lib
[root@mariadb mysql]# vim /etc/profile.d/mysql.sh
         export PATH=/usr/local/mysql/bin:$PATH
[root@mariadb mysql]# ldconfig
[root@mariadb mysql]# ldconfig -p | grep mysql
	libmysqld.so.18 (libc6,x86-64) => /usr/local/mysql/lib/libmysqld.so.18
	libmysqld.so (libc6,x86-64) => /usr/local/mysql/lib/libmysqld.so
	libmysqlclient_r.so.16 (libc6,x86-64) => /usr/lib64/mysql/libmysqlclient_r.so.16
	libmysqlclient.so.18 (libc6,x86-64) => /usr/local/mysql/lib/libmysqlclient.so.18
	libmysqlclient.so.16 (libc6,x86-64) => /usr/lib64/mysql/libmysqlclient.so.16
	libmysqlclient.so (libc6,x86-64) => /usr/local/mysql/lib/libmysqlclient.so
	libgalera_smm.so (libc6,x86-64) => /usr/local/mysql/lib/libgalera_smm.so

　　

对用户进行授权：

MariaDB [(none)]> grant all on *.* to 'root'@'192.168.80.%' identified by 'rootpass';
Query OK, 0 rows affected (0.06 sec)

MariaDB [(none)]> grant all on mogilefs.* to 'moguser'@'192.168.80.%' identified by 'mogpass';
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> flush privileges;
Query OK, 0 rows affected (0.00 sec)

　　

MogileFS配置：

[root@mog1 ~]# yum install *.rpm -y
[root@mog1 ~]# yum install perl-IO-AIO -y

[root@mog2 ~]# yum install *.rpm -y
[root@mog2 ~]# yum install perl-IO-AIO -y

[root@mog3 ~]# yum install *.rpm -y
[root@mog3 ~]# yum install perl-IO-AIO -y　

修改mogilefsd进程的配置文件：

 [root@mog1 ~]# vim /etc/mogilefs/mogilefsd.conf

 # Enable daemon mode to work in background and use syslog
 daemonize =                                            #以守护进程的形式运行
 # Where to store the pid of the daemon (must be the same in the init script)
 pidfile = /var/run/mogilefsd/mogilefsd.pid
 # Database connection information
 db_dsn = DBI:mysql:mogilefs:host=192.168.80.135      #定义数据库名为mogilefs和数据库服务器地址
 db_user = moguser                                     #定义管理此数据库的用户名
 db_pass = mogpass                                     #定义密码
 # IP:PORT to listen on for mogilefs client requests
 listen = 0.0.0.0:                                  #定义监听的地址和端口
 # Optional, if you don't define the port above.
 conf_port =
 # Number of query workers to start by default.
 query_jobs =                                          #定义启动查询线程个数
 # Number of delete workers to start by default.
 delete_jobs =                                           #定义启动删除线程个数
 # Number of replicate workers to start by default.
 replicate_jobs =                                        #定义启动复制线程个数
 # Number of reaper workers to start by default.
 # (you don't usually need to increase this)
 reaper_jobs =                             #响应客户端请求，在磁盘失败后将请求重新放到队列中
 # Number of fsck workers to start by default.
 # (these can cause a lot of load when fsck'ing)
 #fsck_jobs =                               #对磁盘进行检测，默认没有启动
 # Minimum amount of space to reserve in megabytes
 # default:                                #默认保留100M空间
 # Consider setting this to be larger than the largest file you
 # would normally be uploading.
 #min_free_space =                 #最小空闲空间为200M
 # Number of seconds to wait for a storage node to respond.
 # default:                                    #等待存储节点相应的时间，默认为2s
 # Keep this low, so busy storage nodes are quickly ignored.
 #node_timeout =                         #等待节点的超时时长
 # Number of seconds to wait to connect to a storage node.
 # default:                                      #连接存储节点的超时时长
 # Keep this low so overloaded nodes get skipped.
 #conn_timeout =
 # Allow replication to use the secondary node get port,
 # if you have apache or similar configured for GET's
 #repl_use_get_port = 

设置tracker对应的数据库，生成mogilefs库：

[root@mog1 ~]# mogdbsetup --dbhost=192.168.80.135 --dbport=3306 --dbname=mogilefs --dbrootuser=root --dbrootpass=rootpass --dbuser=moguser --dbpass=mogpass --yes

#多个tracker共享一个库，所以其他tracker节点上就不需要在做此操作，直接修改配置文件/etc/mogilefs/mogilefs.conf就可以

　　　　注意：此处有一bug，本人也不知道是什么原因造成的：在生成mogilefs库的时候会出现错误，说root用户没对mogilefs库的访问权限（“Failed to grant privileges: Access denied for user 'root'@'192.168.80.%' to database 'mogilefs'”），但是再一次执行上面的设置数据库的命令时就会成功。前提是确保mariadb对用户授权是正确的。如果有哪位大神知道是什么原因造成的，求指点。。　　

 到MariaDB节点上查看mogilefs库是否成功生成：

[root@Mariadb ~]# mysql　　

 MariaDB [(none)]> show databases;
 +--------------------+
 | Database           |
 +--------------------+
 | information_schema |
 | mogilefs           |
 | mysql              |
 | performance_schema |
 | test               |
 +--------------------+
  rows in set (0.00 sec)

 MariaDB [(none)]> use mogilefs;
 Database changed
 MariaDB [mogilefs]> show tables;
 +----------------------+
 | Tables_in_mogilefs   |
 +----------------------+
 | checksum             |
 | class                |
 | device               |
 | domain               |
 | file                 |
 | file_on              |
 | file_on_corrupt      |
 | file_to_delete       |
 | file_to_delete2      |
 | file_to_delete_later |
 | file_to_queue        |
 | file_to_replicate    |
 | fsck_log             |
 | host                 |
 | server_settings      |
 | tempfile             |
 | unreachable_fids     |
 +----------------------+
  rows in set (0.00 sec)

启动mogilefsd进程：

[root@mog1 ~]# service mogilefsd start

 [root@mog1 ~]# ss -tnl | grep ''
 LISTEN                                  *:                     *:*  

对mogstored进程进行配置：

[root@mog1 ~]# vim /etc/mogilefs/mogstored.conf

maxconns =
httplisten = 0.0.0.0:
mgmtlisten = 0.0.0.0:
docroot = /dfs/mogdata

将mogilefsd进程和mogstored进程的配置文件都同步到其他mog节点上去：

[root@mog1 ~]# scp /etc/mogilefs/mogilefsd.conf 192.168.80.137:/etc/mogilefs/
[root@mog1 ~]# scp /etc/mogilefs/mogilefsd.conf 192.168.80.138:/etc/mogilefs/

[root@mog1 ~]# scp /etc/mogilefs/mogstored.conf 192.168.80.137:/etc/mogilefs/
[root@mog1 ~]# scp /etc/mogilefs/mogstored.conf 192.168.80.138:/etc/mogilefs/

在存储节点上创建挂载点：

[root@mog1 ~]# mkdir /dfs/mogdata/dev1
[root@mog1 ~]# chown -R mogilefs.mogilefs /dfs/mogdata/

[root@mog2 ~]# mkdir /dfs/mogdata/dev2 -pv
[root@mog2 ~]# chown -R mogilefs.mogilefs /dfs/mogdata/

[root@mog3 ~]# mkdir /dfs/mogdata/dev3 -pv
[root@mog3 ~]# chown -R mogilefs.mogilefs /dfs/mogdata/

    注意：在生产环境中，需要将存储磁盘挂载在/dfs/mogdata目录下，当块磁盘挂载成功之后，需要在磁盘上创建共享块设备,即这里博主就不挂载磁盘了，直接创建共享块设备。
            挂载磁盘的配置方法：
                          [root@mog1 ~]# mkdir /dfs/mogdata/
                          [root@mog1 ~]# mount -t ext4 /dev/sdb1 /dfs/mogdata/
                          [root@mog1 ~]# mkdir /dfs/mogdata/dev1
                          [root@mog1 ~]# chown -R mogilefs,mogilefs /dfs/mogdata/

启动mogstored进程：

[root@mog1 ~]# service mogstored start
[root@mog2 ~]# service mogstored start
[root@mog3 ~]# service mogstored start

[root@mog1 ~]# ss -tnlp | grep 'mogstored'
LISTEN                                  *:                     *:*      users:(("mogstored",,))
LISTEN                                  *:                     *:*      users:(("mogstored",,))

[root@mog2 ~]# ss -tnlp | grep 'mogstored'
LISTEN                                  *:                     *:*      users:(("mogstored",,))
LISTEN                                  *:                     *:*      users:(("mogstored",,))

[root@mog3 ~]# ss -tnlp | grep 'mogstored'
LISTEN                                  *:                     *:*      users:(("mogstored",,))
LISTEN                                  *:                     *:*      users:(("mogstored",,))

　　

对各节点进行管理：

[root@mog1 ~]# mogadm --trackers=192.168.80.136:7001 host add mog1 --ip=192.168.80.136 --status=alive
[root@mog1 ~]# mogadm --trackers=192.168.80.136:7001 device add mog1 001 --status=alive

[root@mog1 ~]# mogadm --trackers=192.168.80.136:7001 host add mog2 --ip=192.168.80.137 --status=alive
[root@mog1 ~]# mogadm --trackers=192.168.80.136:7001 device add mog2 002 --status=alive

[root@mog1 ~]# mogadm --trackers=192.168.80.136:7001 host add mog3 --ip=192.168.80.138 --status=alive
[root@mog1 ~]# mogadm --trackers=192.168.80.136:7001 device add mog3 003 --status=alive

[root@mog1 ~]# mogadm domain add linux1
[root@mog1 ~]# mogadm domain add python1

[root@mog1 ~]# mogadm class add linux1 class1 --mindevcount=3
[root@mog1 ~]# mogadm class add linux1 class2 --mindevcount=2
[root@mog1 ~]# mogadm class add python1 dx1 --mindevcount=2

[root@mog1 ~]# mogadm check

Checking trackers...
  127.0.0.1: ... OK

Checking hosts...
  [ ] mog1 ... OK
  [ ] mog2 ... OK
  [ ] mog3 ... OK

Checking devices...
  host device         size(G)    used(G)    free(G)   use%   ob state   I/O%
  ---- ------------ ---------- ---------- ---------- ------ ---------- -----
  [ ] dev1            16.509      3.901     12.608  23.63%  writeable   4.4
  [ ] dev2            16.509      3.901     12.608  23.63%  writeable   0.3
  [ ] dev3            16.509      3.897     12.612  23.60%  writeable   0.0
  ---- ------------ ---------- ---------- ---------- ------
             total:    49.527     11.699     37.828  23.62%

[root@mog1 ~]# mogadm domain list
 domain               class                mindevcount   replpolicy   hashtype
-------------------- -------------------- ------------- ------------ -------
 linux1               class1                            MultipleHosts() NONE
 linux1               class2                            MultipleHosts() NONE
 linux1               default                           MultipleHosts() NONE   

 python1              default                           MultipleHosts() NONE
 python1              dx1                               MultipleHosts() NONE   

　　详细介绍其查看上一篇博客："分布式存储之MogileFS分布式文件系统简单应用"

上传文件进行测试：

[root@mog1 ~]# mogupload --trackers=192.168.80.136:7001 --domain=linux1 --key='/1.jpg' --file='/root/centos.jpg'
[root@mog1 ~]# mogfileinfo --tracker=192.168.80.137:7001 --domain=linux1 --key='/1.jpg'

[root@mog1 ~]# mogupload --trackers=192.168.80.136:7001 --domain=python1 --key='/fstab.html' --file='/etc/fstab'
[root@mog1 ~]# mogfileinfo --tracker=192.168.80.137:7001 --domain=python1 --key='/fstab.html'　　

[root@mog1 ~]# mogfileinfo --tracker=192.168.80.137: --domain=linux1 --key='1.jpg'
- file: .jpg
     class:              default
  devcount:
    domain:               linux1
       fid:
       key:                .jpg
    length:
 - http://192.168.80.137:7500/dev2/0/000/000/0000000005.fid
 - http://192.168.80.136:7500/dev1/0/000/000/0000000005.fid

配置Nginx:

[root@nginx ~]# # yum groupinstall "Development Tools" "Server Platform Deveopment" -y
[root@nginx ~]# # yum install openssl-devel pcre-devel -y
[root@nginx ~]# useradd -r nginx
[root@nginx ~]# tar xf nginx_mogilefs_module-1.0.4.tar.gz
[root@nginx ~]# tar xf nginx-1.10.0.tar.gz
[root@nginx ~]# cd nginx-1.10.0
[root@nginx nginx-1.10.0]#  ./configure \
  --prefix=/usr/local/nginx \
  --sbin-path=/usr/sbin/nginx \
  --conf-path=/etc/nginx/nginx.conf \
  --error-log-path=/var/log/nginx/error.log \
  --http-log-path=/var/log/nginx/access.log \
  --pid-path=/var/run/nginx/nginx.pid  \
  --lock-path=/var/lock/nginx.lock \
  --user=nginx \
  --group=nginx \
  --with-http_ssl_module \
  --with-http_flv_module \
  --with-http_stub_status_module \
  --with-http_gzip_static_module \
  --http-client-body-temp-path=/var/tmp/nginx/client/ \
  --http-proxy-temp-path=/var/tmp/nginx/proxy/ \
  --http-fastcgi-temp-path=/var/tmp/nginx/fcgi/ \
  --http-uwsgi-temp-path=/var/tmp/nginx/uwsgi \
  --http-scgi-temp-path=/var/tmp/nginx/scgi \
  --with-pcre \
  --with-debug \
  --add-module=/root/nginx_mogilefs_module-1.0.4    #添加的第三方模块，实现直接使用key作为uri进行访问

[root@nginx nginx-1.10.0]#  make && make install

　　

提供nginx启动脚本：

[root@nginx ~]# vim /etc/rc.d/init.d/nginx

#!/bin/sh
#
# nginx - this script starts and stops the nginx daemon
#
# chkconfig:   -
# description:  Nginx is an HTTP(S) server, HTTP(S) reverse \
#               proxy and IMAP/POP3 proxy server
# processname: nginx
# config:      /etc/nginx/nginx.conf
# config:      /etc/sysconfig/nginx
# pidfile:     /var/run/nginx.pid

# Source function library.
. /etc/rc.d/init.d/functions

# Source networking configuration.
. /etc/sysconfig/network

# Check that networking is up.
[ "$NETWORKING" = "no" ] && exit 

nginx="/usr/sbin/nginx"
prog=$(basename $nginx)

NGINX_CONF_FILE="/etc/nginx/nginx.conf"

[ -f /etc/sysconfig/nginx ] && . /etc/sysconfig/nginx

lockfile=/var/lock/subsys/nginx

make_dirs() {
   # make required directories
   user=`nginx -V >& | grep "configure arguments:" | sed 's/[^*]*--user=\([^ ]*\).*/\1/g' -`
   options=`$nginx -V >& | grep 'configure arguments:'`
   for opt in $options; do
       if [ `echo $opt | grep '.*-temp-path'` ]; then
           value=`echo $opt | cut -d "=" -f `
           if [ ! -d "$value" ]; then
               # echo "creating" $value
               mkdir -p $value && chown -R $user $value
           fi
       fi
   done
}

start() {
    [ -x $nginx ] || exit
    [ -f $NGINX_CONF_FILE ] || exit
    make_dirs
    echo -n $"Starting $prog: "
    daemon $nginx -c $NGINX_CONF_FILE
    retval=$?
    echo
    [ $retval -eq  ] && touch $lockfile
    return $retval
}

stop() {
    echo -n $"Stopping $prog: "
    killproc $prog -QUIT
    retval=$?
    echo
    [ $retval -eq  ] && rm -f $lockfile
    return $retval
}

restart() {
    configtest || return $?
    stop
    sleep
    start
}

reload() {
    configtest || return $?
    echo -n $"Reloading $prog: "
    killproc $nginx -HUP
    RETVAL=$?
    echo
}

force_reload() {
    restart
}

configtest() {
  $nginx -t -c $NGINX_CONF_FILE
}

rh_status() {
    status $prog
}

rh_status_q() {
    rh_status >/dev/null >&
}

case "$1" in
    start)
        rh_status_q && exit
        $
        ;;
    stop)
        rh_status_q || exit
        $
        ;;
    restart|configtest)
        $
        ;;
    reload)
        rh_status_q || exit
        $
        ;;
    force-reload)
        force_reload
        ;;
    status)
        rh_status
        ;;
    condrestart|try-restart)
        rh_status_q || exit
            ;;
    *)
        echo $"Usage: $0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload|configtest}"
        exit
esac

[root@nginx ~]# chmod +x /etc/rc.d/init.d/nginx　　

修改nginx配置文件如下：

http {

    upstream images {
        server 192.168.80.136:;
        server 192.168.80.137:;
        server 192.168.80.138:;
    }

    server {
        listen       ;
        server_name  localhost;
        location /image {
            mogilefs_tracker images;
            mogilefs_domain linux1;
            mogilefs_methods GET;
            mogilefs_noverify on;
            mogilefs_pass {
                proxy_pass $mogilefs_path;
                proxy_hide_header Content-Type;
                proxy_buffering off;
            }
        }

        location /files {
            mogilefs_tracker images;
            mogilefs_domain python1;
            mogilefs_methods GET;
            mogilefs_noverify on;
            mogilefs_pass {
                proxy_pass $mogilefs_path;
                proxy_hide_header Content-Type;
                proxy_buffering off;
            }
        }
    }
}   

启动nginx：

[root@nginx ~]# service nginx start

访问测试：