Minio 集群扩容存储空间，配合nginx 负载反向代理后端minio 集群服务器，提升高可用性

环境：Centos  7

软件：minio,Etcd

需求：通过联盟两个集群实例，实现水平扩容存储空间问题；

服务器使用阿里云，一共4台服务器(官方说明最好4台服务器做分布式，测试节省服务器所以我们使用两台服务器每台服务器两个目录，做一个假服务器代替，真实环境条件允许最好4台服务器操作)：

集群	Minio-1	Minio-2
IP 地址	172.31.78.244 172.31.78.243	172.31.78.241 172.31.78.242
计算名称地址	topsysfile.server.1.itusye.co topsysfile.server.2.itusye.co	topsysfile.server.3.itusye.co topsysfile.server.4.itusye.co
目录地址	/Data/minio/{data1,data2}	/Data/minio/{data1,data2}

官方说明：
可扩容：不同MinIO集群可以组成联邦，并形成一个全局的命名空间，并跨越多个数据中心，但不支持动态扩容；

https://docs.min.io/docs/minio-federation-quickstart-guide.html  官方参考 文档

1. 由于多个集群使用Etcd 联邦，所以我们先需要安装配置ETCD 服务，etcd 与zookeeper 服务器都是奇数性3，5，7 模式增加，etcd 安装配置我就简单跳过了，列出安装与配置；

[root@Minio-Storage-Services ~]# yum -y install etcd
[root@Minio-Storage-Services ~]# cat /etc/etcd/etcd.conf | grep -v '#'
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="http://172.31.78.244:2380"  #内网通迅
ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"     #外网通迅
ETCD_NAME="top-etcd-01"    #这里是集群连接地址，请注意与cluster 对应，与zookeeper myid  是一样的原理
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://172.31.78.244:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://0.0.0.0:2379"
ETCD_INITIAL_CLUSTER="top-etcd-01=http://172.31.78.244:2380,top-etcd-02=http://172.31.78.243:2380,top-etcd-03=http://172.31.78.241:2380"     #etcd 集群连接分配地址，注意三台服务器一样
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_STRICT_RECONFIG_CHECK="true"

以上服务器配置是172.31.78.244 这台

: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu  qdisc pfifo_fast state UP group default qlen
    link/ether ::3e::: brd ff:ff:ff:ff:ff:ff
    inet 172.31.78.243/ brd 172.31.79.255 scope global dynamic eth0
       valid_lft 315039419sec preferred_lft 315039419sec
[root@Minio-Storage-Services ~]# cat /etc/etcd/etcd.conf  | grep -v '#'
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="http://172.31.78.243:2380"
ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"
ETCD_NAME="top-etcd-02"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://172.31.78.243:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://0.0.0.0:2379"
ETCD_INITIAL_CLUSTER="top-etcd-01=http://172.31.78.244:2380,top-etcd-02=http://172.31.78.243:2380,top-etcd-03=http://172.31.78.241:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_STRICT_RECONFIG_CHECK="true"
[root@Minio-Storage-Services ~]# 

: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu  qdisc pfifo_fast state UP group default qlen
    link/ether ::3e::: brd ff:ff:ff:ff:ff:ff
    inet 172.31.78.241/ brd 172.31.79.255 scope global dynamic eth0
       valid_lft 315040516sec preferred_lft 315040516sec
[root@Minio-Storage-Services ~]# cat /etc/etcd/etcd.conf | grep -v '#'
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="http://172.31.78.241:2380"
ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"
ETCD_NAME="top-etcd-03"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://172.31.78.241:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://0.0.0.0:2379"
ETCD_INITIAL_CLUSTER="top-etcd-01=http://172.31.78.244:2380,top-etcd-02=http://172.31.78.243:2380,top-etcd-03=http://172.31.78.241:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_STRICT_RECONFIG_CHECK="true"
[root@Minio-Storage-Services ~]# 

etcd 已安装完成；

2.  下载minio , 4台配置服务器同样的配置如下；

[root@Minio-Storage-Services ~]#  wget https://dl.min.io/server/minio/release/linux-amd64/minio
[root@Minio-Storage-Services ~]# chmod +x minio
[root@Minio-Storage-Services ~]# mv minio /usr/local/bin/
[root@Minio-Storage-Services ~]# mkdir /usr/local/minio{bin,config} -p
[root@Minio-Storage-Services ~]# mkdir /Data/minio/{data1,data2} -p
[root@Minio-Storage-Services ~]# firewall-cmd --zone=public --add-port=/tcp --add-port=/tcp --permanent && firewall-cmd --reload
[root@Minio-Storage-Services ~]# firewall-cmd --zone=public --add-port=/tcp --permanent
[root@Minio-Storage-Services ~]# systemctl restart firewalld

3. 编辑启动脚本，两个集群，两份配置不一样，由于我配置了hosts，所以etcd 连接使用了域名

[root@Minio-Storage-Services ~]# more /etc/hosts
::     localhost       localhost.localdomain   localhost6      localhost6.localdomain6
127.0.0.1       localhost       localhost.localdomain   localhost4      localhost4.localdomain4

172.31.78.241   Minio-Storage-Services  Minio-Storage-Services

172.31.78.244   topsysfile.server..itusye.co
172.31.78.243   topsysfile.server..itusye.co
172.31.78.241   topsysfile.server..itusye.co
172.31.78.242   topsysfile.server..itusye.co

[root@Minio-Storage-Services ~]# 

Minio-1集群配置，两台服务器同样的配置；

: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu  qdisc pfifo_fast state UP group default qlen
    link/ether ::3e:::6b brd ff:ff:ff:ff:ff:ff
    inet 172.31.78.244/ brd 172.31.79.255 scope global dynamic eth0
       valid_lft 315039437sec preferred_lft 315039437sec
[root@Minio-Storage-Services ~]# more /usr/local/minio/bin/run.sh
#!/bin/bash
export MINIO_ACCESS_KEY=topsysfile
export MINIO_SECRET_KEY=Hao123baidu!

export MINIO_ETCD_ENDPOINTS="http://topsysfile.server.1.海itusye.co:2380,http://topsysfile.server.2.itusye.co:2380,http://topsysfile.server.3.itusye.co:2380"

export MINIO_DOMAIN=topfile.storage.minio.itusye.co
export MINIO_PUBLIC_IPS=172.31.78.244,172.31.78.243

minio server --config-dir /usr/local/minio/config/ \
http://172.31.78.244/Data/minio/data1 \
http://172.31.78.244/Data/minio/data2 \
http://172.31.78.243/Data/minio/data1 \
http://172.31.78.243/Data/minio/data2

Minio-2 集群配置，两台服务器同样的配置；

       valid_lft forever preferred_lft forever
: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu  qdisc pfifo_fast state UP group default qlen
    link/ether ::3e::: brd ff:ff:ff:ff:ff:ff
    inet 172.31.78.241/ brd 172.31.79.255 scope global dynamic eth0
       valid_lft 315034589sec preferred_lft 315034589sec
[root@Minio-Storage-Services ~]# more /usr/local/minio/bin/run.sh
#!/bin/bash
export MINIO_ACCESS_KEY=topsysfile   # 登录账号
export MINIO_SECRET_KEY=Hao123baidu!  # 登录密码

export MINIO_ETCD_ENDPOINTS="http://topsysfile.server.1.itusye.co:2380,http://topsysfile.server.2.itusye.co:2380,http://topsysfile.server.3.itusye.co:2380"   # 配置etcd 连接
export MINIO_DOMAIN=topfile.storage.minio.itusye.co  #此处需特别注意：此域名需要配置与nginx 配置反向代理时使用的域名一致，不然比如分享会直接访问到xml, 开发连接也会有问题，
export MINIO_PUBLIC_IPS=172.31.78.241,172.31.78.242 # 此处需要配置上Minio- 集群的所有ip ,可配置内网外网，因为我测试使用内网通迅，此处配置内网地址了

minio server --config-dir /usr/local/minio/config/ \   # 官方提示：--config-dir 选项要停止使用了，此选择主要创建一个certs目录存放ca 信息
http://172.31.78.241/Data/minio/data1 \     # 以下信息是minio 分布式集群minio-1 启动情况
http://172.31.78.241/Data/minio/data2 \
http://172.31.78.242/Data/minio/data1 \
http://172.31.78.242/Data/minio/data2
[root@Minio-Storage-Services ~]# 

4 配置启动脚本，所有服务器同样的配置；

[root@Minio-Storage-Services ~]# more /usr/lib/systemd/system/minio.service
[Unit]
Description=Minio service
Documentation=https://docs.minio.io/

[Service]
WorkingDirectory=/usr/local/minio/
ExecStart=/usr/local/minio/bin/run.sh

Restart=on-failure
RestartSec=

[Install]
WantedBy=multi-user.target
[root@Minio-Storage-Services ~]# 

5.配置nginx , https://docs.min.io/docs/setup-nginx-proxy-with-minio.html   官方参考文档

主要需要注意：客户端上传限制的问题

client_max_body_size 2048M;
client_body_buffer_size 500M;

以下为nginx Server 模块配置

upstream topfilestorage-minio{
                ip_hash;
                server topsysfile.server..toptech.ph: weight= max_fails= fail_timeout=10s;
                server topsysfile.server..toptech.ph: weight= max_fails= fail_timeout=10s;
                server topsysfile.server..toptech.ph: weight= max_fails= fail_timeout=10s;

                server topsysfile.server..toptech.ph: weight= max_fails= fail_timeout=10s;
                #       check interval= rise= fall= timeout= default_down=true type=http;

}

server {
        listen ;
        server_name localhost;
        if ($scheme = http ) {
                return  https://$host$request_uri;
                }
        }

server {
         server_name localhost;
                listen ;
                ssl on;
                access_log  /home/logs/nginx/www_logs/web_access.log main;
                error_log  /home/logs/nginx/error.log;
                ssl_certificate /usr/local/nginx/ssl/topfile.crt;
                ssl_certificate_key     /usr/local/nginx/ssl/topfile.key;
                ssl_prefer_server_ciphers on;
                ssl_protocols   TLSv1 TLSv1. TLSv1.;
                ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH:ECDHE-RSA-AES128-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA128:DHE-RSA-A
ES128-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA128:ECDHE-RSA-AES128-SHA384:ECDHE-RSA-AES128-SHA128:ECDHE-RSA-AES128-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES
-SHA128:DHE-RSA-AES128-SHA128:DHE-RSA-AES128-SHA:DHE-RSA-AES128-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA384:AES128-GCM-SHA128:AES128-SHA128:AES12
-SHA128:AES128-SHA:AES128-SHA:DES-CBC3-SHA:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4";
                ssl_ecdh_curve secp384r1;
                error_page   /.html;
                location = /.html {
                        root  /var/www/;
                }
                ignore_invalid_headers off;
                proxy_buffering off;

                location / {
                        proxy_pass http://topfilestorage-minio;
                        include conf.d/proxy.conf;
                }

}

以下proxy.conf 文档

[root@Minio-Storage-Services ~]# more /usr/local/nginx/config/conf.d/proxy.conf | grep -v '#'
proxy_http_version 1.1;

proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $http_host;

proxy_set_header Connection "";
chunked_transfer_encoding off;

[root@Minio-Storage-Services ~]# 

以上集群已配置完成，测试新建文件夹会自动选择集群存储，并且测试如果固定用一个集群创建一个根目录文件夹，再用另外一个集群或者反向代理去上传都是可以把数据上传存储到文件夹所在的集群空间里，两个集群内的数据并不会同步，但是在存储时minio 会自动计算选择集群存储；

MinIO是一个开源的对象存储解决方案，优点如下：

1. 高性能：作为高性能对象存储，在标准硬件条件下它能达到55GB/s的读、35GG/s的写速率
2. 可扩容：不同MinIO集群可以组成联邦，并形成一个全局的命名空间，并跨越多个数据中心
3. 云原生：容器化、基于K8S的编排、多租户支持
4. 兼容性：兼容S3 API这一事实上的对象存储标准，最先支持S3 Select
5. 简单：这一设计原则让MinIO不容易出错、更快启动
6. 支持纠删码：MinIO使用纠删码、Checksum来防止硬件错误和静默数据污染。在最高冗余度配置下，即使丢失1/2的磁盘也能恢复数据