对象存储 - Swift 原理 及 Swift+keystone+dashboard 架构搭建

1. 原理介绍

Swift 架构、原理及功能： http://www.cnblogs.com/sammyliu/p/4955241.html

　总结的很详细也很全面，受益匪浅，感谢分享。

2. keystone + swift + dashboard 安装和配置

2.1 基础环境介绍

vmware 11.0
    系统：rhel7.2
    openstack版本：openstack-Mitaka

swift-controller:    192.168.0.11
    swift-object1:      192.168.0.51
    swift-object2:      192.168.0.52

注意：这里的第一张网卡nat作为内网网卡，第二张网卡桥接作为外网网卡（使用一张桥接网卡可实现，这里是为了ip规范使用nat网络）

/etc/hosts
192.168.0.11    controller
192.168.0.51    object1
192.168.0.52    object2

时间同步，三节点都执行：

# ntpdate tiger.sina.com.cn

2.2 安装初始化

swift-controller配置：

# yum install python-openstackclient mariadb mariadb-server python2-PyMySQL rabbitmq-server memcached python-memcached -y

配置数据库

# vim /etc/my.cnf.d/openstack.cnf

[mysqld]
bind-address = 192.168.0.11
default-storage-engine = innodb
innodb_file_per_table
max_connections =
collation-server = utf8_general_ci
character-set-server = utf8

启动及初始化数据库

# systemctl enable mariadb ; systemctl start mariadb
# mysql_secure_installation        # 密码 

启动消息队列 rabbitmq服务

# systemctl enable rabbitmq-server ; systemctl start rabbitmq-server

添加用户openstack的用户并赋权

# rabbitmqctl add_user openstack openstack
# rabbitmqctl set_permissions openstack ".*" ".*" ".*"

启动memcache服务

# systemctl enable memcached ; systemctl start memcached

查看服务启动情况

: mariadb
：memcache
：rabbitmq

# netstat -ntplu | egrep "3306|11211|2567"
tcp               0.0.0.0:           0.0.0.0:*               LISTEN      /beam.smp
tcp               192.168.0.11:       0.0.0.0:*               LISTEN      /mysqld
tcp               127.0.0.1:         0.0.0.0:*               LISTEN      /memcached
tcp6              :::               :::*                    LISTEN      /memcached
udp               127.0.0.1:         0.0.0.0:*                           /memcached
udp6              :::               :::*                                /memcached    

2.3 keystone安装配置

创建keystone数据库并授权

# mysql -p123456
MariaDB [(none)]> create database keystone;
MariaDB [(none)]> GRANT ALL PRIVILEGES ON keystone.* TO 'keystone'@'localhost' IDENTIFIED BY 'keystone';
MariaDB [(none)]> GRANT ALL PRIVILEGES ON keystone.* TO 'keystone'@'%' IDENTIFIED BY 'keystone';

安装程序包

# yum install openstack-keystone httpd mod_wsgi -y

配置keystone.conf

# vim /etc/keystone/keystone.conf

[DEFAULT]
...
admin_token = 2b64e54cdce5900a22f8
...
[database]
...
connection = mysql+pymysql://keystone:keystone@controller/keystone
...
[token]
...
provider = fernet
...

初始化 keystone 数据库

# su -s /bin/sh -c "keystone-manage db_sync" keystone

初始化Fernet keys：

# keystone-manage fernet_setup --keystone-user keystone --keystone-group keystone

配置apache

# vim /etc/httpd/conf/httpd.conf
...
ServerName controller
...

配置openstack使用的虚拟主机：

# vim /etc/httpd/conf.d/wsgi-keystone.conf

Listen
Listen 

<VirtualHost *:>
    WSGIDaemonProcess keystone-public processes= threads= user=keystone group=keystone display-name=%{GROUP}
    WSGIProcessGroup keystone-public
    WSGIScriptAlias / /usr/bin/keystone-wsgi-public
    WSGIApplicationGroup %{GLOBAL}
    WSGIPassAuthorization On
    ErrorLogFormat "%{cu}t %M"
    ErrorLog /var/log/httpd/keystone-error.log
    CustomLog /var/log/httpd/keystone-access.log combined

    <Directory /usr/bin>
        Require all granted
    </Directory>
</VirtualHost>

<VirtualHost *:>
    WSGIDaemonProcess keystone-admin processes= threads= user=keystone group=keystone display-name=%{GROUP}
    WSGIProcessGroup keystone-admin
    WSGIScriptAlias / /usr/bin/keystone-wsgi-admin
    WSGIApplicationGroup %{GLOBAL}
    WSGIPassAuthorization On
    ErrorLogFormat "%{cu}t %M"
    ErrorLog /var/log/httpd/keystone-error.log
    CustomLog /var/log/httpd/keystone-access.log combined

    <Directory /usr/bin>
        Require all granted
    </Directory>
</VirtualHost>

配置完成，启动服务

# systemctl enable httpd ; systemctl start httpd

配置认证令牌、端点URL、api版本

# export OS_TOKEN=2b64e54cdce5900a22f8
# export OS_URL=http://controller:35357/v3
# export OS_IDENTITY_API_VERSION=

（1）创建keystone服务

# openstack service create --name keystone --description "OpenStack Identity" identity

（2）创建api端点

# openstack endpoint create --region RegionOne identity public http://controller:5000/v3
# openstack endpoint create --region RegionOne identity internal http://controller:5000/v3
# openstack endpoint create --region RegionOne identity admin http://controller:35357/v3

（3）创建域

# openstack domain create --description "Default Domain" default

（4）创建项目

# openstack project create --domain default   --description "Admin Project" admin

（5）创建用户（admin密码：admin）

# openstack user create --domain default   --password-prompt admin

（6）创建角色

# openstack role create admin

（7）添加角色到项目和用户上

# openstack role add --project admin --user admin admin

（8）创建 service 项目

# openstack project create --domain default   --description "Service Project" service

验证：

# unset OS_TOKEN OS_URL
# openstack --os-auth-url http://controller:35357/v3 \
--os-project-domain-name default --os-user-domain-name default \
--os-project-name admin --os-username admin token issue
# 密码上面设置： admin

创建认证脚本

# vim admin-openrc

export OS_PROJECT_DOMAIN_NAME=default
export OS_USER_DOMAIN_NAME=default
export OS_PROJECT_NAME=admin
export OS_USERNAME=admin
export OS_PASSWORD=admin
export OS_AUTH_URL=http://controller:35357/v3
export OS_IDENTITY_API_VERSION=
export OS_IMAGE_API_VERSION=

测试认证

# . admin-openrc
# openstack token issue

2.4 对象存储 Swift 安装和配置

2.4.1 swift-controller 配置

# . admin-openrc

（1）创建 swift用户，给swift用户添加admin角色

# openstack user create --domain default --password-prompt swift
密码：swift
# openstack role add --project service --user swift admin

（2）创建swift服务

# openstack service create --name swift   --description "OpenStack Object Storage" object-store

（3）创建对象存储服务 API 端点

# openstack endpoint create --region RegionOne   object-store public http://controller:8080/v1/AUTH_%\(tenant_id\)s
# openstack endpoint create --region RegionOne   object-store internal http://controller:8080/v1/AUTH_%\(tenant_id\)s
# openstack endpoint create --region RegionOne   object-store admin http://controller:8080/v1

安装 swift 程序包

yum install openstack-swift-proxy python-swiftclient python-keystoneclient python-keystonemiddleware memcached -y

获取代理服务的配置文件：

# curl -o /etc/swift/proxy-server.conf https://git.openstack.org/cgit/openstack/swift/plain/etc/proxy-server.conf-sample

修改如下内容：

# vim /etc/swift/proxy-server.conf
[DEFAULT]
bind_port =
user = swift
swift_dir = /etc/swift
...
[pipeline:main]
pipeline = catch_errors gatekeeper healthcheck proxy-logging cache container_sync bulk ratelimit authtoken keystoneauth container-quotas account-quotas slo dlo versioned_writes proxy-logging proxy-server
...
[app:proxy-server]
use = egg:swift#proxy
account_autocreate = True
...
[filter:keystoneauth]
use = egg:swift#keystoneauth
operator_roles = admin,user
...
[filter:authtoken]
paste.filter_factory = keystonemiddleware.auth_token:filter_factory
auth_uri = http://controller:5000
auth_url = http://controller:35357
memcached_servers = 127.0.0.1:
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = swift
password = swift
delay_auth_decision = True
...
[filter:cache]
use = egg:swift#memcache
memcache_servers = 127.0.0.1:

2.4.2 swift-object 配置

以下操作在 object1 和 object2 上执行

    swift-object1:      192.168.0.51
    swift-object2:      192.168.0.52

（1）安装支持工具包

# yum install xfsprogs rsync openstack-swift-account openstack-swift-container openstack-swift-object -y

（2）使用 xfs 格式化磁盘

# mkfs.xfs /dev/sdb
# mkfs.xfs /dev/sdc
# mkfs.xfs /dev/sdd

（3）创建挂载点

# mkdir -pv /srv/node/sd{b,c,d}

（4）编辑 /etc/fstab 添加挂载内容

# vim /etc/fstab
...
/dev/sdb /srv/node/sdb xfs noatime,nodiratime,nobarrier,logbufs=
/dev/sdc /srv/node/sdc xfs noatime,nodiratime,nobarrier,logbufs=
/dev/sdd /srv/node/sdd xfs noatime,nodiratime,nobarrier,logbufs=
# mount -a

（5）创建并编写 /etc/rsyncd.conf 文件

# vim /etc/rsyncd.conf
uid = swift
gid = swift
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
address = 192.168.0.51        # object1为 192.168.0.51 object2为 192.168.0.52

[account]
max connections =
path = /srv/node/
read only = False
lock file = /var/lock/account.lock

[container]
max connections =
path = /srv/node/
read only = False
lock file = /var/lock/container.lock

[object]
max connections =
path = /srv/node/
read only = False
lock file = /var/lock/object.lock

（6）启动 rsyncd 服务并开机启动

# systemctl enable rsyncd.service ; systemctl start rsyncd.service

从仓库下载account、container、object、swift配置文件

# curl -o /etc/swift/account-server.conf https://git.openstack.org/cgit/openstack/swift/plain/etc/account-server.conf-sample
# curl -o /etc/swift/container-server.conf https://git.openstack.org/cgit/openstack/swift/plain/etc/container-server.conf-sample
# curl -o /etc/swift/object-server.conf https://git.openstack.org/cgit/openstack/swift/plain/etc/object-server.conf-sample
# curl -o /etc/swift/swift.conf https://git.openstack.org/cgit/openstack/swift/plain/etc/swift.conf-sample

（1）配置 /etc/swift/account-server.conf

# vim /etc/swift/account-server.conf

[DEFAULT]
bind_ip = 192.168.0.51        # object1为 192.168.0.51 object2为 192.168.0.52
bind_port =
user = swift
swift_dir = /etc/swift
devices = /srv/node
mount_check = True
...
[pipeline:main]
pipeline = healthcheck recon account-server
...
[filter:recon]
use = egg:swift#recon
...
recon_cache_path = /var/cache/swift

（2）配置 /etc/swift/container-server.conf

# vim /etc/swift/container-server.conf
[DEFAULT]
bind_ip = 192.168.0.51        # object1为 192.168.0.51 object2为 192.168.0.52
bind_port =
user = swift
swift_dir = /etc/swift
devices = /srv/node
mount_check = True
...
[pipeline:main]
pipeline = healthcheck recon container-server
...
[filter:recon]
use = egg:swift#recon
...
recon_cache_path = /var/cache/swift

（3）配置 /etc/swift/object-server.conf

# vim /etc/swift/object-server.conf
[DEFAULT]
bind_ip = 192.168.0.51        # object1为 192.168.0.51 object2为 192.168.0.52
bind_port =
user = swift
swift_dir = /etc/swift
devices = /srv/node
mount_check = True
...
[pipeline:main]
pipeline = healthcheck recon object-server
...
[filter:recon]
use = egg:swift#recon
recon_cache_path = /var/cache/swift
recon_lock_path = /var/lock

确认挂载点目录结构是否有合适的所有权

# chown -R swift:swift /srv/node
# mkdir -p /var/cache/swift
# chown -R root:swift /var/cache/swift
# chmod -R  /var/cache/swift

2.4.3 创建，分发并初始化rings

以下操作在 swift-container 上执行

    swift-controller: 192.168.0.11

创建账户ring

切换到 /etc/swift 目录。

# cd /etc/swift/

创建account ring 文件

（1）创建基本 account.builder 文件

# swift-ring-builder account.builder create   

（2）添加每个节点到 ring 中：

object1: 192.168.0.51
# swift-ring-builder account.builder add --region  --zone  --ip 192.168.0.51 --port  --device sdb --weight
# swift-ring-builder account.builder add --region  --zone  --ip 192.168.0.51 --port  --device sdc --weight
# swift-ring-builder account.builder add --region  --zone  --ip 192.168.0.51 --port  --device sdd --weight
object2: 192.168.0.52
# swift-ring-builder account.builder add --region  --zone  --ip 192.168.0.52 --port  --device sdb --weight
# swift-ring-builder account.builder add --region  --zone  --ip 192.168.0.52 --port  --device sdc --weight
# swift-ring-builder account.builder add --region  --zone  --ip 192.168.0.52 --port  --device sdd --weight 

（3）验证 ring 内容

# swift-ring-builder account.builder

（4）平衡 ring

# swift-ring-builder account.builder rebalance

创建container ring 文件

（1）创建基本 container.builder 文件

# swift-ring-builder container.builder create   

（2）添加每个节点到 ring 中：

object1: 192.168.0.51
# swift-ring-builder container.builder add --region  --zone  --ip 192.168.0.51 --port  --device sdb --weight
# swift-ring-builder container.builder add --region  --zone  --ip 192.168.0.51 --port  --device sdc --weight
# swift-ring-builder container.builder add --region  --zone  --ip 192.168.0.51 --port  --device sdd --weight
object2: 192.168.0.52
# swift-ring-builder container.builder add --region  --zone  --ip 192.168.0.52 --port  --device sdb --weight
# swift-ring-builder container.builder add --region  --zone  --ip 192.168.0.52 --port  --device sdc --weight
# swift-ring-builder container.builder add --region  --zone  --ip 192.168.0.52 --port  --device sdd --weight 

（3）验证 ring 内容

# swift-ring-builder container.builder

（4）平衡 ring

# swift-ring-builder container.builder rebalance

创建 object ring 文件

（1）创建基本 container.builder 文件

# swift-ring-builder object.builder create   

（2）添加每个节点到 ring 中：

# swift-ring-builder object.builder add --region  --zone  --ip 192.168.0.51 --port  --device sdb --weight
# swift-ring-builder object.builder add --region  --zone  --ip 192.168.0.51 --port  --device sdc --weight
# swift-ring-builder object.builder add --region  --zone  --ip 192.168.0.51 --port  --device sdd --weight
object2: 192.168.0.52
# swift-ring-builder object.builder add --region  --zone  --ip 192.168.0.52 --port  --device sdb --weight
# swift-ring-builder object.builder add --region  --zone  --ip 192.168.0.52 --port  --device sdc --weight
# swift-ring-builder object.builder add --region  --zone  --ip 192.168.0.52 --port  --device sdd --weight 

（3）验证 ring 内容

# swift-ring-builder object.builder

（4）平衡 ring

# swift-ring-builder object.builder rebalance

将ring配置文件拷贝到每个存储节点，这里也就是 object1 和 object2

# scp *.ring.gz object1:/etc/swift/
# scp *.ring.gz object2:/etc/swift/

配置 /etc/swift/swift.conf 文件

# vim /etc/swift/swift.conf
[swift-hash]
...
swift_hash_path_suffix = mickey
swift_hash_path_prefix = minnie
...

复制 /etc/swift/swift.conf 到其他object节点和代理节点，这里也就是 object1 和 object2

# scp swift.conf object1:/etc/swift/
# scp swift.conf object2:/etc/swift/

确认三个节点配置文件权限，在所有节点执行；

# chown -R root:swift /etc/swift

2.4.4 启动 swift 服务

（1）swift-controller 节点启动服务

# systemctl enable openstack-swift-proxy.service memcached.service; systemctl start openstack-swift-proxy.service memcached.service

问题1：在启动 openstack-swift-proxy.service 服务后，查看启动信息

# systemctl status -l  openstack-swift-proxy.service

Jan  :: controller liberasurecode[]: liberasurecode_backend_open: dynamic linking error libisal.so.: cannot open shared object file: No such file or directory
Jan  :: controller liberasurecode[]: liberasurecode_backend_open: dynamic linking error libshss.so.: cannot open shared object file: No such file or directory

缺少库文件，需要编译安装：

# yum install gcc gcc-c++ make automake autoconf libtool yasm -y
# cd /usr/local/src/
# wget https://codeload.github.com/01org/isa-l/zip/master
# unzip isa-l-master.zip
# ./autogen.sh
# ./configure --prefix=/usr --libdir=/usr/lib64
# make -j  && make install
# systemctl restart openstack-swift-proxy.service
# systemctl status -l openstack-swift-proxy.service
Jan  :: controller liberasurecode[]: liberasurecode_backend_open: dynamic linking error libJerasure.so.: cannot open shared object file: No such file or directory
Jan  :: controller liberasurecode[]: liberasurecode_backend_open: dynamic linking error libshss.so.: cannot open shared object file: No such file or directory

liberasurecode-1.1.0.tar.gz下载地址：   https://bitbucket.org/tsg-/liberasurecode/downloads/

升级liberasurecode版本

# tar xf liberasurecode-1.1..tar.gz
# ./autogen.sh
# ./configure
# make -j  && make install
强制卸载低版本
# rpm -e --nodeps  liberasurecode
# systemctl restart openstack-swift-proxy.service
# systemctl status -l  openstack-swift-proxy.service 

再无报错信息

（2）swift-object 节点启动服务

    swift-object1:      192.168.0.51
    swift-object2:      192.168.0.52

升级liberasurecode版本，安装libisal
python-six 包必须安装，否则服务启动失败

python-six 包必须安装，否则服务启动失败
# yum install gcc gcc-c++ make automake autoconf libtool yasm python-six -y
# cd /usr/local/src/
# wget https://codeload.github.com/01org/isa-l/zip/master
# unzip isa-l-master.zip
# cd isa-l-master
# ./autogen.sh
# ./configure --prefix=/usr --libdir=/usr/lib64
# make -j  && make install
# tar xf liberasurecode-1.1..tar.gz
# cd liberasurecode-1.1.
# ./autogen.sh
# ./configure
# make -j  && make install
强制卸载低版本
# rpm -e --nodeps  liberasurecode

启动服务

# systemctl enable openstack-swift-account.service openstack-swift-account-auditor.service \
  openstack-swift-account-reaper.service openstack-swift-account-replicator.service
# systemctl start openstack-swift-account.service openstack-swift-account-auditor.service \
  openstack-swift-account-reaper.service openstack-swift-account-replicator.service
# systemctl enable openstack-swift-container.service \
  openstack-swift-container-auditor.service openstack-swift-container-replicator.service \
  openstack-swift-container-updater.service
# systemctl start openstack-swift-container.service \
  openstack-swift-container-auditor.service openstack-swift-container-replicator.service \
  openstack-swift-container-updater.service
# systemctl enable openstack-swift-object.service openstack-swift-object-auditor.service \
  openstack-swift-object-replicator.service openstack-swift-object-updater.service
# systemctl start openstack-swift-object.service openstack-swift-object-auditor.service \
  openstack-swift-object-replicator.service openstack-swift-object-updater.service

问题2：启动 openstack-swift-object-replicator.service 报错

Jan  :: object2 object-replicator[]: ERROR creating /srv/node/sdb/objects: #012Traceback (most recent call last):#...
Jan  :: object2 object-replicator[]: ERROR creating /srv/node/sdc/objects: #012Traceback (most recent call last):#...
Jan  :: object2 object-replicator[]: ERROR creating /srv/node/sdd/objects: #012Traceback (most recent call last):#...

检查 /srv/node/ 权限

# ll -d /srv/node/
drwxr-xr-x  root root  Jan  : /srv/node/

# chown -R swift:swift /srv/node
# mkdir -p /var/cache/swift
# chown -R root:swift /var/cache/swift
# chmod -R  /var/cache/swift

重启 openstack-swift-object-replicator.service 恢复正常

2.4.5 验证 swift 服务

在 swift-controller 节点执行：

# . admin-openrc
# swift stat
        Account: AUTH_3b6f963488db4af49e4e0c0d095dd6cf
     Containers:
        Objects:
          Bytes:
X-Put-Timestamp: 1516632330.04818
    X-Timestamp: 1516632330.04818
     X-Trans-Id: tx5fa7f5e817714cbca5cca-005a65f909
   Content-Type: text/plain; charset=utf-

创建一个测试文件，上传该测试文件到 container1 容器中

创建一个测试文件
# touch testfile

上传该测试文件到 container1 容器中
# swift upload container1 testfile
testfile

查看该文件
# swift list container1
testfile

下载该文件
# swift download container1 testfile
testfile [auth .343s, headers .625s, total .626s, 0.000 MB/s]

swift 服务测试成功

2.5 安装 dashboard 服务

在 swift-controller 节点执行：

    swift-controller: 192.168.0.11

# yum install openstack-dashboard -y

编辑配置文件 /etc/openstack-dashboard/local_settings

# vim /etc/openstack-dashboard/local_settings
修改以下内容
...
OPENSTACK_HOST = "controller"
...
ALLOWED_HOSTS = ['*', ]
...
SESSION_ENGINE = 'django.contrib.sessions.backends.cache'
CACHES = {
    'default': {
         'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
         'LOCATION': '127.0.0.1:11211',
    }
}
...
OPENSTACK_KEYSTONE_URL = "http://%s:5000/v3" % OPENSTACK_HOST
...
OPENSTACK_KEYSTONE_MULTIDOMAIN_SUPPORT = True
...
OPENSTACK_API_VERSIONS = {
    "identity": ,
    "image": ,
    "volume": ,
}
...
OPENSTACK_KEYSTONE_DEFAULT_DOMAIN = "default"
...
OPENSTACK_KEYSTONE_DEFAULT_ROLE = "user"
...
TIME_ZONE = "Asia/Shanghai"

配置完成，重启服务

# systemctl restart httpd.service memcached.service

浏览器登录：
    域：    　　   default
    用户名：　　admin
    密码：   　　 admin

dashboard 配置完成。

3. 测试

测试1：swift 能否支持大于5G的文件上传？

生成一个6G的文件，上传单个6G的文件

# dd if=/dev/zero of=bigfile-6G bs=1M count=
# . admin-openrc
# swift upload container1 bigfile-6G
Object PUT failed: http://controller:8080/v1/AUTH_3b6f963488db4af49e4e0c0d095dd6cf/container1/bigfile-6G 413 Request Entity Too Large   Your request is too large.

报错信息：413 请求实体太大，你的请求太大，看来无法普通上传大于5G的文件
swift 客户端版本：python-swiftclient-3.0.0

采用分段上传的方式：
-S 分段大小 （单位：bytes）

# swift upload container1 -S  bigfile-6G
bigfile-6G segment
bigfile-6G segment
bigfile-6G segment
bigfile-6G segment
bigfile-6G segment
bigfile-6G

# swift list container1
bigfile-6G
testfile

上传成功。

总结：
    （1）Swift 对于小的文件，是不分段直接存放的；
    （2）对于大文件（超过5G），需要指定大小分段来存放文件，比如6G文件，设置分段为1G，则会被分为 6 段上传到object集群

测试2：新加object节点

这里由于电脑配置原因不再测试，因为我们在创建时，副本为三份，所以在扩展object节点时候，分区要是 3 的倍数。