OpenStack Mitaka HA部署方案(随笔)

[Toc]

https://github.com/wanstack/AutoMitaka    # 亲情奉献安装openstack HA脚本 使用python + shell，完成了基本的核心功能（纯二层的）。欢迎Fork ，喜欢的请记得start一下。非常感谢。

---

title: Openstack Mitaka 集群安装部署

date: --- :

tags: Openstack

---

==openstack运维开发群: 欢迎牛逼的你==

### Openstack Mitaka HA 实施部署测试文档

#### 一、环境说明

##### 、主机环境

```

controller(VIP) 192.168.10.100

controller01 192.168.10.101, 10.0.0.1

controller02 192.168.10.102, 10.0.0.2

controller03 192.168.10.103, 10.0.0.3

compute01 192.168.10.104, 10.0.0.4

compute02 192.168.10.105, 10.0.0.5

```

本次环境仅限于测试环境，主要测试HA功能。具体生产环境请对网络进行划分。

#### 二、配置基础环境

##### 、配置主机解析

```

在对应节点上配置主机名：

hostnamectl set-hostname controller01

hostname contoller01

hostnamectl set-hostname controller02

hostname contoller02

hostnamectl set-hostname controller03

hostname contoller03

hostnamectl set-hostname compute01

hostname compute01

hostnamectl set-hostname compute02

hostname compute02

```

```

在controller01上配置主机解析：

[root@controller01 ~]# cat /etc/hosts

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4

:: localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.10.100 controller

192.168.10.101 controller01

192.168.10.102 controller02

192.168.10.103 controller03

192.168.10.104 compute01

192.168.10.105 compute02

```

##### 、配置ssh互信

```

在controller01上进行配置：

ssh-keygen -t rsa -f ~/.ssh/id_rsa -P ''

ssh-copy-id -i .ssh/id_rsa.pub root@controller02

ssh-copy-id -i .ssh/id_rsa.pub root@controller03

ssh-copy-id -i .ssh/id_rsa.pub root@compute01

ssh-copy-id -i .ssh/id_rsa.pub root@compute02

```

```

拷贝主机名解析配置文件到其他节点

scp /etc/hosts controller02:/etc/hosts

scp /etc/hosts controller03:/etc/hosts

scp /etc/hosts compute01:/etc/hosts

scp /etc/hosts compute02:/etc/hosts

```

##### 、yum 源配置

本次测试机所有节点都可以正常连接网络，故使用阿里云的openstack和base源

```

# 所有控制和计算节点开启yum缓存

[root@controller01 ~]# cat /etc/yum.conf

[main]

cachedir=/var/cache/yum/$basearch/$releasever

# 开启缓存keepcache=1表示开启缓存，keepcache=0表示不开启缓存，默认为0

keepcache=

debuglevel=

logfile=/var/log/yum.log

exactarch=

obsoletes=

gpgcheck=

plugins=

installonly_limit=

bugtracker_url=http://bugs.centos.org/set_project.php?project_id=23&ref=http://bugs.centos.org/bug_report_page.php?category=yum

distroverpkg=centos-release

# 基础源

yum install wget -y

rm -rf /etc/yum.repos.d/*

wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo

# openstack mitaka源

yum install centos-release-openstack-mitaka -y

# 默认是centos源，建议修改成阿里云的，因为速度快

[root@contoller01 yum.repos.d]# vim CentOS-OpenStack-mitaka.repo

# CentOS-OpenStack-mitaka.repo

#

# Please see http://wiki.centos.org/SpecialInterestGroup/Cloud for more

# information

[centos-openstack-mitaka]

name=CentOS-7 - OpenStack mitaka

baseurl=http://mirrors.aliyun.com//centos/7/cloud/$basearch/openstack-mitaka/

gpgcheck=1

enabled=1

gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Cloud

# galera源

vim mariadb.repo

[mariadb]

name = MariaDB

baseurl = http://yum.mariadb.org/10.1/centos7-amd64/

enable=1

gpgcheck=1

gpgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaDB

```

scp 到其他所有节点

```

scp CentOS-OpenStack-mitaka.repo controller02:/etc/yum.repos.d/CentOS-OpenStack-mitaka.repo

scp CentOS-OpenStack-mitaka.repo controller03:/etc/yum.repos.d/CentOS-OpenStack-mitaka.repo

scp CentOS-OpenStack-mitaka.repo compute01:/etc/yum.repos.d/CentOS-OpenStack-mitaka.repo

scp CentOS-OpenStack-mitaka.repo compute02:/etc/yum.repos.d/CentOS-OpenStack-mitaka.repo

scp mariadb.repo controller02:/etc/yum.repos.d/mariadb.repo

scp mariadb.repo controller03:/etc/yum.repos.d/mariadb.repo

scp mariadb.repo compute01:/etc/yum.repos.d/mariadb.repo

scp mariadb.repo compute02:/etc/yum.repos.d/mariadb.repo

```

##### 4、ntp配置

本机环境已经有ntp服务器，故直接使用。如果没有ntp服务器建议使用controller作为ntp服务器

```

yum install ntpdate -y

echo "*/ * * * * /usr/sbin/ntpdate 192.168.2.161 >/dev/null >&" >> /var/spool/cron/root

/usr/sbin/ntpdate 192.168.2.161

```

##### 、关闭防火墙和selinux

```

systemctl disable firewalld.service

systemctl stop firewalld.service

sed -i -e "s#SELINUX=enforcing#SELINUX=disabled#g" /etc/selinux/config

sed -i -e "s#SELINUXTYPE=targeted#\#SELINUXTYPE=targeted#g" /etc/selinux/config

setenforce

systemctl stop NetworkManager

systemctl disable NetworkManager

```

##### 、安装配置pacemaker

```

# 所有控制节点安装如下软件

yum install -y pcs pacemaker corosync fence-agents-all resource-agents

修改corosync配置文件

[root@contoller01 ~]# cat /etc/corosync/corosync.conf

totem {

version:

secauth: off

cluster_name: openstack-cluster

transport: udpu

}

nodelist {

node {

ring0_addr: controller01

nodeid:

}

node {

ring0_addr: controller02

nodeid:

}

node {

ring0_addr: controller03

nodeid:

}

}

quorum {

provider: corosync_votequorum

two_node:

}

logging {

to_syslog: yes

}

```

```

# 把配置文件拷贝到其他控制节点

scp /etc/corosync/corosync.conf controller02:/etc/corosync/corosync.conf

scp /etc/corosync/corosync.conf controller03:/etc/corosync/corosync.conf

```

```

# 查看成员信息

corosync-cmapctl runtime.totem.pg.mrp.srp.members

```

```

# 所有控制节点启动服务

systemctl enable pcsd

systemctl start pcsd

# 所有控制节点设置hacluster用户的密码

echo hacluster | passwd --stdin hacluster

# [controller01]设置到集群节点的认证

pcs cluster auth controller01 controller02 controller03 -u hacluster -p hacluster --force

# [controller01]创建并启动集群

pcs cluster setup --force --name openstack-cluster controller01 controller02 controller03

pcs cluster start --all

# [controller01]设置集群属性

pcs property set pe-warn-series-max= pe-input-series-max= pe-error-series-max= cluster-recheck-interval=5min

# [controller01] 暂时禁用STONISH，否则资源无法启动

pcs property set stonith-enabled=false

# [controller01] 忽略投票

pcs property set no-quorum-policy=ignore

# [controller01]配置VIP资源，VIP可以在集群节点间浮动

pcs resource create vip ocf:heartbeat:IPaddr2 params ip=192.168.10.100 cidr_netmask="" op monitor interval="30s"

```

##### 、安装haproxy

```

# [所有控制节点] 安装软件

yum install -y haproxy

# [所有控制节点] 修改/etc/rsyslog.d/haproxy.conf文件

echo "\$ModLoad imudp" >> /etc/rsyslog.d/haproxy.conf;

echo "\$UDPServerRun 514" >> /etc/rsyslog.d/haproxy.conf;

echo "local3.* /var/log/haproxy.log" >> /etc/rsyslog.d/haproxy.conf;

echo "&~" >> /etc/rsyslog.d/haproxy.conf;

# [所有控制节点] 修改/etc/sysconfig/rsyslog文件

sed -i -e 's#SYSLOGD_OPTIONS=\"\"#SYSLOGD_OPTIONS=\"-c 2 -r -m 0\"#g' /etc/sysconfig/rsyslog

# [所有控制节点] 重启rsyslog服务

systemctl restart rsyslog

# 创建haproxy基础配置

vim /etc/haproxy/haproxy.cfg

#---------------------------------------------------------------------

# Example configuration for a possible web application. See the

# full configuration options online.

#

# http://haproxy.1wt.eu/download/1.4/doc/configuration.txt

#

#---------------------------------------------------------------------

#---------------------------------------------------------------------

# Global settings

#---------------------------------------------------------------------

global

# to have these messages end up in /var/log/haproxy.log you will

# need to:

#

# ) configure syslog to accept network log events. This is done

# by adding the '-r' option to the SYSLOGD_OPTIONS in

# /etc/sysconfig/syslog

#

# ) configure local2 events to go to the /var/log/haproxy.log

# file. A line like the following can be added to

# /etc/sysconfig/syslog

#

# local2.* /var/log/haproxy.log

#

log 127.0.0.1 local3

chroot /var/lib/haproxy

daemon

group haproxy

maxconn

pidfile /var/run/haproxy.pid

user haproxy

#---------------------------------------------------------------------

# common defaults that all the 'listen' and 'backend' sections will

# use if not designated in their block

#---------------------------------------------------------------------

defaults

log global

maxconn

option redispatch

retries

timeout http-request 10s

timeout queue 1m

timeout connect 10s

timeout client 1m

timeout server 1m

timeout check 10s

include conf.d/*.cfg

```

```

# 拷贝到其他控制节点

scp /etc/haproxy/haproxy.cfg controller02:/etc/haproxy/haproxy.cfg

scp /etc/haproxy/haproxy.cfg controller03:/etc/haproxy/haproxy.cfg

```

```

# [controller01]在pacemaker集群增加haproxy资源

pcs resource create haproxy systemd:haproxy --clone

# Optional表示只在同时停止和/或启动两个资源时才会产生影响。对第一个指定资源进行的任何更改都不会对第二个指定的资源产生影响，定义在前面的资源先确保运行。

pcs constraint order start vip then haproxy-clone kind=Optional

# vip的资源决定了haproxy-clone资源的位置约束

pcs constraint colocation add haproxy-clone with vip

ping -c 3 192.168.10.100

```

##### 8、galera安装配置

```

#所有控制节点上操作基本操作 ：安装、设置配置文件

yum install -y MariaDB-server xinetd

# 在所有控制节点进行配置

vim /usr/lib/systemd/system/mariadb.service

[Service]新添加两行如下参数：

LimitNOFILE=10000

LimitNPROC=10000

systemctl --system daemon-reload

systemctl restart mariadb.service

# 初始化数据库，在controller01上执行即可

systemctl start mariadb

mysql_secure_installation

# 查看并发数

show variables like 'max_connections';

# 关闭服务修改配置文件

systemctl stop mariadb

# 备份原始配置文件

cp /etc/my.cnf.d/server.cnf /etc/my.cnf.d/bak.server.cnf

```

```

# 配置controller01上的配置文件

cat /etc/my.cnf.d/server.cnf

[mysqld]

datadir=/var/lib/mysql

socket=/var/lib/mysql/mysql.sock

user=mysql

binlog_format=ROW

max_connections = 4096

bind-address= 192.168.10.101

default_storage_engine=innodb

innodb_autoinc_lock_mode=2

innodb_flush_log_at_trx_commit=0

innodb_buffer_pool_size=122M

wsrep_on=ON

wsrep_provider=/usr/lib64/galera/libgalera_smm.so

wsrep_provider_options="pc.recovery=TRUE;gcache.size=300M"

wsrep_cluster_name="galera_cluster"

wsrep_cluster_address="gcomm://controller01,controller02,controller03"

wsrep_node_name= controller01

wsrep_node_address= 192.168.10.101

wsrep_sst_method=rsync

```

```

# 配置controller02上的配置文件

cat /etc/my.cnf.d/server.cnf

[mysqld]

datadir=/var/lib/mysql

socket=/var/lib/mysql/mysql.sock

user=mysql

binlog_format=ROW

max_connections = 4096

bind-address= 192.168.10.102

default_storage_engine=innodb

innodb_autoinc_lock_mode=2

innodb_flush_log_at_trx_commit=0

innodb_buffer_pool_size=122M

wsrep_on=ON

wsrep_provider=/usr/lib64/galera/libgalera_smm.so

wsrep_provider_options="pc.recovery=TRUE;gcache.size=300M"

wsrep_cluster_name="galera_cluster"

wsrep_cluster_address="gcomm://controller01,controller02,controller03"

wsrep_node_name= controller02

wsrep_node_address= 192.168.10.102

wsrep_sst_method=rsync

```

```

# 配置controller03上的配置文件

cat /etc/my.cnf.d/server.cnf

[mysqld]

datadir=/var/lib/mysql

socket=/var/lib/mysql/mysql.sock

user=mysql

binlog_format=ROW

max_connections = 4096

bind-address= 192.168.10.103

default_storage_engine=innodb

innodb_autoinc_lock_mode=2

innodb_flush_log_at_trx_commit=0

innodb_buffer_pool_size=122M

wsrep_on=ON

wsrep_provider=/usr/lib64/galera/libgalera_smm.so

wsrep_provider_options="pc.recovery=TRUE;gcache.size=300M"

wsrep_cluster_name="galera_cluster"

wsrep_cluster_address="gcomm://controller01,controller02,controller03"

wsrep_node_name= controller03

wsrep_node_address= 192.168.10.103

wsrep_sst_method=rsync

```

```

# 在controller01上执行

galera_new_cluster

#查看日志

tail -f /var/log/messages

# 启动其他控制节点

systemctl enable mariadb

systemctl start mariadb

```

```

# 添加check

mysql -uroot -popenstack -e "use mysql;INSERT INTO user(Host, User) VALUES('192.168.10.100', 'haproxy_check');FLUSH PRIVILEGES;"

mysql -uroot -popenstack -e "GRANT ALL PRIVILEGES ON *.* TO 'root'@'controller01' IDENTIFIED BY '"openstack"'";

mysql -uroot -popenstack -h 192.168.10.100 -e "SHOW STATUS LIKE 'wsrep_cluster_size';"

```

```

# 配置haproxy for galera

# 所有控制节点创建haproxy配置文件目录

cat /etc/haproxy/haproxy.cfg

listen galera_cluster

bind 192.168.10.100:3306

balance source

#option mysql-check user haproxy_check

server controller01 192.168.10.101:3306 check port 9200 inter 2000 rise 2 fall 5

server controller02 192.168.10.102:3306 check port 9200 inter 2000 rise 2 fall 5

server controller03 192.168.10.103:3306 check port 9200 inter 2000 rise 2 fall 5

# 拷贝配置文件到其他控制节点

scp /etc/haproxy/haproxy.cfg controller02:/etc/haproxy/

scp /etc/haproxy/haproxy.cfg controller03:/etc/haproxy/

```

```

# 重启pacemaker,corosync集群脚本

vim restart-pcs-cluster.sh

#!/bin/sh

pcs cluster stop --all

sleep 10

#ps aux|grep "pcs cluster stop --all"|grep -v grep|awk '{print $2 }'|xargs kill

for i in 01 02 03; do ssh controller$i pcs cluster kill; done

pcs cluster stop --all

pcs cluster start --all

sleep 5

watch -n 0.5 pcs resource

echo "pcs resource"

pcs resource

pcs resource|grep Stop

pcs resource|grep FAILED

# 执行脚本

bash restart-pcs-cluster.sh

```

##### 9、安装配置rabbitmq-server集群

```

# 所有控制节点

yum install -y rabbitmq-server

# 拷贝controller01上的cookie到其他控制节点上

scp /var/lib/rabbitmq/.erlang.cookie root@controller02:/var/lib/rabbitmq/.erlang.cookie

scp /var/lib/rabbitmq/.erlang.cookie root@controller03:/var/lib/rabbitmq/.erlang.cookie

# controller01以外的其他节点设置权限

chown rabbitmq:rabbitmq /var/lib/rabbitmq/.erlang.cookie

chmod 400 /var/lib/rabbitmq/.erlang.cookie

# 启动服务

systemctl enable rabbitmq-server.service

systemctl start rabbitmq-server.service

# 在任意控制节点上查看集群配置

rabbitmqctl cluster_status

# controller01以外的其他节点 加入集群

rabbitmqctl stop_app

rabbitmqctl join_cluster --ram rabbit@controller01

rabbitmqctl start_app

# 在任意节点 设置ha-mode

rabbitmqctl cluster_status;

rabbitmqctl set_policy ha-all '^(?!amq\.).*' '{"ha-mode": "all"}'

# 在任意节点执行创建用户

rabbitmqctl add_user openstack openstack

rabbitmqctl set_permissions openstack ".*" ".*" ".*"

```

##### 10、安装配置memcache

```

yum install -y memcached

# controller01上修改配置

cat /etc/sysconfig/memcached

PORT="11211"

USER="memcached"

MAXCONN="1024"

CACHESIZE="64"

OPTIONS="-l 192.168.10.101,::1"

# controller02上修改配置

cat /etc/sysconfig/memcached

PORT="11211"

USER="memcached"

MAXCONN="1024"

CACHESIZE="64"

OPTIONS="-l 192.168.10.102,::1"

# controller03上修改配置

cat /etc/sysconfig/memcached

PORT="11211"

USER="memcached"

MAXCONN="1024"

CACHESIZE="64"

OPTIONS="-l 192.168.10.103,::1"

# 所有节点启动服务

systemctl enable memcached.service

systemctl start memcached.service

```

#### 三、安装配置openstack软件集群

```

# 所有控制节点和计算节点安装openstack 基础包

yum upgrade -y

yum install -y python-openstackclient openstack-selinux openstack-utils

```

##### 1、安装openstack Identity

```

# 在任意节点创建keystone数据库

mysql -uroot -popenstack -e "CREATE DATABASE keystone;

GRANT ALL PRIVILEGES ON keystone.* TO 'keystone'@'localhost' IDENTIFIED BY '"keystone"';

GRANT ALL PRIVILEGES ON keystone.* TO 'keystone'@'%' IDENTIFIED BY '"keystone"';

FLUSH PRIVILEGES;"

# 所有节点安装keystone软件包

yum install -y openstack-keystone httpd mod_wsgi

# 任意节点生成临时token

openssl rand -hex 10

8464d030a1f7ac3f7207

# 修改keystone配置文件

openstack-config --set /etc/keystone/keystone.conf DEFAULT admin_token 8464d030a1f7ac3f7207

openstack-config --set /etc/keystone/keystone.conf database connection mysql+pymysql://keystone:keystone@controller/keystone

#openstack-config --set /etc/keystone/keystone.conf token provider fernet

openstack-config --set /etc/keystone/keystone.conf oslo_messaging_rabbit rabbit_hosts controller01:5672,controller02:5672,controller03:5672

openstack-config --set /etc/keystone/keystone.conf oslo_messaging_rabbit rabbit_ha_queues true

openstack-config --set /etc/keystone/keystone.conf oslo_messaging_rabbit rabbit_retry_interval 1

openstack-config --set /etc/keystone/keystone.conf oslo_messaging_rabbit rabbit_retry_backoff 2

openstack-config --set /etc/keystone/keystone.conf oslo_messaging_rabbit rabbit_max_retries 0

openstack-config --set /etc/keystone/keystone.conf oslo_messaging_rabbit rabbit_durable_queues true

# 拷贝配置文件到其他控制节点

scp /etc/keystone/keystone.conf controller02:/etc/keystone/keystone.conf

scp /etc/keystone/keystone.conf controller03:/etc/keystone/keystone.conf

sed -i -e 's#\#ServerName www.example.com:80#ServerName '"controller01"'#g' /etc/httpd/conf/httpd.conf

sed -i -e 's#\#ServerName www.example.com:80#ServerName '"controller02"'#g' /etc/httpd/conf/httpd.conf

sed -i -e 's#\#ServerName www.example.com:80#ServerName '"controller03"'#g' /etc/httpd/conf/httpd.conf

# controller01上的配置

vim /etc/httpd/conf.d/wsgi-keystone.conf

Listen 192.168.10.101:5000

Listen 192.168.10.101:35357

<VirtualHost 192.168.10.101:5000>

WSGIDaemonProcess keystone-public processes=5 threads=1 user=keystone group=keystone display-name=%{GROUP}

WSGIProcessGroup keystone-public

WSGIScriptAlias / /usr/bin/keystone-wsgi-public

WSGIApplicationGroup %{GLOBAL}

WSGIPassAuthorization On

ErrorLogFormat "%{cu}t %M"

ErrorLog /var/log/httpd/keystone-error.log

CustomLog /var/log/httpd/keystone-access.log combined

<Directory /usr/bin>

Require all granted

</Directory>

</VirtualHost>

<VirtualHost 192.168.10.101:35357>

WSGIDaemonProcess keystone-admin processes=5 threads=1 user=keystone group=keystone display-name=%{GROUP}

WSGIProcessGroup keystone-admin

WSGIScriptAlias / /usr/bin/keystone-wsgi-admin

WSGIApplicationGroup %{GLOBAL}

WSGIPassAuthorization On

ErrorLogFormat "%{cu}t %M"

ErrorLog /var/log/httpd/keystone-error.log

CustomLog /var/log/httpd/keystone-access.log combined

<Directory /usr/bin>

Require all granted

</Directory>

</VirtualHost>

# controller02上的配置

vim /etc/httpd/conf.d/wsgi-keystone.conf

Listen 192.168.10.102:5000

Listen 192.168.10.102:35357

<VirtualHost 192.168.10.102:5000>

WSGIDaemonProcess keystone-public processes=5 threads=1 user=keystone group=keystone display-name=%{GROUP}

WSGIProcessGroup keystone-public

WSGIScriptAlias / /usr/bin/keystone-wsgi-public

WSGIApplicationGroup %{GLOBAL}

WSGIPassAuthorization On

ErrorLogFormat "%{cu}t %M"

ErrorLog /var/log/httpd/keystone-error.log

CustomLog /var/log/httpd/keystone-access.log combined

<Directory /usr/bin>

Require all granted

</Directory>

</VirtualHost>

<VirtualHost 192.168.10.102:35357>

WSGIDaemonProcess keystone-admin processes=5 threads=1 user=keystone group=keystone display-name=%{GROUP}

WSGIProcessGroup keystone-admin

WSGIScriptAlias / /usr/bin/keystone-wsgi-admin

WSGIApplicationGroup %{GLOBAL}

WSGIPassAuthorization On

ErrorLogFormat "%{cu}t %M"

ErrorLog /var/log/httpd/keystone-error.log

CustomLog /var/log/httpd/keystone-access.log combined

<Directory /usr/bin>

Require all granted

</Directory>

</VirtualHost>

# controller03上的配置

vim /etc/httpd/conf.d/wsgi-keystone.conf

Listen 192.168.10.103:5000

Listen 192.168.10.103:35357

<VirtualHost 192.168.10.103:5000>

WSGIDaemonProcess keystone-public processes=5 threads=1 user=keystone group=keystone display-name=%{GROUP}

WSGIProcessGroup keystone-public

WSGIScriptAlias / /usr/bin/keystone-wsgi-public

WSGIApplicationGroup %{GLOBAL}

WSGIPassAuthorization On

ErrorLogFormat "%{cu}t %M"

ErrorLog /var/log/httpd/keystone-error.log

CustomLog /var/log/httpd/keystone-access.log combined

<Directory /usr/bin>

Require all granted

</Directory>

</VirtualHost>

<VirtualHost 192.168.10.103:35357>

WSGIDaemonProcess keystone-admin processes=5 threads=1 user=keystone group=keystone display-name=%{GROUP}

WSGIProcessGroup keystone-admin

WSGIScriptAlias / /usr/bin/keystone-wsgi-admin

WSGIApplicationGroup %{GLOBAL}

WSGIPassAuthorization On

ErrorLogFormat "%{cu}t %M"

ErrorLog /var/log/httpd/keystone-error.log

CustomLog /var/log/httpd/keystone-access.log combined

<Directory /usr/bin>

Require all granted

</Directory>

</VirtualHost>

# 添加haproxy配置

vim /etc/haproxy/haproxy.cfg

listen keystone_admin_cluster

bind 192.168.10.100:35357

balance source

option tcpka

option httpchk

option tcplog

server controller01 192.168.10.101:35357 check inter 2000 rise 2 fall 5

server controller02 192.168.10.102:35357 check inter 2000 rise 2 fall 5

server controller03 192.168.10.103:35357 check inter 2000 rise 2 fall 5

listen keystone_public_internal_cluster

bind 192.168.10.100:5000

balance source

option tcpka

option httpchk

option tcplog

server controller01 192.168.10.101:5000 check inter 2000 rise 2 fall 5

server controller02 192.168.10.102:5000 check inter 2000 rise 2 fall 5

server controller03 192.168.10.103:5000 check inter 2000 rise 2 fall 5

# 把haproxy配置拷贝到其他控制节点

scp /etc/haproxy/haproxy.cfg controller02:/etc/haproxy/haproxy.cfg

scp /etc/haproxy/haproxy.cfg controller03:/etc/haproxy/haproxy.cfg

# [任一节点]生成数据库

su -s /bin/sh -c "keystone-manage db_sync" keystone

# [任一节点/controller01]初始化Fernet key，并共享给其他节点

#keystone-manage fernet_setup --keystone-user keystone --keystone-group keystone

# 在其他控制节点

#mkdir -p /etc/keystone/fernet-keys/

# 在controller01上

#scp /etc/keystone/fernet-keys/* root@controller02:/etc/keystone/fernet-keys/

#scp /etc/keystone/fernet-keys/* root@controller03:/etc/keystone/fernet-keys/

# 在其他控制节点

chown keystone:keystone /etc/keystone/fernet-keys/*

# [任一节点]添加pacemaker资源，openstack资源和haproxy资源无关，可以开启A/A模式

# interleave=true副本交错启动/停止，改变master/clone间的order约束，每个实例像其他克隆实例一样可快速启动/停止，无需等待其他克隆实例。

# interleave这个设置为false的时候,constraint的order顺序的受到其他节点的影响,为true不受其他节点影响

pcs resource create openstack-keystone systemd:httpd --clone interleave=true

bash restart-pcs-cluster.sh

# 在任意节点创建临时token

export OS_TOKEN=8464d030a1f7ac3f7207

export OS_URL=http://controller:35357/v3

export OS_IDENTITY_API_VERSION=3

# [任一节点]service entity and API endpoints

openstack service create --name keystone --description "OpenStack Identity" identity

openstack endpoint create --region RegionOne identity public http://controller:5000/v3

openstack endpoint create --region RegionOne identity internal http://controller:5000/v3

openstack endpoint create --region RegionOne identity admin http://controller:35357/v3

# [任一节点]创建项目和用户

openstack domain create --description "Default Domain" default

openstack project create --domain default --description "Admin Project" admin

openstack user create --domain default --password admin admin

openstack role create admin

openstack role add --project admin --user admin admin

### [任一节点]创建service项目

openstack project create --domain default --description "Service Project" service

# 在任意节点创建demo项目和用户

openstack project create --domain default --description "Demo Project" demo

openstack user create --domain default --password demo demo

openstack role create user

openstack role add --project demo --user demo user

# 生成keystonerc_admin脚本

echo "export OS_PROJECT_DOMAIN_NAME=default

export OS_USER_DOMAIN_NAME=default

export OS_PROJECT_NAME=admin

export OS_USERNAME=admin

export OS_PASSWORD=admin

export OS_AUTH_URL=http://controller:35357/v3

export OS_IDENTITY_API_VERSION=3

export OS_IMAGE_API_VERSION=2

export PS1='[\u@\h \W(keystone_admin)]\$ '

">/root/keystonerc_admin

chmod +x /root/keystonerc_admin

# 生成keystonerc_demo脚本

echo "export OS_PROJECT_DOMAIN_NAME=default

export OS_USER_DOMAIN_NAME=default

export OS_PROJECT_NAME=demo

export OS_USERNAME=demo

export OS_PASSWORD=demo

export OS_AUTH_URL=http://controller:5000/v3

export OS_IDENTITY_API_VERSION=3

export OS_IMAGE_API_VERSION=2

export PS1='[\u@\h \W(keystone_admin)]\$ '

">/root/keystonerc_demo

chmod +x /root/keystonerc_demo

source keystonerc_admin

### check

openstack token issue

source keystonerc_demo

### check

openstack token issue

```

##### 2、安装openstack Image集群

```

# [任一节点]创建数据库

mysql -uroot -popenstack -e "CREATE DATABASE glance;

GRANT ALL PRIVILEGES ON glance.* TO 'glance'@'localhost' IDENTIFIED BY '"glance"';

GRANT ALL PRIVILEGES ON glance.* TO 'glance'@'%' IDENTIFIED BY '"glance"';

FLUSH PRIVILEGES;"

# [任一节点]创建用户等

source keystonerc_admin

openstack user create --domain default --password glance glance

openstack role add --project service --user glance admin

openstack service create --name glance --description "OpenStack Image" image

openstack endpoint create --region RegionOne image public http://controller:9292

openstack endpoint create --region RegionOne image internal http://controller:9292

openstack endpoint create --region RegionOne image admin http://controller:9292

# 所有控制节点安装glance软件包

yum install -y openstack-glance

# [所有控制节点]配置/etc/glance/glance-api.conf文件

openstack-config --set /etc/glance/glance-api.conf database connection mysql+pymysql://glance:glance@controller/glance

openstack-config --set /etc/glance/glance-api.conf keystone_authtoken auth_uri http://controller:5000

openstack-config --set /etc/glance/glance-api.conf keystone_authtoken auth_url http://controller:35357

openstack-config --set /etc/glance/glance-api.conf keystone_authtoken memcached_servers controller01:11211,controller02:11211,controller03:11211

openstack-config --set /etc/glance/glance-api.conf keystone_authtoken auth_type password

openstack-config --set /etc/glance/glance-api.conf keystone_authtoken project_domain_name default

openstack-config --set /etc/glance/glance-api.conf keystone_authtoken user_domain_name default

openstack-config --set /etc/glance/glance-api.conf keystone_authtoken project_name service

openstack-config --set /etc/glance/glance-api.conf keystone_authtoken username glance

openstack-config --set /etc/glance/glance-api.conf keystone_authtoken password glance

openstack-config --set /etc/glance/glance-api.conf paste_deploy flavor keystone

openstack-config --set /etc/glance/glance-api.conf glance_store stores file,http

openstack-config --set /etc/glance/glance-api.conf glance_store default_store file

openstack-config --set /etc/glance/glance-api.conf glance_store filesystem_store_datadir /var/lib/glance/images/

openstack-config --set /etc/glance/glance-api.conf DEFAULT registry_host controller

openstack-config --set /etc/glance/glance-api.conf DEFAULT bind_host controller01

# [所有控制节点]配置/etc/glance/glance-registry.conf文件

openstack-config --set /etc/glance/glance-registry.conf database connection mysql+pymysql://glance:glance@controller/glance

openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken auth_uri http://controller:5000

openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken auth_url http://controller:35357

openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken memcached_servers controller01:11211,controller02:11211,controller03:11211

openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken auth_type password

openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken project_domain_name default

openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken user_domain_name default

openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken project_name service

openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken username glance

openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken password glance

openstack-config --set /etc/glance/glance-registry.conf paste_deploy flavor keystone

openstack-config --set /etc/glance/glance-registry.conf oslo_messaging_rabbit rabbit_hosts controller01:5672,controller02:5672,controller03:5672

openstack-config --set /etc/glance/glance-registry.conf oslo_messaging_rabbit rabbit_ha_queues true

openstack-config --set /etc/glance/glance-registry.conf oslo_messaging_rabbit rabbit_retry_interval 1

openstack-config --set /etc/glance/glance-registry.conf oslo_messaging_rabbit rabbit_retry_backoff 2

openstack-config --set /etc/glance/glance-registry.conf oslo_messaging_rabbit rabbit_max_retries 0

openstack-config --set /etc/glance/glance-registry.conf oslo_messaging_rabbit rabbit_durable_queues true

openstack-config --set /etc/glance/glance-registry.conf DEFAULT registry_host controller

openstack-config --set /etc/glance/glance-registry.conf DEFAULT bind_host controller01

scp /etc/glance/glance-api.conf controller02:/etc/glance/glance-api.conf

scp /etc/glance/glance-api.conf controller03:/etc/glance/glance-api.conf

# 修改bind_host为对应的controller02,controller03

scp /etc/glance/glance-registry.conf controller02:/etc/glance/glance-registry.conf

scp /etc/glance/glance-registry.conf controller03:/etc/glance/glance-registry.conf

# 修改bind_host为对应的controller02,controller03

vim /etc/haproxy/haproxy.cfg

# 增加如下配置

listen glance_api_cluster

bind 192.168.10.100:9292

balance source

option tcpka

option httpchk

option tcplog

server controller01 192.168.10.101:9292 check inter 2000 rise 2 fall 5

server controller02 192.168.10.102:9292 check inter 2000 rise 2 fall 5

server controller03 192.168.10.103:9292 check inter 2000 rise 2 fall 5

listen glance_registry_cluster

bind 192.168.10.100:9191

balance source

option tcpka

option tcplog

server controller01 192.168.10.101:9191 check inter 2000 rise 2 fall 5

server controller02 192.168.10.102:9191 check inter 2000 rise 2 fall 5

server controller03 192.168.10.103:9191 check inter 2000 rise 2 fall 5

scp /etc/haproxy/haproxy.cfg controller02:/etc/haproxy/haproxy.cfg

scp /etc/haproxy/haproxy.cfg controller03:/etc/haproxy/haproxy.cfg

# [任一节点]生成数据库

su -s /bin/sh -c "glance-manage db_sync" glance

# [任一节点]添加pacemaker资源

pcs resource create openstack-glance-registry systemd:openstack-glance-registry --clone interleave=true

pcs resource create openstack-glance-api systemd:openstack-glance-api --clone interleave=true

# 下面2条表示先启动openstack-keystone-clone然后启动openstack-glance-registry-clone然后启动openstack-glance-api-clone

pcs constraint order start openstack-keystone-clone then openstack-glance-registry-clone

pcs constraint order start openstack-glance-registry-clone then openstack-glance-api-clone

# api随着registry启动而启动，如果registry启动不了，则api也启动不了

pcs constraint colocation add openstack-glance-api-clone with openstack-glance-registry-clone

# 在任意节点重启pacemaker

bash restart-pcs-cluster.sh

# 上传测试镜像

openstack image create "cirros" --file cirros-0.3.4-x86_64-disk.img --disk-format qcow2 --container-format bare --public

openstack image list

```

##### 3、安装openstack Compute集群(控制节点)

```

# 所有控制节点安装软件包

yum install -y openstack-nova-api openstack-nova-conductor openstack-nova-console openstack-nova-novncproxy openstack-nova-scheduler

# [任一节点]创建数据库

mysql -uroot -popenstack -e "CREATE DATABASE nova;

GRANT ALL PRIVILEGES ON nova.* TO 'nova'@'localhost' IDENTIFIED BY '"nova"';

GRANT ALL PRIVILEGES ON nova.* TO 'nova'@'%' IDENTIFIED BY '"nova"';

CREATE DATABASE nova_api;

GRANT ALL PRIVILEGES ON nova_api.* TO 'nova'@'localhost' IDENTIFIED BY '"nova"';

GRANT ALL PRIVILEGES ON nova_api.* TO 'nova'@'%' IDENTIFIED BY '"nova"';

FLUSH PRIVILEGES;"

# [任一节点]创建用户等

source keystonerc_admin

openstack user create --domain default --password nova nova

openstack role add --project service --user nova admin

openstack service create --name nova --description "OpenStack Compute" compute

openstack endpoint create --region RegionOne compute public http://controller:8774/v2.1/%\(tenant_id\)s

openstack endpoint create --region RegionOne compute internal http://controller:8774/v2.1/%\(tenant_id\)s

openstack endpoint create --region RegionOne compute admin http://controller:8774/v2.1/%\(tenant_id\)s

# [所有控制节点]配置配置nova组件，/etc/nova/nova.conf文件

openstack-config --set /etc/nova/nova.conf DEFAULT enabled_apis osapi_compute,metadata

# openstack-config --set /etc/nova/nova.conf DEFAULT memcached_servers controller01:11211,controller02:11211,controller03:11211

openstack-config --set /etc/nova/nova.conf api_database connection mysql+pymysql://nova:nova@controller/nova_api

openstack-config --set /etc/nova/nova.conf database connection mysql+pymysql://nova:nova@controller/nova

openstack-config --set /etc/nova/nova.conf DEFAULT rpc_backend rabbit

openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_hosts controller01:5672,controller02:5672,controller03:5672

openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_ha_queues true

openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_retry_interval 1

openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_retry_backoff 2

openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_max_retries 0

openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_durable_queues true

openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_userid openstack

openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_password openstack

openstack-config --set /etc/nova/nova.conf DEFAULT auth_strategy keystone

openstack-config --set /etc/nova/nova.conf keystone_authtoken auth_uri http://controller:5000

openstack-config --set /etc/nova/nova.conf keystone_authtoken auth_url http://controller:35357

openstack-config --set /etc/nova/nova.conf keystone_authtoken memcached_servers controller01:11211,controller02:11211,controller03:11211

openstack-config --set /etc/nova/nova.conf keystone_authtoken auth_type password

openstack-config --set /etc/nova/nova.conf keystone_authtoken project_domain_name default

openstack-config --set /etc/nova/nova.conf keystone_authtoken user_domain_name default

openstack-config --set /etc/nova/nova.conf keystone_authtoken project_name service

openstack-config --set /etc/nova/nova.conf keystone_authtoken username nova

openstack-config --set /etc/nova/nova.conf keystone_authtoken password nova

openstack-config --set /etc/nova/nova.conf DEFAULT my_ip 192.168.10.101

openstack-config --set /etc/nova/nova.conf DEFAULT use_neutron True

openstack-config --set /etc/nova/nova.conf DEFAULT firewall_driver nova.virt.firewall.NoopFirewallDriver

openstack-config --set /etc/nova/nova.conf vnc vncserver_listen 192.168.10.101

openstack-config --set /etc/nova/nova.conf vnc vncserver_proxyclient_address 192.168.10.101

openstack-config --set /etc/nova/nova.conf vnc novncproxy_host 192.168.10.101

openstack-config --set /etc/nova/nova.conf glance api_servers http://controller:9292

openstack-config --set /etc/nova/nova.conf oslo_concurrency lock_path /var/lib/nova/tmp

openstack-config --set /etc/nova/nova.conf DEFAULT osapi_compute_listen 192.168.10.101

openstack-config --set /etc/nova/nova.conf DEFAULT metadata_listen 192.168.10.101

scp /etc/nova/nova.conf controller02:/etc/nova/nova.conf

scp /etc/nova/nova.conf controller03:/etc/nova/nova.conf

# 其他节点修改my_ip,vncserver_listen,vncserver_proxyclient_address,osapi_compute_listen,metadata_listen,vnc novncproxy_host

openstack-config --set /etc/nova/nova.conf DEFAULT my_ip 192.168.10.102

openstack-config --set /etc/nova/nova.conf vnc vncserver_listen 192.168.10.102

openstack-config --set /etc/nova/nova.conf vnc vncserver_proxyclient_address 192.168.10.102

openstack-config --set /etc/nova/nova.conf vnc novncproxy_host 192.168.10.102

openstack-config --set /etc/nova/nova.conf DEFAULT osapi_compute_listen 192.168.10.102

openstack-config --set /etc/nova/nova.conf DEFAULT metadata_listen 192.168.10.102

################################

openstack-config --set /etc/nova/nova.conf DEFAULT my_ip 192.168.10.103

openstack-config --set /etc/nova/nova.conf vnc vncserver_listen 192.168.10.103

openstack-config --set /etc/nova/nova.conf vnc vncserver_proxyclient_address 192.168.10.103

openstack-config --set /etc/nova/nova.conf vnc novncproxy_host 192.168.10.103

openstack-config --set /etc/nova/nova.conf DEFAULT osapi_compute_listen 192.168.10.103

openstack-config --set /etc/nova/nova.conf DEFAULT metadata_listen 192.168.10.103

##################################

# 配置haproxy

vim /etc/haproxy/haproxy.cfg

listen nova_compute_api_cluster

bind 192.168.10.100:8774

balance source

option tcpka

option httpchk

option tcplog

server controller01 192.168.10.101:8774 check inter 2000 rise 2 fall 5

server controller02 192.168.10.102:8774 check inter 2000 rise 2 fall 5

server controller03 192.168.10.103:8774 check inter 2000 rise 2 fall 5

listen nova_metadata_api_cluster

bind 192.168.10.100:8775

balance source

option tcpka

option tcplog

server controller01 192.168.10.101:8775 check inter 2000 rise 2 fall 5

server controller02 192.168.10.102:8775 check inter 2000 rise 2 fall 5

server controller03 192.168.10.103:8775 check inter 2000 rise 2 fall 5

listen nova_vncproxy_cluster

bind 192.168.10.100:6080

balance source

option tcpka

option tcplog

server controller01 192.168.10.101:6080 check inter 2000 rise 2 fall 5

server controller02 192.168.10.102:6080 check inter 2000 rise 2 fall 5

server controller03 192.168.10.103:6080 check inter 2000 rise 2 fall 5

scp /etc/haproxy/haproxy.cfg controller02:/etc/haproxy/haproxy.cfg

scp /etc/haproxy/haproxy.cfg controller03:/etc/haproxy/haproxy.cfg

# [任一节点]生成数据库

su -s /bin/sh -c "nova-manage api_db sync" nova

su -s /bin/sh -c "nova-manage db sync" nova

# [任一节点]添加pacemaker资源

pcs resource create openstack-nova-consoleauth systemd:openstack-nova-consoleauth --clone interleave=true

pcs resource create openstack-nova-novncproxy systemd:openstack-nova-novncproxy --clone interleave=true

pcs resource create openstack-nova-api systemd:openstack-nova-api --clone interleave=true

pcs resource create openstack-nova-scheduler systemd:openstack-nova-scheduler --clone interleave=true

pcs resource create openstack-nova-conductor systemd:openstack-nova-conductor --clone interleave=true

# 下面几个order属性表示先启动 openstack-keystone-clone 然后启动openstack-nova-consoleauth-clone

# 然后启动openstack-nova-novncproxy-clone，然后启动openstack-nova-api-clone，然后启动openstack-nova-scheduler-clone

# 然后启动openstack-nova-conductor-clone

# 下面几个colocation属性表示consoleauth约束着novncproxy资源的位置，也就是说consoleauth停止，则novncproxy停止。

# 下面的几个colocation属性依次类推

pcs constraint order start openstack-keystone-clone then openstack-nova-consoleauth-clone

pcs constraint order start openstack-nova-consoleauth-clone then openstack-nova-novncproxy-clone

pcs constraint colocation add openstack-nova-novncproxy-clone with openstack-nova-consoleauth-clone

pcs constraint order start openstack-nova-novncproxy-clone then openstack-nova-api-clone

pcs constraint colocation add openstack-nova-api-clone with openstack-nova-novncproxy-clone

pcs constraint order start openstack-nova-api-clone then openstack-nova-scheduler-clone

pcs constraint colocation add openstack-nova-scheduler-clone with openstack-nova-api-clone

pcs constraint order start openstack-nova-scheduler-clone then openstack-nova-conductor-clone

pcs constraint colocation add openstack-nova-conductor-clone with openstack-nova-scheduler-clone

bash restart-pcs-cluster.sh

### [任一节点]测试

source keystonerc_admin

openstack compute service list

```

##### 4、安装配置neutron集群(控制节点)

```

# [任一节点]创建数据库

mysql -uroot -popenstack -e "CREATE DATABASE neutron;

GRANT ALL PRIVILEGES ON neutron.* TO 'neutron'@'localhost' IDENTIFIED BY '"neutron"';

GRANT ALL PRIVILEGES ON neutron.* TO 'neutron'@'%' IDENTIFIED BY '"neutron"';

FLUSH PRIVILEGES;"

# [任一节点]创建用户等

source /root/keystonerc_admin

openstack user create --domain default --password neutron neutron

openstack role add --project service --user neutron admin

openstack service create --name neutron --description "OpenStack Networking" network

openstack endpoint create --region RegionOne network public http://controller:9696

openstack endpoint create --region RegionOne network internal http://controller:9696

openstack endpoint create --region RegionOne network admin http://controller:9696

# 所有控制节点

yum install -y openstack-neutron openstack-neutron-ml2 openstack-neutron-openvswitch ebtables

# [所有控制节点]配置neutron server，/etc/neutron/neutron.conf

openstack-config --set /etc/neutron/neutron.conf DEFAULT bind_host 192.168.10.101

openstack-config --set /etc/neutron/neutron.conf database connection mysql+pymysql://neutron:neutron@controller/neutron

openstack-config --set /etc/neutron/neutron.conf DEFAULT core_plugin ml2

openstack-config --set /etc/neutron/neutron.conf DEFAULT service_plugins router

openstack-config --set /etc/neutron/neutron.conf DEFAULT allow_overlapping_ips True

openstack-config --set /etc/neutron/neutron.conf DEFAULT rpc_backend rabbit

openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_hosts controller01:5672,controller02:5672,controller03:5672

openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_ha_queues true

openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_retry_interval 1

openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_retry_backoff 2

openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_max_retries 0

openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_durable_queues true

openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_userid openstack

openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_password openstack

openstack-config --set /etc/neutron/neutron.conf DEFAULT auth_strategy keystone

openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_uri http://controller:5000

openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_url http://controller:35357

openstack-config --set /etc/neutron/neutron.conf keystone_authtoken memcached_servers controller01:11211,controller02:11211,controller03:11211

openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_type password

openstack-config --set /etc/neutron/neutron.conf keystone_authtoken project_domain_name default

openstack-config --set /etc/neutron/neutron.conf keystone_authtoken user_domain_name default

openstack-config --set /etc/neutron/neutron.conf keystone_authtoken project_name service

openstack-config --set /etc/neutron/neutron.conf keystone_authtoken username neutron

openstack-config --set /etc/neutron/neutron.conf keystone_authtoken password neutron

openstack-config --set /etc/neutron/neutron.conf DEFAULT notify_nova_on_port_status_changes True

openstack-config --set /etc/neutron/neutron.conf DEFAULT notify_nova_on_port_data_changes True

openstack-config --set /etc/neutron/neutron.conf nova auth_url http://controller:35357

openstack-config --set /etc/neutron/neutron.conf nova auth_type password

openstack-config --set /etc/neutron/neutron.conf nova project_domain_name default

openstack-config --set /etc/neutron/neutron.conf nova user_domain_name default

openstack-config --set /etc/neutron/neutron.conf nova region_name RegionOne

openstack-config --set /etc/neutron/neutron.conf nova project_name service

openstack-config --set /etc/neutron/neutron.conf nova username nova

openstack-config --set /etc/neutron/neutron.conf nova password nova

openstack-config --set /etc/neutron/neutron.conf oslo_concurrency lock_path /var/lib/neutron/tmp

# [所有控制节点]配置ML2 plugin，/etc/neutron/plugins/ml2/ml2_conf.ini

openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2 type_drivers flat,vlan,vxlan,gre

openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2 tenant_network_types vxlan

openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2 mechanism_drivers openvswitch,l2population

openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2 extension_drivers port_security

openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2_type_flat flat_networks external

openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2_type_vxlan vni_ranges 1:1000

openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2_type_vlan network_vlan_ranges external:1:4090

openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini securitygroup enable_security_group True

openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini securitygroup enable_ipset True

openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini securitygroup firewall_driver iptables_hybrid

# [所有控制节点]配置Open vSwitch agent，/etc/neutron/plugins/ml2/openvswitch_agent.ini，注意，此处填写第二块网卡

openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini securitygroup enable_security_group True

openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini securitygroup enable_ipset True

openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini securitygroup firewall_driver iptables_hybrid

openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini ovs integration_bridge br-int

openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini ovs tunnel_bridge br-tun

openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini ovs local_ip 10.0.0.1

openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini ovs bridge_mappings external:br-ex

openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini agent tunnel_types vxlan

openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini agent l2_population True

# [所有控制节点]配置L3 agent，/etc/neutron/l3_agent.ini

openstack-config --set /etc/neutron/l3_agent.ini DEFAULT interface_driver neutron.agent.linux.interface.OVSInterfaceDriver

openstack-config --set /etc/neutron/l3_agent.ini DEFAULT external_network_bridge

# [所有控制节点]配置DHCP agent，/etc/neutron/dhcp_agent.ini

openstack-config --set /etc/neutron/dhcp_agent.ini DEFAULT interface_driver neutron.agent.linux.interface.OVSInterfaceDriver

openstack-config --set /etc/neutron/dhcp_agent.ini DEFAULT dhcp_driver neutron.agent.linux.dhcp.Dnsmasq

openstack-config --set /etc/neutron/dhcp_agent.ini DEFAULT enable_isolated_metadata True

# [所有控制节点]配置metadata agent，/etc/neutron/metadata_agent.ini

openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT nova_metadata_ip 192.168.10.100

openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT metadata_proxy_shared_secret openstack

# [所有控制节点]配置nova和neutron集成，/etc/nova/nova.conf

openstack-config --set /etc/nova/nova.conf neutron url http://controller:9696

openstack-config --set /etc/nova/nova.conf neutron auth_url http://controller:35357

openstack-config --set /etc/nova/nova.conf neutron auth_type password

openstack-config --set /etc/nova/nova.conf neutron project_domain_name default

openstack-config --set /etc/nova/nova.conf neutron user_domain_name default

openstack-config --set /etc/nova/nova.conf neutron region_name RegionOne

openstack-config --set /etc/nova/nova.conf neutron project_name service

openstack-config --set /etc/nova/nova.conf neutron username neutron

openstack-config --set /etc/nova/nova.conf neutron password neutron

openstack-config --set /etc/nova/nova.conf neutron service_metadata_proxy True

openstack-config --set /etc/nova/nova.conf neutron metadata_proxy_shared_secret openstack

# [所有控制节点]配置L3 agent HA、/etc/neutron/neutron.conf

openstack-config --set /etc/neutron/neutron.conf DEFAULT l3_ha True

openstack-config --set /etc/neutron/neutron.conf DEFAULT allow_automatic_l3agent_failover True

openstack-config --set /etc/neutron/neutron.conf DEFAULT max_l3_agents_per_router 3

openstack-config --set /etc/neutron/neutron.conf DEFAULT min_l3_agents_per_router 2

# [所有控制节点]配置DHCP agent HA、/etc/neutron/neutron.conf

openstack-config --set /etc/neutron/neutron.conf DEFAULT dhcp_agents_per_network 3

# [所有控制节点] 配置Open vSwitch (OVS) 服务，创建网桥和端口

systemctl enable openvswitch.service

systemctl start openvswitch.service

# [所有控制节点] 创建ML2配置文件软连接

ln -s /etc/neutron/plugins/ml2/ml2_conf.ini /etc/neutron/plugin.ini

vim /etc/haproxy/haproxy.cfg

listen neutron_api_cluster

bind 192.168.10.100:9696

balance source

option tcpka

option httpchk

option tcplog

server controller01 192.168.10.101:9696 check inter 2000 rise 2 fall 5

server controller02 192.168.10.102:9696 check inter 2000 rise 2 fall 5

server controller03 192.168.10.103:9696 check inter 2000 rise 2 fall 5

scp /etc/haproxy/haproxy.cfg controller02:/etc/haproxy/haproxy.cfg

scp /etc/haproxy/haproxy.cfg controller03:/etc/haproxy/haproxy.cfg

# 备份原来配置文件

cp /etc/sysconfig/network-scripts/ifcfg-ens160 /etc/sysconfig/network-scripts/bak-ifcfg-ens160

echo "DEVICE=br-ex

DEVICETYPE=ovs

TYPE=OVSBridge

BOOTPROTO=static

IPADDR=$(cat /etc/sysconfig/network-scripts/ifcfg-ens160 |grep IPADDR|awk -F '=' '{print $2}')

NETMASK=$(cat /etc/sysconfig/network-scripts/ifcfg-ens160 |grep NETMASK|awk -F '=' '{print $2}')

GATEWAY=$(cat /etc/sysconfig/network-scripts/ifcfg-ens160 |grep GATEWAY|awk -F '=' '{print $2}')

DNS1=$(cat /etc/sysconfig/network-scripts/ifcfg-ens160 |grep DNS1|awk -F '=' '{print $2}')

DNS2=218.2.2.2

ONBOOT=yes">/etc/sysconfig/network-scripts/ifcfg-br-ex

echo "TYPE=OVSPort

DEVICETYPE=ovs

OVS_BRIDGE=br-ex

NAME=ens160

DEVICE=ens160

ONBOOT=yes">/etc/sysconfig/network-scripts/ifcfg-ens160

ovs-vsctl add-br br-ex

ovs-vsctl add-port br-ex ens160

systemctl restart network.service

# 拷贝配置文件到其他控制节点并作相应修改

scp /etc/neutron/neutron.conf controller02:/etc/neutron/neutron.conf

scp /etc/neutron/neutron.conf controller03:/etc/neutron/neutron.conf

scp /etc/neutron/plugins/ml2/ml2_conf.ini controller02:/etc/neutron/plugins/ml2/ml2_conf.ini

scp /etc/neutron/plugins/ml2/ml2_conf.ini controller03:/etc/neutron/plugins/ml2/ml2_conf.ini

scp /etc/neutron/plugins/ml2/openvswitch_agent.ini controller02:/etc/neutron/plugins/ml2/openvswitch_agent.ini

scp /etc/neutron/plugins/ml2/openvswitch_agent.ini controller03:/etc/neutron/plugins/ml2/openvswitch_agent.ini

scp /etc/neutron/l3_agent.ini controller02:/etc/neutron/l3_agent.ini

scp /etc/neutron/l3_agent.ini controller03:/etc/neutron/l3_agent.ini

scp /etc/neutron/dhcp_agent.ini controller02:/etc/neutron/dhcp_agent.ini

scp /etc/neutron/dhcp_agent.ini controller03:/etc/neutron/dhcp_agent.ini

scp /etc/neutron/metadata_agent.ini controller02:/etc/neutron/metadata_agent.ini

scp /etc/neutron/metadata_agent.ini controller03:/etc/neutron/metadata_agent.ini

# [任一节点]生成数据库

su -s /bin/sh -c "neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini upgrade head" neutron

# [任一节点]添加pacemaker资源

pcs resource create neutron-server systemd:neutron-server op start timeout=90 --clone interleave=true

pcs constraint order start openstack-keystone-clone then neutron-server-clone

# 全局唯一克隆：参数globally-unique=true。这些资源各不相同。一个节点上运行的克隆实例与另一个节点上运行的实例不同,同一个节点上运行的任何两个实例也不同。

# clone-max: 在集群中最多能运行多少份克隆资源，默认和集群中的节点数相同； clone-node-max：每个节点上最多能运行多少份克隆资源，默认是1；

pcs resource create neutron-scale ocf:neutron:NeutronScale --clone globally-unique=true clone-max=3 interleave=true

pcs constraint order start neutron-server-clone then neutron-scale-clone

pcs resource create neutron-ovs-cleanup ocf:neutron:OVSCleanup --clone interleave=true

pcs resource create neutron-netns-cleanup ocf:neutron:NetnsCleanup --clone interleave=true

pcs resource create neutron-openvswitch-agent systemd:neutron-openvswitch-agent --clone interleave=true

pcs resource create neutron-dhcp-agent systemd:neutron-dhcp-agent --clone interleave=true

pcs resource create neutron-l3-agent systemd:neutron-l3-agent --clone interleave=true

pcs resource create neutron-metadata-agent systemd:neutron-metadata-agent --clone interleave=true

pcs constraint order start neutron-scale-clone then neutron-ovs-cleanup-clone

pcs constraint colocation add neutron-ovs-cleanup-clone with neutron-scale-clone

pcs constraint order start neutron-ovs-cleanup-clone then neutron-netns-cleanup-clone

pcs constraint colocation add neutron-netns-cleanup-clone with neutron-ovs-cleanup-clone

pcs constraint order start neutron-netns-cleanup-clone then neutron-openvswitch-agent-clone

pcs constraint colocation add neutron-openvswitch-agent-clone with neutron-netns-cleanup-clone

pcs constraint order start neutron-openvswitch-agent-clone then neutron-dhcp-agent-clone

pcs constraint colocation add neutron-dhcp-agent-clone with neutron-openvswitch-agent-clone

pcs constraint order start neutron-dhcp-agent-clone then neutron-l3-agent-clone

pcs constraint colocation add neutron-l3-agent-clone with neutron-dhcp-agent-clone

pcs constraint order start neutron-l3-agent-clone then neutron-metadata-agent-clone

pcs constraint colocation add neutron-metadata-agent-clone with neutron-l3-agent-clone

bash restart-pcs-cluster.sh

# [任一节点] 测试

soource keystonerc_admin

neutron ext-list

neutron agent-list

ovs-vsctl show

neutron agent-list

```

##### 5、安装配置dashboard集群

```

# 所有节点安装

yum install -y openstack-dashboard

# [所有控制节点] 修改配置文件/etc/openstack-dashboard/local_settings

sed -i \

-e 's#OPENSTACK_HOST =.*#OPENSTACK_HOST = "'"192.168.10.101"'"#g' \

-e "s#ALLOWED_HOSTS.*#ALLOWED_HOSTS = ['*',]#g" \

-e "s#^CACHES#SESSION_ENGINE = 'django.contrib.sessions.backends.cache'\nCACHES#g#" \

-e "s#locmem.LocMemCache'#memcached.MemcachedCache',\n 'LOCATION' : [ 'controller01:11211', 'controller02:11211', 'controller03:11211', ]#g" \

-e 's#^OPENSTACK_KEYSTONE_URL =.*#OPENSTACK_KEYSTONE_URL = "http://%s:5000/v3" % OPENSTACK_HOST#g' \

-e "s/^#OPENSTACK_KEYSTONE_MULTIDOMAIN_SUPPORT.*/OPENSTACK_KEYSTONE_MULTIDOMAIN_SUPPORT = True/g" \

-e 's/^#OPENSTACK_API_VERSIONS.*/OPENSTACK_API_VERSIONS = {\n "identity": 3,\n "image": 2,\n "volume": 2,\n}\n#OPENSTACK_API_VERSIONS = {/g' \

-e "s/^#OPENSTACK_KEYSTONE_DEFAULT_DOMAIN.*/OPENSTACK_KEYSTONE_DEFAULT_DOMAIN = 'default'/g" \

-e 's#^OPENSTACK_KEYSTONE_DEFAULT_ROLE.*#OPENSTACK_KEYSTONE_DEFAULT_ROLE = "user"#g' \

-e "s#^LOCAL_PATH.*#LOCAL_PATH = '/var/lib/openstack-dashboard'#g" \

-e "s#^SECRET_KEY.*#SECRET_KEY = '4050e76a15dfb7755fe3'#g" \

-e "s#'enable_ha_router'.*#'enable_ha_router': True,#g" \

-e 's#TIME_ZONE = .*#TIME_ZONE = "'"Asia/Shanghai"'"#g' \

/etc/openstack-dashboard/local_settings

scp /etc/openstack-dashboard/local_settings controller02:/etc/openstack-dashboard/local_settings

scp /etc/openstack-dashboard/local_settings controller03:/etc/openstack-dashboard/local_settings

# 在controller02上修改

sed -i -e 's#OPENSTACK_HOST =.*#OPENSTACK_HOST = "'"192.168.10.102"'"#g' /etc/openstack-dashboard/local_settings

# 在controiller03上修改

sed -i -e 's#OPENSTACK_HOST =.*#OPENSTACK_HOST = "'"192.168.10.103"'"#g' /etc/openstack-dashboard/local_settings

# [所有控制节点]

echo "COMPRESS_OFFLINE = True" >> /etc/openstack-dashboard/local_settings

python /usr/share/openstack-dashboard/manage.py compress

# [所有控制节点] 设置HTTPD在特定的IP上监听

sed -i -e 's/^Listen.*/Listen '"$(ip addr show dev br-ex scope global | grep "inet " | sed -e 's#.*inet ##g' -e 's#/.*##g'|head -n 1)"':80/g' /etc/httpd/conf/httpd.conf

vim /etc/haproxy/haproxy.cfg

listen dashboard_cluster

bind 192.168.10.100:

balance source

option tcpka

option httpchk

option tcplog

server controller01 192.168.10.101: check inter  rise  fall

server controller02 192.168.10.102: check inter  rise  fall

server controller03 192.168.10.103: check inter  rise  fall 

scp /etc/haproxy/haproxy.cfg controller02:/etc/haproxy/haproxy.cfg

scp /etc/haproxy/haproxy.cfg controller03:/etc/haproxy/haproxy.cfg

```

##### 、安装配置cinder

```

# 所有控制节点

yum install -y openstack-cinder

# [任一节点]创建数据库

mysql -uroot -popenstack -e "CREATE DATABASE cinder;

GRANT ALL PRIVILEGES ON cinder.* TO 'cinder'@'localhost' IDENTIFIED BY '"cinder"';

GRANT ALL PRIVILEGES ON cinder.* TO 'cinder'@'%' IDENTIFIED BY '"cinder"';

FLUSH PRIVILEGES;"

# [任一节点]创建用户等

. /root/keystonerc_admin

# [任一节点]创建用户等

openstack user create --domain default --password cinder cinder

openstack role add --project service --user cinder admin

openstack service create --name cinder --description "OpenStack Block Storage" volume

openstack service create --name cinderv2 --description "OpenStack Block Storage" volumev2

#创建cinder服务的API endpoints

openstack endpoint create --region RegionOne volume public http://controller:8776/v1/%\(tenant_id\)s

openstack endpoint create --region RegionOne volume internal http://controller:8776/v1/%\(tenant_id\)s

openstack endpoint create --region RegionOne volume admin http://controller:8776/v1/%\(tenant_id\)s

openstack endpoint create --region RegionOne volumev2 public http://controller:8776/v2/%\(tenant_id\)s

openstack endpoint create --region RegionOne volumev2 internal http://controller:8776/v2/%\(tenant_id\)s

openstack endpoint create --region RegionOne volumev2 admin http://controller:8776/v2/%\(tenant_id\)s

#[所有控制节点] 修改/etc/cinder/cinder.conf

openstack-config --set /etc/cinder/cinder.conf database connection mysql+pymysql://cinder:cinder@controller/cinder

openstack-config --set /etc/cinder/cinder.conf database max_retries -

openstack-config --set /etc/cinder/cinder.conf DEFAULT auth_strategy keystone

openstack-config --set /etc/cinder/cinder.conf keystone_authtoken auth_uri http://controller:5000

openstack-config --set /etc/cinder/cinder.conf keystone_authtoken auth_url http://controller:35357

openstack-config --set /etc/cinder/cinder.conf keystone_authtoken memcached_servers controller01:,controller02:,controller03:

openstack-config --set /etc/cinder/cinder.conf keystone_authtoken auth_type password

openstack-config --set /etc/cinder/cinder.conf keystone_authtoken project_domain_name default

openstack-config --set /etc/cinder/cinder.conf keystone_authtoken user_domain_name default

openstack-config --set /etc/cinder/cinder.conf keystone_authtoken project_name service

openstack-config --set /etc/cinder/cinder.conf keystone_authtoken username cinder

openstack-config --set /etc/cinder/cinder.conf keystone_authtoken password cinder

openstack-config --set /etc/cinder/cinder.conf DEFAULT rpc_backend rabbit

openstack-config --set /etc/cinder/cinder.conf oslo_messaging_rabbit rabbit_hosts controller01:,controller02:,controller03:

openstack-config --set /etc/cinder/cinder.conf oslo_messaging_rabbit rabbit_ha_queues true

openstack-config --set /etc/cinder/cinder.conf oslo_messaging_rabbit rabbit_retry_interval

openstack-config --set /etc/cinder/cinder.conf oslo_messaging_rabbit rabbit_retry_backoff

openstack-config --set /etc/cinder/cinder.conf oslo_messaging_rabbit rabbit_max_retries

openstack-config --set /etc/cinder/cinder.conf oslo_messaging_rabbit rabbit_durable_queues true

openstack-config --set /etc/cinder/cinder.conf oslo_messaging_rabbit rabbit_userid openstack

openstack-config --set /etc/cinder/cinder.conf oslo_messaging_rabbit rabbit_password openstack

openstack-config --set /etc/cinder/cinder.conf DEFAULT control_exchange cinder

openstack-config --set /etc/cinder/cinder.conf DEFAULT osapi_volume_listen $(ip addr show dev br-ex scope global | grep "inet " | sed -e 's#.*inet ##g' -e 's#/.*##g'|head -n )

openstack-config --set /etc/cinder/cinder.conf DEFAULT my_ip $(ip addr show dev br-ex scope global | grep "inet " | sed -e 's#.*inet ##g' -e 's#/.*##g'|head -n )

openstack-config --set /etc/cinder/cinder.conf oslo_concurrency lock_path /var/lib/cinder/tmp

openstack-config --set /etc/cinder/cinder.conf DEFAULT glance_api_servers http://controller:9292

# [任一节点]生成数据库

su -s /bin/sh -c "cinder-manage db sync" cinder

# 所有控制节点修改计算节点配置

openstack-config --set /etc/nova/nova.conf cinder os_region_name RegionOne

# 重启计算节点 nova-api

# pcs resource restart openstack-nova-api-clone

# 安装配置存储节点 ，存储节点和控制节点复用

# 所有节点

yum install lvm2 -y

systemctl enable lvm2-lvmetad.service

systemctl start lvm2-lvmetad.service

pvcreate /dev/sdb

vgcreate cinder-volumes /dev/sdb

yum install openstack-cinder targetcli python-keystone -y

# 所有控制节点修改部分配置文件

openstack-config --set /etc/cinder/cinder.conf lvm volume_driver cinder.volume.drivers.lvm.LVMVolumeDriver

openstack-config --set /etc/cinder/cinder.conf lvm volume_group cinder-volumes

openstack-config --set /etc/cinder/cinder.conf lvm iscsi_protocol iscsi

openstack-config --set /etc/cinder/cinder.conf lvm iscsi_helper lioadm

openstack-config --set /etc/cinder/cinder.conf DEFAULT enabled_backends lvm

# 增加haproxy.cfg配置文件

vim /etc/haproxy/haproxy.cfg

listen cinder_api_cluster

bind 192.168.10.100:

balance source

option tcpka

option httpchk

option tcplog

server controller01 192.168.10.101: check inter  rise  fall

server controller02 192.168.10.102: check inter  rise  fall

server controller03 192.168.10.103: check inter  rise  fall 

scp /etc/haproxy/haproxy.cfg controller02:/etc/haproxy/haproxy.cfg

scp /etc/haproxy/haproxy.cfg controller03:/etc/haproxy/haproxy.cfg

# [任一节点]添加pacemaker资源

pcs resource create openstack-cinder-api systemd:openstack-cinder-api --clone interleave=true

pcs resource create openstack-cinder-scheduler systemd:openstack-cinder-scheduler --clone interleave=true

pcs resource create openstack-cinder-volume systemd:openstack-cinder-volume

pcs constraint order start openstack-keystone-clone then openstack-cinder-api-clone

pcs constraint order start openstack-cinder-api-clone then openstack-cinder-scheduler-clone

pcs constraint colocation add openstack-cinder-scheduler-clone with openstack-cinder-api-clone

pcs constraint order start openstack-cinder-scheduler-clone then openstack-cinder-volume

pcs constraint colocation add openstack-cinder-volume with openstack-cinder-scheduler-clone

# 重启集群

bash restart-pcs-cluster.sh

# [任一节点]测试

. /root/keystonerc_admin

cinder service-list

```

#### 、安装配置ceilometer和aodh集群

##### 7.1 安装配置ceilometer集群

实在无力吐槽这个项目，所以不想写了

##### 7.2 安装配置aodh集群

实在无力吐槽这个项目，所以不想写了

#### 四、安装配置计算节点

##### 4.1 OpenStack Compute service

```

# 所有计算节点

yum install -y openstack-nova-compute

# 修改配置文件/etc/nova/nova.conf

openstack-config --set /etc/nova/nova.conf DEFAULT my_ip $(ip addr show dev ens160 scope global | grep "inet " | sed -e 's#.*inet ##g' -e 's#/.*##g')

openstack-config --set /etc/nova/nova.conf DEFAULT use_neutron True

openstack-config --set /etc/nova/nova.conf DEFAULT firewall_driver nova.virt.firewall.NoopFirewallDriver

openstack-config --set /etc/nova/nova.conf DEFAULT memcached_servers controller01:,controller02:,controller03:

openstack-config --set /etc/nova/nova.conf DEFAULT rpc_backend rabbit

openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_hosts controller01:,controller02:,controller03:

openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_ha_queues true

openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_retry_interval

openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_retry_backoff

openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_max_retries

openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_durable_queues true

openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_userid openstack

openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_password openstack

openstack-config --set /etc/nova/nova.conf DEFAULT auth_strategy keystone

openstack-config --set /etc/nova/nova.conf keystone_authtoken auth_uri http://controller:5000

openstack-config --set /etc/nova/nova.conf keystone_authtoken auth_url http://controller:35357

openstack-config --set /etc/nova/nova.conf keystone_authtoken memcached_servers controller01:,controller02:,controller03:

openstack-config --set /etc/nova/nova.conf keystone_authtoken auth_type password

openstack-config --set /etc/nova/nova.conf keystone_authtoken project_domain_name default

openstack-config --set /etc/nova/nova.conf keystone_authtoken user_domain_name default

openstack-config --set /etc/nova/nova.conf keystone_authtoken project_name service

openstack-config --set /etc/nova/nova.conf keystone_authtoken username nova

openstack-config --set /etc/nova/nova.conf keystone_authtoken password nova

openstack-config --set /etc/nova/nova.conf oslo_concurrency lock_path /var/lib/nova/tmp

openstack-config --set /etc/nova/nova.conf vnc enabled True

openstack-config --set /etc/nova/nova.conf vnc vncserver_listen 0.0.0.0

openstack-config --set /etc/nova/nova.conf vnc vncserver_proxyclient_address $(ip addr show dev ens160 scope global | grep "inet " | sed -e 's#.*inet ##g' -e 's#/.*##g')

openstack-config --set /etc/nova/nova.conf vnc novncproxy_base_url http://192.168.10.100:6080/vnc_auto.html

openstack-config --set /etc/nova/nova.conf glance api_servers http://controller:9292

openstack-config --set /etc/nova/nova.conf libvirt virt_type $(count=$(egrep -c '(vmx|svm)' /proc/cpuinfo); if [ $count -eq  ];then echo "qemu"; else echo "kvm"; fi)

# 打开虚拟机迁移的监听端口

sed -i -e "s#\#listen_tls *= *0#listen_tls = 0#g" /etc/libvirt/libvirtd.conf

sed -i -e "s#\#listen_tcp *= *1#listen_tcp = 1#g" /etc/libvirt/libvirtd.conf

sed -i -e "s#\#auth_tcp *= *\"sasl\"#auth_tcp = \"none\"#g" /etc/libvirt/libvirtd.conf

sed -i -e "s#\#LIBVIRTD_ARGS *= *\"--listen\"#LIBVIRTD_ARGS=\"--listen\"#g" /etc/sysconfig/libvirtd

#启动服务

systemctl enable libvirtd.service openstack-nova-compute.service

systemctl start libvirtd.service openstack-nova-compute.service

```

##### 4.2 OpenStack Network service

```

# 安装组件

yum install -y openstack-neutron-openvswitch ebtables ipset

yum install -y openstack-neutron openstack-neutron-ml2 openstack-neutron-openvswitch

# 修改/etc/neutron/neutron.conf

openstack-config --set /etc/neutron/neutron.conf DEFAULT rpc_backend rabbit

openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_hosts controller01:,controller02:,controller03:

openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_ha_queues true

openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_retry_interval

openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_retry_backoff

openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_max_retries

openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_durable_queues true

openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_userid openstack

openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_password openstack

openstack-config --set /etc/neutron/neutron.conf DEFAULT auth_strategy keystone

openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_uri http://controller:5000

openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_url http://controller:35357

openstack-config --set /etc/neutron/neutron.conf keystone_authtoken memcached_servers controller01:,controller02:,controller03:

openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_type password

openstack-config --set /etc/neutron/neutron.conf keystone_authtoken project_domain_name default

openstack-config --set /etc/neutron/neutron.conf keystone_authtoken user_domain_name default

openstack-config --set /etc/neutron/neutron.conf keystone_authtoken project_name service

openstack-config --set /etc/neutron/neutron.conf keystone_authtoken username neutron

openstack-config --set /etc/neutron/neutron.conf keystone_authtoken password neutron

openstack-config --set /etc/neutron/neutron.conf oslo_concurrency lock_path /var/lib/neutron/tmp

### 配置Open vSwitch agent，/etc/neutron/plugins/ml2/openvswitch_agent.ini，注意，此处填写第二块网卡

openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini securitygroup enable_security_group True

openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini securitygroup enable_ipset True

openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini securitygroup firewall_driver iptables_hybrid

openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini ovs local_ip $(ip addr show dev ens192 scope global | grep "inet " | sed -e 's#.*inet ##g' -e 's#/.*##g')

openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini agent tunnel_types vxlan

openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini agent l2_population True

### 配置nova和neutron集成，/etc/nova/nova.conf

openstack-config --set /etc/nova/nova.conf neutron url http://controller:9696

openstack-config --set /etc/nova/nova.conf neutron auth_url http://controller:35357

openstack-config --set /etc/nova/nova.conf neutron auth_type password

openstack-config --set /etc/nova/nova.conf neutron project_domain_name default

openstack-config --set /etc/nova/nova.conf neutron user_domain_name default

openstack-config --set /etc/nova/nova.conf neutron region_name RegionOne

openstack-config --set /etc/nova/nova.conf neutron project_name service

openstack-config --set /etc/nova/nova.conf neutron username neutron

openstack-config --set /etc/nova/nova.conf neutron password neutron

ln -s /etc/neutron/plugins/ml2/ml2_conf.ini /etc/neutron/plugin.ini

systemctl restart openstack-nova-compute.service

systemctl start openvswitch.service

systemctl restart neutron-openvswitch-agent.service

systemctl enable openvswitch.service

systemctl enable neutron-openvswitch-agent.service

```

#### 五 修补

控制节点：

```

GRANT ALL PRIVILEGES ON *.* TO 'root'@'controller01' IDENTIFIED BY "openstack";

GRANT ALL PRIVILEGES ON *.* TO 'root'@'controller02' IDENTIFIED BY "openstack";

GRANT ALL PRIVILEGES ON *.* TO 'root'@'controller03' IDENTIFIED BY "openstack";

GRANT ALL PRIVILEGES ON *.* TO 'root'@'192.168.10.101' IDENTIFIED BY "openstack";

GRANT ALL PRIVILEGES ON *.* TO 'root'@'192.168.10.102' IDENTIFIED BY "openstack";

GRANT ALL PRIVILEGES ON *.* TO 'root'@'192.168.10.103' IDENTIFIED BY "openstack";

```

rabbitmq集群相关：

```

/sbin/service rabbitmq-server stop

/sbin/service rabbitmq-server start

```

```

# 设置资源超时时间

pcs resource op defaults timeout=90s

# 清除错误

pcs resource cleanup openstack-keystone-clone

```

##### mariadb集群排错

```

报错描述如下：节点启动不了，查看 tailf /var/log/messages日志发现如下报错：

[ERROR] WSREP: gcs/src/gcs_group.cpp:group_post_state_exchange():

解决错误: rm -f /var/lib/mysql/grastate.dat

然后重启服务即可

```

#### 六 增加dvr功能

##### 6.1 控制节点配置

```

vim /etc/neutron/neutron.conf

[DEFAULT]

router_distributed = true

vim /etc/neutron/plugins/ml2/ml2_conf.ini

mechanism_drivers = openvswitch,linuxbridge,l2population

vim /etc/neutron/plugins/ml2/openvswitch_agent.ini

[DEFAULT]

enable_distributed_routing = true

[agent]

l2_population = True

vim /etc/neutron/l3_agent.ini

[DEFAULT]

agent_mode = dvr_snat

vim /etc/openstack-dashboard/local_settings

'enable_distributed_router': True,

重启控制节点neutron相关服务，重启httpd服务

```

##### 6.2 计算节点配置

```

vim /etc/neutron/plugins/ml2/ml2_conf.ini

[ml2]

mechanism_drivers = openvswitch,l2population

vim /etc/neutron/plugins/ml2/openvswitch_agent.ini

[DEFAULT]

enable_distributed_routing = true

[agent]

l2_population = True

vim /etc/neutron/l3_agent.ini

[DEFAULT]

interface_driver = openvswitch

external_network_bridge =

agent_mode = dvr

重启neutron相关服务

ovs-vsctl add-br br-ex

ovs-vsctl add-port br-ex ens160

openstack-service restart neutron

```

关于rabbitmq连接数限制问题：

```

[root@controller01 ~]# cat /etc/security/limits.d/-nproc.conf

# Default limit for number of user's processes to prevent

# accidental fork bombs.

# See rhbz # for reasoning.

* soft nproc

root soft nproc unlimited

*    soft    nofile

*    hard    nofile    

[root@controller01 ~]#ulimit -n 

[root@controller01 ~]#cat /usr/lib/systemd/system/rabbitmq-server.service

[Service]

LimitNOFILE= #在启动脚本中添加此参数

[root@controller01 ~]#systemctl daemon-reload

[root@controller01 ~]#systemctl restart rabbitmq-server.service

[root@controller01 ~]#rabbitctl status

{file_descriptors,[{total_limit,},

{total_used,},

{sockets_limit,},

{sockets_used,}]}

```

#### 关于高可用路由器

只能在系统管理员页面上创建高可用或者DVR分布式路由器

# 关于image镜像共享

把控制节点中的/var/lib/glance/images 镜像目录共享出来。

yum -y install nfs-utils rpcbind -y

mkdir /opt/glance/images/ -p

vim /etc/exports

/opt/glance/images/ 10.128.246.0/(rw,no_root_squash,no_all_squash,sync)

exportfs -r

systemctl enable rpcbind.service

systemctl start rpcbind.service

systemctl enable nfs-server.service

systemctl start nfs-server.service

、2个nova节点查看

showmount -e 10.128.247.153

# 三个控制节点挂载

mount -t nfs 10.128.247.153:/opt/glance/images/ /var/lib/glance/iamges/

chown -R glance.glance /opt/glance/images/

##########

普通用户创建HA路由器

```

neutron router-create router_demo --ha True

```
OpenStack Mitaka HA部署方案(随笔)的更多相关文章

Neutron 物理部署方案 - 每天5分钟玩转 OpenStack（68）
前面我们讨论了 Neutron 的架构,本节讨论 Neutron 的物理部署方案:不同节点部署不同的 Neutron 服务组件. 方案1:控制节点 + 计算节点在这个部署方案中,OpenStack ...
openstack项目【day24】：OpenStack mitaka部署
前言: openstack的部署非常简单,简单的前提建立在扎实的理论功底,本人一直觉得,玩技术一定是理论指导实践,网上遍布个种搭建方法都可以实现一个基本的私有云环境,但是诸位可曾发现,很多配置都是重复 ...
OpenStack Mitaka安装
http://egon09.blog.51cto.com/9161406/1839667 前言: openstack的部署非常简单,简单的前提建立在扎实的理论功底,本人一直觉得,玩技术一定是理论指导实 ...
在Openstack H版部署Nova Cell 时，终端输入nova service-list 和 nova host-list 命令将报错
关于Cell的基本介绍,可以参考贤哥的一篇文章: [OpenStack]G版中关于Nova的Cell http://blog.csdn.net/lynn_kong/article/details/8 ...
openstack controller ha测试环境搭建记录（一）——操作系统准备
为了初步了解openstack controller ha的工作原理,搭建测试环境进行学习. 在学习该方面知识时,当前采用的操作系统版本是centos 7.1 x64.首先在ESXi中建立2台用于测试 ...
Hadoop2.0 Namenode HA实现方案
Hadoop2.0 Namenode HA实现方案介绍及汇总基于社区最新release的Hadoop2.2.0版本,调研了hadoop HA方面的内容.hadoop2.0主要的新特性(Hadoop2 ...
eql高可用部署方案
运行环境服务器两台(后面的所有配置案例都是以10.96.0.64和10.96.0.66为例) 操作系统CentOS release 6.2 必须要有共同的局域网网段两台服务器都要安装keepali ...
Win10+VirtualBox+Openstack Mitaka
首先VirtualBox安装的话,没有什么可演示的,去官网(https://www.virtualbox.org/wiki/Downloads)下载,或者可以去(https://www.virtual ...
openstack高可用集群21-生产环境高可用openstack集群部署记录
第一篇集群概述 keepalived + haproxy +Rabbitmq集群+MariaDB Galera高可用集群部署openstack时使用单个控制节点是非常危险的,这样就意味着单个节 ...
随机推荐

SQL Server 加前导0
declare @a int declare @b int set @a = 1 --需要显示的数字 set @b = 3 --显示位数 select right(cast(power(10,@b) ...
教你高速入门Excel-宏与VBA（续）
版权声明:本文为博主原创文章,未经博主同意不得转载. https://blog.csdn.net/small_baby01/article/details/24028407 引言通过 ...
Android Studio 启动时不自动打开项目
主界面中,选择单击菜单栏 File ,单击 Settings 选择 Appearance & Behavior 选项.选择System Settings选项.取消勾选Reopen last p ...
对 tensorflow 中 tf.nn.embedding_lookup 函数的解释
http://stackoverflow.com/questions/34870614/what-does-tf-nn-embedding-lookup-function-do embedding_l ...
RS232串口通信
RS232串口经常使用在PC机与FPGA通信中,用于两者之间的数据传输,因为UART协议简单.易实现,故经常使用. DB9接口只需要使用3根线,RXD(2).TXD(3)和GND(5),如下图所示.而 ...
HDU1081：To The Max（最大子矩阵，线性DP）
题目:http://acm.hdu.edu.cn/showproblem.php?pid=1081 自己真够垃圾的,明明做过一维的这种题,但遇到二维的这种题目,竟然不会了,我也是服了(ps:猪啊). ...
［StringUtil ］ isEmpty VS isBlank
昨天才意识到这两个的存在. Blank(空字符串 blank) StringUtils.isNoneBlank(null) = false StringUtils.isNoneBlank(null, ...
google protobuf使用2
protobuf mutable_* 函数从该函数的实现上来看,该函数返回指向该字段的一个指针.同时将该字段置为被设置状态. 若该对象存在,则直接返回该对象,若不存在则新new 一个.
cdoj1339郭大侠与线上游戏
地址:http://acm.uestc.edu.cn/#/problem/show/1339 题目: 郭大侠与线上游戏 Time Limit: 6000/2000MS (Java/Others) ...
python之路——MySQL数据库
1 MySQL相关概念介绍 MySQL为关系型数据库(Relational Database Management System), 这种所谓的"关系型"可以理解为"表格 ...
OpenStack Mitaka HA部署方案(随笔)

OpenStack Mitaka HA部署方案(随笔)的更多相关文章

随机推荐

热门专题
