OpenStack Mitaka HA部署方案(随笔)
- [Toc]
- https://github.com/wanstack/AutoMitaka # 亲情奉献安装openstack HA脚本 使用python + shell,完成了基本的核心功能(纯二层的)。欢迎Fork ,喜欢的请记得start一下。非常感谢。
- ---
- title: Openstack Mitaka 集群安装部署
- date: --- :
- tags: Openstack
- ---
- ==openstack运维开发群: 欢迎牛逼的你==
- ### Openstack Mitaka HA 实施部署测试文档
- #### 一、环境说明
- ##### 、主机环境
- ```
- controller(VIP) 192.168.10.100
- controller01 192.168.10.101, 10.0.0.1
- controller02 192.168.10.102, 10.0.0.2
- controller03 192.168.10.103, 10.0.0.3
- compute01 192.168.10.104, 10.0.0.4
- compute02 192.168.10.105, 10.0.0.5
- ```
- 本次环境仅限于测试环境,主要测试HA功能。具体生产环境请对网络进行划分。
- #### 二、配置基础环境
- ##### 、配置主机解析
- ```
- 在对应节点上配置主机名:
- hostnamectl set-hostname controller01
- hostname contoller01
- hostnamectl set-hostname controller02
- hostname contoller02
- hostnamectl set-hostname controller03
- hostname contoller03
- hostnamectl set-hostname compute01
- hostname compute01
- hostnamectl set-hostname compute02
- hostname compute02
- ```
- ```
- 在controller01上配置主机解析:
- [root@controller01 ~]# cat /etc/hosts
- 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
- :: localhost localhost.localdomain localhost6 localhost6.localdomain6
- 192.168.10.100 controller
- 192.168.10.101 controller01
- 192.168.10.102 controller02
- 192.168.10.103 controller03
- 192.168.10.104 compute01
- 192.168.10.105 compute02
- ```
- ##### 、配置ssh互信
- ```
- 在controller01上进行配置:
- ssh-keygen -t rsa -f ~/.ssh/id_rsa -P ''
- ssh-copy-id -i .ssh/id_rsa.pub root@controller02
- ssh-copy-id -i .ssh/id_rsa.pub root@controller03
- ssh-copy-id -i .ssh/id_rsa.pub root@compute01
- ssh-copy-id -i .ssh/id_rsa.pub root@compute02
- ```
- ```
- 拷贝主机名解析配置文件到其他节点
- scp /etc/hosts controller02:/etc/hosts
- scp /etc/hosts controller03:/etc/hosts
- scp /etc/hosts compute01:/etc/hosts
- scp /etc/hosts compute02:/etc/hosts
- ```
- ##### 、yum 源配置
- 本次测试机所有节点都可以正常连接网络,故使用阿里云的openstack和base源
- ```
- # 所有控制和计算节点开启yum缓存
- [root@controller01 ~]# cat /etc/yum.conf
- [main]
- cachedir=/var/cache/yum/$basearch/$releasever
- # 开启缓存keepcache=1表示开启缓存,keepcache=0表示不开启缓存,默认为0
- keepcache=
- debuglevel=
- logfile=/var/log/yum.log
- exactarch=
- obsoletes=
- gpgcheck=
- plugins=
- installonly_limit=
- bugtracker_url=http://bugs.centos.org/set_project.php?project_id=23&ref=http://bugs.centos.org/bug_report_page.php?category=yum
- distroverpkg=centos-release
- # 基础源
- yum install wget -y
- rm -rf /etc/yum.repos.d/*
- wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
- # openstack mitaka源
- yum install centos-release-openstack-mitaka -y
- # 默认是centos源,建议修改成阿里云的,因为速度快
- [root@contoller01 yum.repos.d]# vim CentOS-OpenStack-mitaka.repo
- # CentOS-OpenStack-mitaka.repo
- #
- # Please see http://wiki.centos.org/SpecialInterestGroup/Cloud for more
- # information
- [centos-openstack-mitaka]
- name=CentOS-7 - OpenStack mitaka
- baseurl=http://mirrors.aliyun.com//centos/7/cloud/$basearch/openstack-mitaka/
- gpgcheck=1
- enabled=1
- gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Cloud
- # galera源
- vim mariadb.repo
- [mariadb]
- name = MariaDB
- baseurl = http://yum.mariadb.org/10.1/centos7-amd64/
- enable=1
- gpgcheck=1
- gpgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaDB
- ```
- scp 到其他所有节点
- ```
- scp CentOS-OpenStack-mitaka.repo controller02:/etc/yum.repos.d/CentOS-OpenStack-mitaka.repo
- scp CentOS-OpenStack-mitaka.repo controller03:/etc/yum.repos.d/CentOS-OpenStack-mitaka.repo
- scp CentOS-OpenStack-mitaka.repo compute01:/etc/yum.repos.d/CentOS-OpenStack-mitaka.repo
- scp CentOS-OpenStack-mitaka.repo compute02:/etc/yum.repos.d/CentOS-OpenStack-mitaka.repo
- scp mariadb.repo controller02:/etc/yum.repos.d/mariadb.repo
- scp mariadb.repo controller03:/etc/yum.repos.d/mariadb.repo
- scp mariadb.repo compute01:/etc/yum.repos.d/mariadb.repo
- scp mariadb.repo compute02:/etc/yum.repos.d/mariadb.repo
- ```
- ##### 4、ntp配置
- 本机环境已经有ntp服务器,故直接使用。如果没有ntp服务器建议使用controller作为ntp服务器
- ```
- yum install ntpdate -y
- echo "*/ * * * * /usr/sbin/ntpdate 192.168.2.161 >/dev/null >&" >> /var/spool/cron/root
- /usr/sbin/ntpdate 192.168.2.161
- ```
- ##### 、关闭防火墙和selinux
- ```
- systemctl disable firewalld.service
- systemctl stop firewalld.service
- sed -i -e "s#SELINUX=enforcing#SELINUX=disabled#g" /etc/selinux/config
- sed -i -e "s#SELINUXTYPE=targeted#\#SELINUXTYPE=targeted#g" /etc/selinux/config
- setenforce
- systemctl stop NetworkManager
- systemctl disable NetworkManager
- ```
- ##### 、安装配置pacemaker
- ```
- # 所有控制节点安装如下软件
- yum install -y pcs pacemaker corosync fence-agents-all resource-agents
- 修改corosync配置文件
- [root@contoller01 ~]# cat /etc/corosync/corosync.conf
- totem {
- version:
- secauth: off
- cluster_name: openstack-cluster
- transport: udpu
- }
- nodelist {
- node {
- ring0_addr: controller01
- nodeid:
- }
- node {
- ring0_addr: controller02
- nodeid:
- }
- node {
- ring0_addr: controller03
- nodeid:
- }
- }
- quorum {
- provider: corosync_votequorum
- two_node:
- }
- logging {
- to_syslog: yes
- }
- ```
- ```
- # 把配置文件拷贝到其他控制节点
- scp /etc/corosync/corosync.conf controller02:/etc/corosync/corosync.conf
- scp /etc/corosync/corosync.conf controller03:/etc/corosync/corosync.conf
- ```
- ```
- # 查看成员信息
- corosync-cmapctl runtime.totem.pg.mrp.srp.members
- ```
- ```
- # 所有控制节点启动服务
- systemctl enable pcsd
- systemctl start pcsd
- # 所有控制节点设置hacluster用户的密码
- echo hacluster | passwd --stdin hacluster
- # [controller01]设置到集群节点的认证
- pcs cluster auth controller01 controller02 controller03 -u hacluster -p hacluster --force
- # [controller01]创建并启动集群
- pcs cluster setup --force --name openstack-cluster controller01 controller02 controller03
- pcs cluster start --all
- # [controller01]设置集群属性
- pcs property set pe-warn-series-max= pe-input-series-max= pe-error-series-max= cluster-recheck-interval=5min
- # [controller01] 暂时禁用STONISH,否则资源无法启动
- pcs property set stonith-enabled=false
- # [controller01] 忽略投票
- pcs property set no-quorum-policy=ignore
- # [controller01]配置VIP资源,VIP可以在集群节点间浮动
- pcs resource create vip ocf:heartbeat:IPaddr2 params ip=192.168.10.100 cidr_netmask="" op monitor interval="30s"
- ```
- ##### 、安装haproxy
- ```
- # [所有控制节点] 安装软件
- yum install -y haproxy
- # [所有控制节点] 修改/etc/rsyslog.d/haproxy.conf文件
- echo "\$ModLoad imudp" >> /etc/rsyslog.d/haproxy.conf;
- echo "\$UDPServerRun 514" >> /etc/rsyslog.d/haproxy.conf;
- echo "local3.* /var/log/haproxy.log" >> /etc/rsyslog.d/haproxy.conf;
- echo "&~" >> /etc/rsyslog.d/haproxy.conf;
- # [所有控制节点] 修改/etc/sysconfig/rsyslog文件
- sed -i -e 's#SYSLOGD_OPTIONS=\"\"#SYSLOGD_OPTIONS=\"-c 2 -r -m 0\"#g' /etc/sysconfig/rsyslog
- # [所有控制节点] 重启rsyslog服务
- systemctl restart rsyslog
- # 创建haproxy基础配置
- vim /etc/haproxy/haproxy.cfg
- #---------------------------------------------------------------------
- # Example configuration for a possible web application. See the
- # full configuration options online.
- #
- # http://haproxy.1wt.eu/download/1.4/doc/configuration.txt
- #
- #---------------------------------------------------------------------
- #---------------------------------------------------------------------
- # Global settings
- #---------------------------------------------------------------------
- global
- # to have these messages end up in /var/log/haproxy.log you will
- # need to:
- #
- # ) configure syslog to accept network log events. This is done
- # by adding the '-r' option to the SYSLOGD_OPTIONS in
- # /etc/sysconfig/syslog
- #
- # ) configure local2 events to go to the /var/log/haproxy.log
- # file. A line like the following can be added to
- # /etc/sysconfig/syslog
- #
- # local2.* /var/log/haproxy.log
- #
- log 127.0.0.1 local3
- chroot /var/lib/haproxy
- daemon
- group haproxy
- maxconn
- pidfile /var/run/haproxy.pid
- user haproxy
- #---------------------------------------------------------------------
- # common defaults that all the 'listen' and 'backend' sections will
- # use if not designated in their block
- #---------------------------------------------------------------------
- defaults
- log global
- maxconn
- option redispatch
- retries
- timeout http-request 10s
- timeout queue 1m
- timeout connect 10s
- timeout client 1m
- timeout server 1m
- timeout check 10s
- include conf.d/*.cfg
- ```
- ```
- # 拷贝到其他控制节点
- scp /etc/haproxy/haproxy.cfg controller02:/etc/haproxy/haproxy.cfg
- scp /etc/haproxy/haproxy.cfg controller03:/etc/haproxy/haproxy.cfg
- ```
- ```
- # [controller01]在pacemaker集群增加haproxy资源
- pcs resource create haproxy systemd:haproxy --clone
- # Optional表示只在同时停止和/或启动两个资源时才会产生影响。对第一个指定资源进行的任何更改都不会对第二个指定的资源产生影响,定义在前面的资源先确保运行。
- pcs constraint order start vip then haproxy-clone kind=Optional
- # vip的资源决定了haproxy-clone资源的位置约束
- pcs constraint colocation add haproxy-clone with vip
- ping -c 3 192.168.10.100
- ```
- ##### 8、galera安装配置
- ```
- #所有控制节点上操作基本操作 :安装、设置配置文件
- yum install -y MariaDB-server xinetd
- # 在所有控制节点进行配置
- vim /usr/lib/systemd/system/mariadb.service
- [Service]新添加两行如下参数:
- LimitNOFILE=10000
- LimitNPROC=10000
- systemctl --system daemon-reload
- systemctl restart mariadb.service
- # 初始化数据库,在controller01上执行即可
- systemctl start mariadb
- mysql_secure_installation
- # 查看并发数
- show variables like 'max_connections';
- # 关闭服务修改配置文件
- systemctl stop mariadb
- # 备份原始配置文件
- cp /etc/my.cnf.d/server.cnf /etc/my.cnf.d/bak.server.cnf
- ```
- ```
- # 配置controller01上的配置文件
- cat /etc/my.cnf.d/server.cnf
- [mysqld]
- datadir=/var/lib/mysql
- socket=/var/lib/mysql/mysql.sock
- user=mysql
- binlog_format=ROW
- max_connections = 4096
- bind-address= 192.168.10.101
- default_storage_engine=innodb
- innodb_autoinc_lock_mode=2
- innodb_flush_log_at_trx_commit=0
- innodb_buffer_pool_size=122M
- wsrep_on=ON
- wsrep_provider=/usr/lib64/galera/libgalera_smm.so
- wsrep_provider_options="pc.recovery=TRUE;gcache.size=300M"
- wsrep_cluster_name="galera_cluster"
- wsrep_cluster_address="gcomm://controller01,controller02,controller03"
- wsrep_node_name= controller01
- wsrep_node_address= 192.168.10.101
- wsrep_sst_method=rsync
- ```
- ```
- # 配置controller02上的配置文件
- cat /etc/my.cnf.d/server.cnf
- [mysqld]
- datadir=/var/lib/mysql
- socket=/var/lib/mysql/mysql.sock
- user=mysql
- binlog_format=ROW
- max_connections = 4096
- bind-address= 192.168.10.102
- default_storage_engine=innodb
- innodb_autoinc_lock_mode=2
- innodb_flush_log_at_trx_commit=0
- innodb_buffer_pool_size=122M
- wsrep_on=ON
- wsrep_provider=/usr/lib64/galera/libgalera_smm.so
- wsrep_provider_options="pc.recovery=TRUE;gcache.size=300M"
- wsrep_cluster_name="galera_cluster"
- wsrep_cluster_address="gcomm://controller01,controller02,controller03"
- wsrep_node_name= controller02
- wsrep_node_address= 192.168.10.102
- wsrep_sst_method=rsync
- ```
- ```
- # 配置controller03上的配置文件
- cat /etc/my.cnf.d/server.cnf
- [mysqld]
- datadir=/var/lib/mysql
- socket=/var/lib/mysql/mysql.sock
- user=mysql
- binlog_format=ROW
- max_connections = 4096
- bind-address= 192.168.10.103
- default_storage_engine=innodb
- innodb_autoinc_lock_mode=2
- innodb_flush_log_at_trx_commit=0
- innodb_buffer_pool_size=122M
- wsrep_on=ON
- wsrep_provider=/usr/lib64/galera/libgalera_smm.so
- wsrep_provider_options="pc.recovery=TRUE;gcache.size=300M"
- wsrep_cluster_name="galera_cluster"
- wsrep_cluster_address="gcomm://controller01,controller02,controller03"
- wsrep_node_name= controller03
- wsrep_node_address= 192.168.10.103
- wsrep_sst_method=rsync
- ```
- ```
- # 在controller01上执行
- galera_new_cluster
- #查看日志
- tail -f /var/log/messages
- # 启动其他控制节点
- systemctl enable mariadb
- systemctl start mariadb
- ```
- ```
- # 添加check
- mysql -uroot -popenstack -e "use mysql;INSERT INTO user(Host, User) VALUES('192.168.10.100', 'haproxy_check');FLUSH PRIVILEGES;"
- mysql -uroot -popenstack -e "GRANT ALL PRIVILEGES ON *.* TO 'root'@'controller01' IDENTIFIED BY '"openstack"'";
- mysql -uroot -popenstack -h 192.168.10.100 -e "SHOW STATUS LIKE 'wsrep_cluster_size';"
- ```
- ```
- # 配置haproxy for galera
- # 所有控制节点创建haproxy配置文件目录
- cat /etc/haproxy/haproxy.cfg
- listen galera_cluster
- bind 192.168.10.100:3306
- balance source
- #option mysql-check user haproxy_check
- server controller01 192.168.10.101:3306 check port 9200 inter 2000 rise 2 fall 5
- server controller02 192.168.10.102:3306 check port 9200 inter 2000 rise 2 fall 5
- server controller03 192.168.10.103:3306 check port 9200 inter 2000 rise 2 fall 5
- # 拷贝配置文件到其他控制节点
- scp /etc/haproxy/haproxy.cfg controller02:/etc/haproxy/
- scp /etc/haproxy/haproxy.cfg controller03:/etc/haproxy/
- ```
- ```
- # 重启pacemaker,corosync集群脚本
- vim restart-pcs-cluster.sh
- #!/bin/sh
- pcs cluster stop --all
- sleep 10
- #ps aux|grep "pcs cluster stop --all"|grep -v grep|awk '{print $2 }'|xargs kill
- for i in 01 02 03; do ssh controller$i pcs cluster kill; done
- pcs cluster stop --all
- pcs cluster start --all
- sleep 5
- watch -n 0.5 pcs resource
- echo "pcs resource"
- pcs resource
- pcs resource|grep Stop
- pcs resource|grep FAILED
- # 执行脚本
- bash restart-pcs-cluster.sh
- ```
- ##### 9、安装配置rabbitmq-server集群
- ```
- # 所有控制节点
- yum install -y rabbitmq-server
- # 拷贝controller01上的cookie到其他控制节点上
- scp /var/lib/rabbitmq/.erlang.cookie root@controller02:/var/lib/rabbitmq/.erlang.cookie
- scp /var/lib/rabbitmq/.erlang.cookie root@controller03:/var/lib/rabbitmq/.erlang.cookie
- # controller01以外的其他节点设置权限
- chown rabbitmq:rabbitmq /var/lib/rabbitmq/.erlang.cookie
- chmod 400 /var/lib/rabbitmq/.erlang.cookie
- # 启动服务
- systemctl enable rabbitmq-server.service
- systemctl start rabbitmq-server.service
- # 在任意控制节点上查看集群配置
- rabbitmqctl cluster_status
- # controller01以外的其他节点 加入集群
- rabbitmqctl stop_app
- rabbitmqctl join_cluster --ram rabbit@controller01
- rabbitmqctl start_app
- # 在任意节点 设置ha-mode
- rabbitmqctl cluster_status;
- rabbitmqctl set_policy ha-all '^(?!amq\.).*' '{"ha-mode": "all"}'
- # 在任意节点执行创建用户
- rabbitmqctl add_user openstack openstack
- rabbitmqctl set_permissions openstack ".*" ".*" ".*"
- ```
- ##### 10、安装配置memcache
- ```
- yum install -y memcached
- # controller01上修改配置
- cat /etc/sysconfig/memcached
- PORT="11211"
- USER="memcached"
- MAXCONN="1024"
- CACHESIZE="64"
- OPTIONS="-l 192.168.10.101,::1"
- # controller02上修改配置
- cat /etc/sysconfig/memcached
- PORT="11211"
- USER="memcached"
- MAXCONN="1024"
- CACHESIZE="64"
- OPTIONS="-l 192.168.10.102,::1"
- # controller03上修改配置
- cat /etc/sysconfig/memcached
- PORT="11211"
- USER="memcached"
- MAXCONN="1024"
- CACHESIZE="64"
- OPTIONS="-l 192.168.10.103,::1"
- # 所有节点启动服务
- systemctl enable memcached.service
- systemctl start memcached.service
- ```
- #### 三、安装配置openstack软件集群
- ```
- # 所有控制节点和计算节点安装openstack 基础包
- yum upgrade -y
- yum install -y python-openstackclient openstack-selinux openstack-utils
- ```
- ##### 1、安装openstack Identity
- ```
- # 在任意节点创建keystone数据库
- mysql -uroot -popenstack -e "CREATE DATABASE keystone;
- GRANT ALL PRIVILEGES ON keystone.* TO 'keystone'@'localhost' IDENTIFIED BY '"keystone"';
- GRANT ALL PRIVILEGES ON keystone.* TO 'keystone'@'%' IDENTIFIED BY '"keystone"';
- FLUSH PRIVILEGES;"
- # 所有节点安装keystone软件包
- yum install -y openstack-keystone httpd mod_wsgi
- # 任意节点生成临时token
- openssl rand -hex 10
- 8464d030a1f7ac3f7207
- # 修改keystone配置文件
- openstack-config --set /etc/keystone/keystone.conf DEFAULT admin_token 8464d030a1f7ac3f7207
- openstack-config --set /etc/keystone/keystone.conf database connection mysql+pymysql://keystone:keystone@controller/keystone
- #openstack-config --set /etc/keystone/keystone.conf token provider fernet
- openstack-config --set /etc/keystone/keystone.conf oslo_messaging_rabbit rabbit_hosts controller01:5672,controller02:5672,controller03:5672
- openstack-config --set /etc/keystone/keystone.conf oslo_messaging_rabbit rabbit_ha_queues true
- openstack-config --set /etc/keystone/keystone.conf oslo_messaging_rabbit rabbit_retry_interval 1
- openstack-config --set /etc/keystone/keystone.conf oslo_messaging_rabbit rabbit_retry_backoff 2
- openstack-config --set /etc/keystone/keystone.conf oslo_messaging_rabbit rabbit_max_retries 0
- openstack-config --set /etc/keystone/keystone.conf oslo_messaging_rabbit rabbit_durable_queues true
- # 拷贝配置文件到其他控制节点
- scp /etc/keystone/keystone.conf controller02:/etc/keystone/keystone.conf
- scp /etc/keystone/keystone.conf controller03:/etc/keystone/keystone.conf
- sed -i -e 's#\#ServerName www.example.com:80#ServerName '"controller01"'#g' /etc/httpd/conf/httpd.conf
- sed -i -e 's#\#ServerName www.example.com:80#ServerName '"controller02"'#g' /etc/httpd/conf/httpd.conf
- sed -i -e 's#\#ServerName www.example.com:80#ServerName '"controller03"'#g' /etc/httpd/conf/httpd.conf
- # controller01上的配置
- vim /etc/httpd/conf.d/wsgi-keystone.conf
- Listen 192.168.10.101:5000
- Listen 192.168.10.101:35357
- <VirtualHost 192.168.10.101:5000>
- WSGIDaemonProcess keystone-public processes=5 threads=1 user=keystone group=keystone display-name=%{GROUP}
- WSGIProcessGroup keystone-public
- WSGIScriptAlias / /usr/bin/keystone-wsgi-public
- WSGIApplicationGroup %{GLOBAL}
- WSGIPassAuthorization On
- ErrorLogFormat "%{cu}t %M"
- ErrorLog /var/log/httpd/keystone-error.log
- CustomLog /var/log/httpd/keystone-access.log combined
- <Directory /usr/bin>
- Require all granted
- </Directory>
- </VirtualHost>
- <VirtualHost 192.168.10.101:35357>
- WSGIDaemonProcess keystone-admin processes=5 threads=1 user=keystone group=keystone display-name=%{GROUP}
- WSGIProcessGroup keystone-admin
- WSGIScriptAlias / /usr/bin/keystone-wsgi-admin
- WSGIApplicationGroup %{GLOBAL}
- WSGIPassAuthorization On
- ErrorLogFormat "%{cu}t %M"
- ErrorLog /var/log/httpd/keystone-error.log
- CustomLog /var/log/httpd/keystone-access.log combined
- <Directory /usr/bin>
- Require all granted
- </Directory>
- </VirtualHost>
- # controller02上的配置
- vim /etc/httpd/conf.d/wsgi-keystone.conf
- Listen 192.168.10.102:5000
- Listen 192.168.10.102:35357
- <VirtualHost 192.168.10.102:5000>
- WSGIDaemonProcess keystone-public processes=5 threads=1 user=keystone group=keystone display-name=%{GROUP}
- WSGIProcessGroup keystone-public
- WSGIScriptAlias / /usr/bin/keystone-wsgi-public
- WSGIApplicationGroup %{GLOBAL}
- WSGIPassAuthorization On
- ErrorLogFormat "%{cu}t %M"
- ErrorLog /var/log/httpd/keystone-error.log
- CustomLog /var/log/httpd/keystone-access.log combined
- <Directory /usr/bin>
- Require all granted
- </Directory>
- </VirtualHost>
- <VirtualHost 192.168.10.102:35357>
- WSGIDaemonProcess keystone-admin processes=5 threads=1 user=keystone group=keystone display-name=%{GROUP}
- WSGIProcessGroup keystone-admin
- WSGIScriptAlias / /usr/bin/keystone-wsgi-admin
- WSGIApplicationGroup %{GLOBAL}
- WSGIPassAuthorization On
- ErrorLogFormat "%{cu}t %M"
- ErrorLog /var/log/httpd/keystone-error.log
- CustomLog /var/log/httpd/keystone-access.log combined
- <Directory /usr/bin>
- Require all granted
- </Directory>
- </VirtualHost>
- # controller03上的配置
- vim /etc/httpd/conf.d/wsgi-keystone.conf
- Listen 192.168.10.103:5000
- Listen 192.168.10.103:35357
- <VirtualHost 192.168.10.103:5000>
- WSGIDaemonProcess keystone-public processes=5 threads=1 user=keystone group=keystone display-name=%{GROUP}
- WSGIProcessGroup keystone-public
- WSGIScriptAlias / /usr/bin/keystone-wsgi-public
- WSGIApplicationGroup %{GLOBAL}
- WSGIPassAuthorization On
- ErrorLogFormat "%{cu}t %M"
- ErrorLog /var/log/httpd/keystone-error.log
- CustomLog /var/log/httpd/keystone-access.log combined
- <Directory /usr/bin>
- Require all granted
- </Directory>
- </VirtualHost>
- <VirtualHost 192.168.10.103:35357>
- WSGIDaemonProcess keystone-admin processes=5 threads=1 user=keystone group=keystone display-name=%{GROUP}
- WSGIProcessGroup keystone-admin
- WSGIScriptAlias / /usr/bin/keystone-wsgi-admin
- WSGIApplicationGroup %{GLOBAL}
- WSGIPassAuthorization On
- ErrorLogFormat "%{cu}t %M"
- ErrorLog /var/log/httpd/keystone-error.log
- CustomLog /var/log/httpd/keystone-access.log combined
- <Directory /usr/bin>
- Require all granted
- </Directory>
- </VirtualHost>
- # 添加haproxy配置
- vim /etc/haproxy/haproxy.cfg
- listen keystone_admin_cluster
- bind 192.168.10.100:35357
- balance source
- option tcpka
- option httpchk
- option tcplog
- server controller01 192.168.10.101:35357 check inter 2000 rise 2 fall 5
- server controller02 192.168.10.102:35357 check inter 2000 rise 2 fall 5
- server controller03 192.168.10.103:35357 check inter 2000 rise 2 fall 5
- listen keystone_public_internal_cluster
- bind 192.168.10.100:5000
- balance source
- option tcpka
- option httpchk
- option tcplog
- server controller01 192.168.10.101:5000 check inter 2000 rise 2 fall 5
- server controller02 192.168.10.102:5000 check inter 2000 rise 2 fall 5
- server controller03 192.168.10.103:5000 check inter 2000 rise 2 fall 5
- # 把haproxy配置拷贝到其他控制节点
- scp /etc/haproxy/haproxy.cfg controller02:/etc/haproxy/haproxy.cfg
- scp /etc/haproxy/haproxy.cfg controller03:/etc/haproxy/haproxy.cfg
- # [任一节点]生成数据库
- su -s /bin/sh -c "keystone-manage db_sync" keystone
- # [任一节点/controller01]初始化Fernet key,并共享给其他节点
- #keystone-manage fernet_setup --keystone-user keystone --keystone-group keystone
- # 在其他控制节点
- #mkdir -p /etc/keystone/fernet-keys/
- # 在controller01上
- #scp /etc/keystone/fernet-keys/* root@controller02:/etc/keystone/fernet-keys/
- #scp /etc/keystone/fernet-keys/* root@controller03:/etc/keystone/fernet-keys/
- # 在其他控制节点
- chown keystone:keystone /etc/keystone/fernet-keys/*
- # [任一节点]添加pacemaker资源,openstack资源和haproxy资源无关,可以开启A/A模式
- # interleave=true副本交错启动/停止,改变master/clone间的order约束,每个实例像其他克隆实例一样可快速启动/停止,无需等待其他克隆实例。
- # interleave这个设置为false的时候,constraint的order顺序的受到其他节点的影响,为true不受其他节点影响
- pcs resource create openstack-keystone systemd:httpd --clone interleave=true
- bash restart-pcs-cluster.sh
- # 在任意节点创建临时token
- export OS_TOKEN=8464d030a1f7ac3f7207
- export OS_URL=http://controller:35357/v3
- export OS_IDENTITY_API_VERSION=3
- # [任一节点]service entity and API endpoints
- openstack service create --name keystone --description "OpenStack Identity" identity
- openstack endpoint create --region RegionOne identity public http://controller:5000/v3
- openstack endpoint create --region RegionOne identity internal http://controller:5000/v3
- openstack endpoint create --region RegionOne identity admin http://controller:35357/v3
- # [任一节点]创建项目和用户
- openstack domain create --description "Default Domain" default
- openstack project create --domain default --description "Admin Project" admin
- openstack user create --domain default --password admin admin
- openstack role create admin
- openstack role add --project admin --user admin admin
- ### [任一节点]创建service项目
- openstack project create --domain default --description "Service Project" service
- # 在任意节点创建demo项目和用户
- openstack project create --domain default --description "Demo Project" demo
- openstack user create --domain default --password demo demo
- openstack role create user
- openstack role add --project demo --user demo user
- # 生成keystonerc_admin脚本
- echo "export OS_PROJECT_DOMAIN_NAME=default
- export OS_USER_DOMAIN_NAME=default
- export OS_PROJECT_NAME=admin
- export OS_USERNAME=admin
- export OS_PASSWORD=admin
- export OS_AUTH_URL=http://controller:35357/v3
- export OS_IDENTITY_API_VERSION=3
- export OS_IMAGE_API_VERSION=2
- export PS1='[\u@\h \W(keystone_admin)]\$ '
- ">/root/keystonerc_admin
- chmod +x /root/keystonerc_admin
- # 生成keystonerc_demo脚本
- echo "export OS_PROJECT_DOMAIN_NAME=default
- export OS_USER_DOMAIN_NAME=default
- export OS_PROJECT_NAME=demo
- export OS_USERNAME=demo
- export OS_PASSWORD=demo
- export OS_AUTH_URL=http://controller:5000/v3
- export OS_IDENTITY_API_VERSION=3
- export OS_IMAGE_API_VERSION=2
- export PS1='[\u@\h \W(keystone_admin)]\$ '
- ">/root/keystonerc_demo
- chmod +x /root/keystonerc_demo
- source keystonerc_admin
- ### check
- openstack token issue
- source keystonerc_demo
- ### check
- openstack token issue
- ```
- ##### 2、安装openstack Image集群
- ```
- # [任一节点]创建数据库
- mysql -uroot -popenstack -e "CREATE DATABASE glance;
- GRANT ALL PRIVILEGES ON glance.* TO 'glance'@'localhost' IDENTIFIED BY '"glance"';
- GRANT ALL PRIVILEGES ON glance.* TO 'glance'@'%' IDENTIFIED BY '"glance"';
- FLUSH PRIVILEGES;"
- # [任一节点]创建用户等
- source keystonerc_admin
- openstack user create --domain default --password glance glance
- openstack role add --project service --user glance admin
- openstack service create --name glance --description "OpenStack Image" image
- openstack endpoint create --region RegionOne image public http://controller:9292
- openstack endpoint create --region RegionOne image internal http://controller:9292
- openstack endpoint create --region RegionOne image admin http://controller:9292
- # 所有控制节点安装glance软件包
- yum install -y openstack-glance
- # [所有控制节点]配置/etc/glance/glance-api.conf文件
- openstack-config --set /etc/glance/glance-api.conf database connection mysql+pymysql://glance:glance@controller/glance
- openstack-config --set /etc/glance/glance-api.conf keystone_authtoken auth_uri http://controller:5000
- openstack-config --set /etc/glance/glance-api.conf keystone_authtoken auth_url http://controller:35357
- openstack-config --set /etc/glance/glance-api.conf keystone_authtoken memcached_servers controller01:11211,controller02:11211,controller03:11211
- openstack-config --set /etc/glance/glance-api.conf keystone_authtoken auth_type password
- openstack-config --set /etc/glance/glance-api.conf keystone_authtoken project_domain_name default
- openstack-config --set /etc/glance/glance-api.conf keystone_authtoken user_domain_name default
- openstack-config --set /etc/glance/glance-api.conf keystone_authtoken project_name service
- openstack-config --set /etc/glance/glance-api.conf keystone_authtoken username glance
- openstack-config --set /etc/glance/glance-api.conf keystone_authtoken password glance
- openstack-config --set /etc/glance/glance-api.conf paste_deploy flavor keystone
- openstack-config --set /etc/glance/glance-api.conf glance_store stores file,http
- openstack-config --set /etc/glance/glance-api.conf glance_store default_store file
- openstack-config --set /etc/glance/glance-api.conf glance_store filesystem_store_datadir /var/lib/glance/images/
- openstack-config --set /etc/glance/glance-api.conf DEFAULT registry_host controller
- openstack-config --set /etc/glance/glance-api.conf DEFAULT bind_host controller01
- # [所有控制节点]配置/etc/glance/glance-registry.conf文件
- openstack-config --set /etc/glance/glance-registry.conf database connection mysql+pymysql://glance:glance@controller/glance
- openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken auth_uri http://controller:5000
- openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken auth_url http://controller:35357
- openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken memcached_servers controller01:11211,controller02:11211,controller03:11211
- openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken auth_type password
- openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken project_domain_name default
- openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken user_domain_name default
- openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken project_name service
- openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken username glance
- openstack-config --set /etc/glance/glance-registry.conf keystone_authtoken password glance
- openstack-config --set /etc/glance/glance-registry.conf paste_deploy flavor keystone
- openstack-config --set /etc/glance/glance-registry.conf oslo_messaging_rabbit rabbit_hosts controller01:5672,controller02:5672,controller03:5672
- openstack-config --set /etc/glance/glance-registry.conf oslo_messaging_rabbit rabbit_ha_queues true
- openstack-config --set /etc/glance/glance-registry.conf oslo_messaging_rabbit rabbit_retry_interval 1
- openstack-config --set /etc/glance/glance-registry.conf oslo_messaging_rabbit rabbit_retry_backoff 2
- openstack-config --set /etc/glance/glance-registry.conf oslo_messaging_rabbit rabbit_max_retries 0
- openstack-config --set /etc/glance/glance-registry.conf oslo_messaging_rabbit rabbit_durable_queues true
- openstack-config --set /etc/glance/glance-registry.conf DEFAULT registry_host controller
- openstack-config --set /etc/glance/glance-registry.conf DEFAULT bind_host controller01
- scp /etc/glance/glance-api.conf controller02:/etc/glance/glance-api.conf
- scp /etc/glance/glance-api.conf controller03:/etc/glance/glance-api.conf
- # 修改bind_host为对应的controller02,controller03
- scp /etc/glance/glance-registry.conf controller02:/etc/glance/glance-registry.conf
- scp /etc/glance/glance-registry.conf controller03:/etc/glance/glance-registry.conf
- # 修改bind_host为对应的controller02,controller03
- vim /etc/haproxy/haproxy.cfg
- # 增加如下配置
- listen glance_api_cluster
- bind 192.168.10.100:9292
- balance source
- option tcpka
- option httpchk
- option tcplog
- server controller01 192.168.10.101:9292 check inter 2000 rise 2 fall 5
- server controller02 192.168.10.102:9292 check inter 2000 rise 2 fall 5
- server controller03 192.168.10.103:9292 check inter 2000 rise 2 fall 5
- listen glance_registry_cluster
- bind 192.168.10.100:9191
- balance source
- option tcpka
- option tcplog
- server controller01 192.168.10.101:9191 check inter 2000 rise 2 fall 5
- server controller02 192.168.10.102:9191 check inter 2000 rise 2 fall 5
- server controller03 192.168.10.103:9191 check inter 2000 rise 2 fall 5
- scp /etc/haproxy/haproxy.cfg controller02:/etc/haproxy/haproxy.cfg
- scp /etc/haproxy/haproxy.cfg controller03:/etc/haproxy/haproxy.cfg
- # [任一节点]生成数据库
- su -s /bin/sh -c "glance-manage db_sync" glance
- # [任一节点]添加pacemaker资源
- pcs resource create openstack-glance-registry systemd:openstack-glance-registry --clone interleave=true
- pcs resource create openstack-glance-api systemd:openstack-glance-api --clone interleave=true
- # 下面2条表示先启动openstack-keystone-clone然后启动openstack-glance-registry-clone然后启动openstack-glance-api-clone
- pcs constraint order start openstack-keystone-clone then openstack-glance-registry-clone
- pcs constraint order start openstack-glance-registry-clone then openstack-glance-api-clone
- # api随着registry启动而启动,如果registry启动不了,则api也启动不了
- pcs constraint colocation add openstack-glance-api-clone with openstack-glance-registry-clone
- # 在任意节点重启pacemaker
- bash restart-pcs-cluster.sh
- # 上传测试镜像
- openstack image create "cirros" --file cirros-0.3.4-x86_64-disk.img --disk-format qcow2 --container-format bare --public
- openstack image list
- ```
- ##### 3、安装openstack Compute集群(控制节点)
- ```
- # 所有控制节点安装软件包
- yum install -y openstack-nova-api openstack-nova-conductor openstack-nova-console openstack-nova-novncproxy openstack-nova-scheduler
- # [任一节点]创建数据库
- mysql -uroot -popenstack -e "CREATE DATABASE nova;
- GRANT ALL PRIVILEGES ON nova.* TO 'nova'@'localhost' IDENTIFIED BY '"nova"';
- GRANT ALL PRIVILEGES ON nova.* TO 'nova'@'%' IDENTIFIED BY '"nova"';
- CREATE DATABASE nova_api;
- GRANT ALL PRIVILEGES ON nova_api.* TO 'nova'@'localhost' IDENTIFIED BY '"nova"';
- GRANT ALL PRIVILEGES ON nova_api.* TO 'nova'@'%' IDENTIFIED BY '"nova"';
- FLUSH PRIVILEGES;"
- # [任一节点]创建用户等
- source keystonerc_admin
- openstack user create --domain default --password nova nova
- openstack role add --project service --user nova admin
- openstack service create --name nova --description "OpenStack Compute" compute
- openstack endpoint create --region RegionOne compute public http://controller:8774/v2.1/%\(tenant_id\)s
- openstack endpoint create --region RegionOne compute internal http://controller:8774/v2.1/%\(tenant_id\)s
- openstack endpoint create --region RegionOne compute admin http://controller:8774/v2.1/%\(tenant_id\)s
- # [所有控制节点]配置配置nova组件,/etc/nova/nova.conf文件
- openstack-config --set /etc/nova/nova.conf DEFAULT enabled_apis osapi_compute,metadata
- # openstack-config --set /etc/nova/nova.conf DEFAULT memcached_servers controller01:11211,controller02:11211,controller03:11211
- openstack-config --set /etc/nova/nova.conf api_database connection mysql+pymysql://nova:nova@controller/nova_api
- openstack-config --set /etc/nova/nova.conf database connection mysql+pymysql://nova:nova@controller/nova
- openstack-config --set /etc/nova/nova.conf DEFAULT rpc_backend rabbit
- openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_hosts controller01:5672,controller02:5672,controller03:5672
- openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_ha_queues true
- openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_retry_interval 1
- openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_retry_backoff 2
- openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_max_retries 0
- openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_durable_queues true
- openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_userid openstack
- openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_password openstack
- openstack-config --set /etc/nova/nova.conf DEFAULT auth_strategy keystone
- openstack-config --set /etc/nova/nova.conf keystone_authtoken auth_uri http://controller:5000
- openstack-config --set /etc/nova/nova.conf keystone_authtoken auth_url http://controller:35357
- openstack-config --set /etc/nova/nova.conf keystone_authtoken memcached_servers controller01:11211,controller02:11211,controller03:11211
- openstack-config --set /etc/nova/nova.conf keystone_authtoken auth_type password
- openstack-config --set /etc/nova/nova.conf keystone_authtoken project_domain_name default
- openstack-config --set /etc/nova/nova.conf keystone_authtoken user_domain_name default
- openstack-config --set /etc/nova/nova.conf keystone_authtoken project_name service
- openstack-config --set /etc/nova/nova.conf keystone_authtoken username nova
- openstack-config --set /etc/nova/nova.conf keystone_authtoken password nova
- openstack-config --set /etc/nova/nova.conf DEFAULT my_ip 192.168.10.101
- openstack-config --set /etc/nova/nova.conf DEFAULT use_neutron True
- openstack-config --set /etc/nova/nova.conf DEFAULT firewall_driver nova.virt.firewall.NoopFirewallDriver
- openstack-config --set /etc/nova/nova.conf vnc vncserver_listen 192.168.10.101
- openstack-config --set /etc/nova/nova.conf vnc vncserver_proxyclient_address 192.168.10.101
- openstack-config --set /etc/nova/nova.conf vnc novncproxy_host 192.168.10.101
- openstack-config --set /etc/nova/nova.conf glance api_servers http://controller:9292
- openstack-config --set /etc/nova/nova.conf oslo_concurrency lock_path /var/lib/nova/tmp
- openstack-config --set /etc/nova/nova.conf DEFAULT osapi_compute_listen 192.168.10.101
- openstack-config --set /etc/nova/nova.conf DEFAULT metadata_listen 192.168.10.101
- scp /etc/nova/nova.conf controller02:/etc/nova/nova.conf
- scp /etc/nova/nova.conf controller03:/etc/nova/nova.conf
- # 其他节点修改my_ip,vncserver_listen,vncserver_proxyclient_address,osapi_compute_listen,metadata_listen,vnc novncproxy_host
- openstack-config --set /etc/nova/nova.conf DEFAULT my_ip 192.168.10.102
- openstack-config --set /etc/nova/nova.conf vnc vncserver_listen 192.168.10.102
- openstack-config --set /etc/nova/nova.conf vnc vncserver_proxyclient_address 192.168.10.102
- openstack-config --set /etc/nova/nova.conf vnc novncproxy_host 192.168.10.102
- openstack-config --set /etc/nova/nova.conf DEFAULT osapi_compute_listen 192.168.10.102
- openstack-config --set /etc/nova/nova.conf DEFAULT metadata_listen 192.168.10.102
- ################################
- openstack-config --set /etc/nova/nova.conf DEFAULT my_ip 192.168.10.103
- openstack-config --set /etc/nova/nova.conf vnc vncserver_listen 192.168.10.103
- openstack-config --set /etc/nova/nova.conf vnc vncserver_proxyclient_address 192.168.10.103
- openstack-config --set /etc/nova/nova.conf vnc novncproxy_host 192.168.10.103
- openstack-config --set /etc/nova/nova.conf DEFAULT osapi_compute_listen 192.168.10.103
- openstack-config --set /etc/nova/nova.conf DEFAULT metadata_listen 192.168.10.103
- ##################################
- # 配置haproxy
- vim /etc/haproxy/haproxy.cfg
- listen nova_compute_api_cluster
- bind 192.168.10.100:8774
- balance source
- option tcpka
- option httpchk
- option tcplog
- server controller01 192.168.10.101:8774 check inter 2000 rise 2 fall 5
- server controller02 192.168.10.102:8774 check inter 2000 rise 2 fall 5
- server controller03 192.168.10.103:8774 check inter 2000 rise 2 fall 5
- listen nova_metadata_api_cluster
- bind 192.168.10.100:8775
- balance source
- option tcpka
- option tcplog
- server controller01 192.168.10.101:8775 check inter 2000 rise 2 fall 5
- server controller02 192.168.10.102:8775 check inter 2000 rise 2 fall 5
- server controller03 192.168.10.103:8775 check inter 2000 rise 2 fall 5
- listen nova_vncproxy_cluster
- bind 192.168.10.100:6080
- balance source
- option tcpka
- option tcplog
- server controller01 192.168.10.101:6080 check inter 2000 rise 2 fall 5
- server controller02 192.168.10.102:6080 check inter 2000 rise 2 fall 5
- server controller03 192.168.10.103:6080 check inter 2000 rise 2 fall 5
- scp /etc/haproxy/haproxy.cfg controller02:/etc/haproxy/haproxy.cfg
- scp /etc/haproxy/haproxy.cfg controller03:/etc/haproxy/haproxy.cfg
- # [任一节点]生成数据库
- su -s /bin/sh -c "nova-manage api_db sync" nova
- su -s /bin/sh -c "nova-manage db sync" nova
- # [任一节点]添加pacemaker资源
- pcs resource create openstack-nova-consoleauth systemd:openstack-nova-consoleauth --clone interleave=true
- pcs resource create openstack-nova-novncproxy systemd:openstack-nova-novncproxy --clone interleave=true
- pcs resource create openstack-nova-api systemd:openstack-nova-api --clone interleave=true
- pcs resource create openstack-nova-scheduler systemd:openstack-nova-scheduler --clone interleave=true
- pcs resource create openstack-nova-conductor systemd:openstack-nova-conductor --clone interleave=true
- # 下面几个order属性表示先启动 openstack-keystone-clone 然后启动openstack-nova-consoleauth-clone
- # 然后启动openstack-nova-novncproxy-clone,然后启动openstack-nova-api-clone,然后启动openstack-nova-scheduler-clone
- # 然后启动openstack-nova-conductor-clone
- # 下面几个colocation属性表示consoleauth约束着novncproxy资源的位置,也就是说consoleauth停止,则novncproxy停止。
- # 下面的几个colocation属性依次类推
- pcs constraint order start openstack-keystone-clone then openstack-nova-consoleauth-clone
- pcs constraint order start openstack-nova-consoleauth-clone then openstack-nova-novncproxy-clone
- pcs constraint colocation add openstack-nova-novncproxy-clone with openstack-nova-consoleauth-clone
- pcs constraint order start openstack-nova-novncproxy-clone then openstack-nova-api-clone
- pcs constraint colocation add openstack-nova-api-clone with openstack-nova-novncproxy-clone
- pcs constraint order start openstack-nova-api-clone then openstack-nova-scheduler-clone
- pcs constraint colocation add openstack-nova-scheduler-clone with openstack-nova-api-clone
- pcs constraint order start openstack-nova-scheduler-clone then openstack-nova-conductor-clone
- pcs constraint colocation add openstack-nova-conductor-clone with openstack-nova-scheduler-clone
- bash restart-pcs-cluster.sh
- ### [任一节点]测试
- source keystonerc_admin
- openstack compute service list
- ```
- ##### 4、安装配置neutron集群(控制节点)
- ```
- # [任一节点]创建数据库
- mysql -uroot -popenstack -e "CREATE DATABASE neutron;
- GRANT ALL PRIVILEGES ON neutron.* TO 'neutron'@'localhost' IDENTIFIED BY '"neutron"';
- GRANT ALL PRIVILEGES ON neutron.* TO 'neutron'@'%' IDENTIFIED BY '"neutron"';
- FLUSH PRIVILEGES;"
- # [任一节点]创建用户等
- source /root/keystonerc_admin
- openstack user create --domain default --password neutron neutron
- openstack role add --project service --user neutron admin
- openstack service create --name neutron --description "OpenStack Networking" network
- openstack endpoint create --region RegionOne network public http://controller:9696
- openstack endpoint create --region RegionOne network internal http://controller:9696
- openstack endpoint create --region RegionOne network admin http://controller:9696
- # 所有控制节点
- yum install -y openstack-neutron openstack-neutron-ml2 openstack-neutron-openvswitch ebtables
- # [所有控制节点]配置neutron server,/etc/neutron/neutron.conf
- openstack-config --set /etc/neutron/neutron.conf DEFAULT bind_host 192.168.10.101
- openstack-config --set /etc/neutron/neutron.conf database connection mysql+pymysql://neutron:neutron@controller/neutron
- openstack-config --set /etc/neutron/neutron.conf DEFAULT core_plugin ml2
- openstack-config --set /etc/neutron/neutron.conf DEFAULT service_plugins router
- openstack-config --set /etc/neutron/neutron.conf DEFAULT allow_overlapping_ips True
- openstack-config --set /etc/neutron/neutron.conf DEFAULT rpc_backend rabbit
- openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_hosts controller01:5672,controller02:5672,controller03:5672
- openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_ha_queues true
- openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_retry_interval 1
- openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_retry_backoff 2
- openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_max_retries 0
- openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_durable_queues true
- openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_userid openstack
- openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_password openstack
- openstack-config --set /etc/neutron/neutron.conf DEFAULT auth_strategy keystone
- openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_uri http://controller:5000
- openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_url http://controller:35357
- openstack-config --set /etc/neutron/neutron.conf keystone_authtoken memcached_servers controller01:11211,controller02:11211,controller03:11211
- openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_type password
- openstack-config --set /etc/neutron/neutron.conf keystone_authtoken project_domain_name default
- openstack-config --set /etc/neutron/neutron.conf keystone_authtoken user_domain_name default
- openstack-config --set /etc/neutron/neutron.conf keystone_authtoken project_name service
- openstack-config --set /etc/neutron/neutron.conf keystone_authtoken username neutron
- openstack-config --set /etc/neutron/neutron.conf keystone_authtoken password neutron
- openstack-config --set /etc/neutron/neutron.conf DEFAULT notify_nova_on_port_status_changes True
- openstack-config --set /etc/neutron/neutron.conf DEFAULT notify_nova_on_port_data_changes True
- openstack-config --set /etc/neutron/neutron.conf nova auth_url http://controller:35357
- openstack-config --set /etc/neutron/neutron.conf nova auth_type password
- openstack-config --set /etc/neutron/neutron.conf nova project_domain_name default
- openstack-config --set /etc/neutron/neutron.conf nova user_domain_name default
- openstack-config --set /etc/neutron/neutron.conf nova region_name RegionOne
- openstack-config --set /etc/neutron/neutron.conf nova project_name service
- openstack-config --set /etc/neutron/neutron.conf nova username nova
- openstack-config --set /etc/neutron/neutron.conf nova password nova
- openstack-config --set /etc/neutron/neutron.conf oslo_concurrency lock_path /var/lib/neutron/tmp
- # [所有控制节点]配置ML2 plugin,/etc/neutron/plugins/ml2/ml2_conf.ini
- openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2 type_drivers flat,vlan,vxlan,gre
- openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2 tenant_network_types vxlan
- openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2 mechanism_drivers openvswitch,l2population
- openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2 extension_drivers port_security
- openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2_type_flat flat_networks external
- openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2_type_vxlan vni_ranges 1:1000
- openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini ml2_type_vlan network_vlan_ranges external:1:4090
- openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini securitygroup enable_security_group True
- openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini securitygroup enable_ipset True
- openstack-config --set /etc/neutron/plugins/ml2/ml2_conf.ini securitygroup firewall_driver iptables_hybrid
- # [所有控制节点]配置Open vSwitch agent,/etc/neutron/plugins/ml2/openvswitch_agent.ini,注意,此处填写第二块网卡
- openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini securitygroup enable_security_group True
- openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini securitygroup enable_ipset True
- openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini securitygroup firewall_driver iptables_hybrid
- openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini ovs integration_bridge br-int
- openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini ovs tunnel_bridge br-tun
- openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini ovs local_ip 10.0.0.1
- openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini ovs bridge_mappings external:br-ex
- openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini agent tunnel_types vxlan
- openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini agent l2_population True
- # [所有控制节点]配置L3 agent,/etc/neutron/l3_agent.ini
- openstack-config --set /etc/neutron/l3_agent.ini DEFAULT interface_driver neutron.agent.linux.interface.OVSInterfaceDriver
- openstack-config --set /etc/neutron/l3_agent.ini DEFAULT external_network_bridge
- # [所有控制节点]配置DHCP agent,/etc/neutron/dhcp_agent.ini
- openstack-config --set /etc/neutron/dhcp_agent.ini DEFAULT interface_driver neutron.agent.linux.interface.OVSInterfaceDriver
- openstack-config --set /etc/neutron/dhcp_agent.ini DEFAULT dhcp_driver neutron.agent.linux.dhcp.Dnsmasq
- openstack-config --set /etc/neutron/dhcp_agent.ini DEFAULT enable_isolated_metadata True
- # [所有控制节点]配置metadata agent,/etc/neutron/metadata_agent.ini
- openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT nova_metadata_ip 192.168.10.100
- openstack-config --set /etc/neutron/metadata_agent.ini DEFAULT metadata_proxy_shared_secret openstack
- # [所有控制节点]配置nova和neutron集成,/etc/nova/nova.conf
- openstack-config --set /etc/nova/nova.conf neutron url http://controller:9696
- openstack-config --set /etc/nova/nova.conf neutron auth_url http://controller:35357
- openstack-config --set /etc/nova/nova.conf neutron auth_type password
- openstack-config --set /etc/nova/nova.conf neutron project_domain_name default
- openstack-config --set /etc/nova/nova.conf neutron user_domain_name default
- openstack-config --set /etc/nova/nova.conf neutron region_name RegionOne
- openstack-config --set /etc/nova/nova.conf neutron project_name service
- openstack-config --set /etc/nova/nova.conf neutron username neutron
- openstack-config --set /etc/nova/nova.conf neutron password neutron
- openstack-config --set /etc/nova/nova.conf neutron service_metadata_proxy True
- openstack-config --set /etc/nova/nova.conf neutron metadata_proxy_shared_secret openstack
- # [所有控制节点]配置L3 agent HA、/etc/neutron/neutron.conf
- openstack-config --set /etc/neutron/neutron.conf DEFAULT l3_ha True
- openstack-config --set /etc/neutron/neutron.conf DEFAULT allow_automatic_l3agent_failover True
- openstack-config --set /etc/neutron/neutron.conf DEFAULT max_l3_agents_per_router 3
- openstack-config --set /etc/neutron/neutron.conf DEFAULT min_l3_agents_per_router 2
- # [所有控制节点]配置DHCP agent HA、/etc/neutron/neutron.conf
- openstack-config --set /etc/neutron/neutron.conf DEFAULT dhcp_agents_per_network 3
- # [所有控制节点] 配置Open vSwitch (OVS) 服务,创建网桥和端口
- systemctl enable openvswitch.service
- systemctl start openvswitch.service
- # [所有控制节点] 创建ML2配置文件软连接
- ln -s /etc/neutron/plugins/ml2/ml2_conf.ini /etc/neutron/plugin.ini
- vim /etc/haproxy/haproxy.cfg
- listen neutron_api_cluster
- bind 192.168.10.100:9696
- balance source
- option tcpka
- option httpchk
- option tcplog
- server controller01 192.168.10.101:9696 check inter 2000 rise 2 fall 5
- server controller02 192.168.10.102:9696 check inter 2000 rise 2 fall 5
- server controller03 192.168.10.103:9696 check inter 2000 rise 2 fall 5
- scp /etc/haproxy/haproxy.cfg controller02:/etc/haproxy/haproxy.cfg
- scp /etc/haproxy/haproxy.cfg controller03:/etc/haproxy/haproxy.cfg
- # 备份原来配置文件
- cp /etc/sysconfig/network-scripts/ifcfg-ens160 /etc/sysconfig/network-scripts/bak-ifcfg-ens160
- echo "DEVICE=br-ex
- DEVICETYPE=ovs
- TYPE=OVSBridge
- BOOTPROTO=static
- IPADDR=$(cat /etc/sysconfig/network-scripts/ifcfg-ens160 |grep IPADDR|awk -F '=' '{print $2}')
- NETMASK=$(cat /etc/sysconfig/network-scripts/ifcfg-ens160 |grep NETMASK|awk -F '=' '{print $2}')
- GATEWAY=$(cat /etc/sysconfig/network-scripts/ifcfg-ens160 |grep GATEWAY|awk -F '=' '{print $2}')
- DNS1=$(cat /etc/sysconfig/network-scripts/ifcfg-ens160 |grep DNS1|awk -F '=' '{print $2}')
- DNS2=218.2.2.2
- ONBOOT=yes">/etc/sysconfig/network-scripts/ifcfg-br-ex
- echo "TYPE=OVSPort
- DEVICETYPE=ovs
- OVS_BRIDGE=br-ex
- NAME=ens160
- DEVICE=ens160
- ONBOOT=yes">/etc/sysconfig/network-scripts/ifcfg-ens160
- ovs-vsctl add-br br-ex
- ovs-vsctl add-port br-ex ens160
- systemctl restart network.service
- # 拷贝配置文件到其他控制节点并作相应修改
- scp /etc/neutron/neutron.conf controller02:/etc/neutron/neutron.conf
- scp /etc/neutron/neutron.conf controller03:/etc/neutron/neutron.conf
- scp /etc/neutron/plugins/ml2/ml2_conf.ini controller02:/etc/neutron/plugins/ml2/ml2_conf.ini
- scp /etc/neutron/plugins/ml2/ml2_conf.ini controller03:/etc/neutron/plugins/ml2/ml2_conf.ini
- scp /etc/neutron/plugins/ml2/openvswitch_agent.ini controller02:/etc/neutron/plugins/ml2/openvswitch_agent.ini
- scp /etc/neutron/plugins/ml2/openvswitch_agent.ini controller03:/etc/neutron/plugins/ml2/openvswitch_agent.ini
- scp /etc/neutron/l3_agent.ini controller02:/etc/neutron/l3_agent.ini
- scp /etc/neutron/l3_agent.ini controller03:/etc/neutron/l3_agent.ini
- scp /etc/neutron/dhcp_agent.ini controller02:/etc/neutron/dhcp_agent.ini
- scp /etc/neutron/dhcp_agent.ini controller03:/etc/neutron/dhcp_agent.ini
- scp /etc/neutron/metadata_agent.ini controller02:/etc/neutron/metadata_agent.ini
- scp /etc/neutron/metadata_agent.ini controller03:/etc/neutron/metadata_agent.ini
- # [任一节点]生成数据库
- su -s /bin/sh -c "neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini upgrade head" neutron
- # [任一节点]添加pacemaker资源
- pcs resource create neutron-server systemd:neutron-server op start timeout=90 --clone interleave=true
- pcs constraint order start openstack-keystone-clone then neutron-server-clone
- # 全局唯一克隆:参数globally-unique=true。这些资源各不相同。一个节点上运行的克隆实例与另一个节点上运行的实例不同,同一个节点上运行的任何两个实例也不同。
- # clone-max: 在集群中最多能运行多少份克隆资源,默认和集群中的节点数相同; clone-node-max:每个节点上最多能运行多少份克隆资源,默认是1;
- pcs resource create neutron-scale ocf:neutron:NeutronScale --clone globally-unique=true clone-max=3 interleave=true
- pcs constraint order start neutron-server-clone then neutron-scale-clone
- pcs resource create neutron-ovs-cleanup ocf:neutron:OVSCleanup --clone interleave=true
- pcs resource create neutron-netns-cleanup ocf:neutron:NetnsCleanup --clone interleave=true
- pcs resource create neutron-openvswitch-agent systemd:neutron-openvswitch-agent --clone interleave=true
- pcs resource create neutron-dhcp-agent systemd:neutron-dhcp-agent --clone interleave=true
- pcs resource create neutron-l3-agent systemd:neutron-l3-agent --clone interleave=true
- pcs resource create neutron-metadata-agent systemd:neutron-metadata-agent --clone interleave=true
- pcs constraint order start neutron-scale-clone then neutron-ovs-cleanup-clone
- pcs constraint colocation add neutron-ovs-cleanup-clone with neutron-scale-clone
- pcs constraint order start neutron-ovs-cleanup-clone then neutron-netns-cleanup-clone
- pcs constraint colocation add neutron-netns-cleanup-clone with neutron-ovs-cleanup-clone
- pcs constraint order start neutron-netns-cleanup-clone then neutron-openvswitch-agent-clone
- pcs constraint colocation add neutron-openvswitch-agent-clone with neutron-netns-cleanup-clone
- pcs constraint order start neutron-openvswitch-agent-clone then neutron-dhcp-agent-clone
- pcs constraint colocation add neutron-dhcp-agent-clone with neutron-openvswitch-agent-clone
- pcs constraint order start neutron-dhcp-agent-clone then neutron-l3-agent-clone
- pcs constraint colocation add neutron-l3-agent-clone with neutron-dhcp-agent-clone
- pcs constraint order start neutron-l3-agent-clone then neutron-metadata-agent-clone
- pcs constraint colocation add neutron-metadata-agent-clone with neutron-l3-agent-clone
- bash restart-pcs-cluster.sh
- # [任一节点] 测试
- soource keystonerc_admin
- neutron ext-list
- neutron agent-list
- ovs-vsctl show
- neutron agent-list
- ```
- ##### 5、安装配置dashboard集群
- ```
- # 所有节点安装
- yum install -y openstack-dashboard
- # [所有控制节点] 修改配置文件/etc/openstack-dashboard/local_settings
- sed -i \
- -e 's#OPENSTACK_HOST =.*#OPENSTACK_HOST = "'"192.168.10.101"'"#g' \
- -e "s#ALLOWED_HOSTS.*#ALLOWED_HOSTS = ['*',]#g" \
- -e "s#^CACHES#SESSION_ENGINE = 'django.contrib.sessions.backends.cache'\nCACHES#g#" \
- -e "s#locmem.LocMemCache'#memcached.MemcachedCache',\n 'LOCATION' : [ 'controller01:11211', 'controller02:11211', 'controller03:11211', ]#g" \
- -e 's#^OPENSTACK_KEYSTONE_URL =.*#OPENSTACK_KEYSTONE_URL = "http://%s:5000/v3" % OPENSTACK_HOST#g' \
- -e "s/^#OPENSTACK_KEYSTONE_MULTIDOMAIN_SUPPORT.*/OPENSTACK_KEYSTONE_MULTIDOMAIN_SUPPORT = True/g" \
- -e 's/^#OPENSTACK_API_VERSIONS.*/OPENSTACK_API_VERSIONS = {\n "identity": 3,\n "image": 2,\n "volume": 2,\n}\n#OPENSTACK_API_VERSIONS = {/g' \
- -e "s/^#OPENSTACK_KEYSTONE_DEFAULT_DOMAIN.*/OPENSTACK_KEYSTONE_DEFAULT_DOMAIN = 'default'/g" \
- -e 's#^OPENSTACK_KEYSTONE_DEFAULT_ROLE.*#OPENSTACK_KEYSTONE_DEFAULT_ROLE = "user"#g' \
- -e "s#^LOCAL_PATH.*#LOCAL_PATH = '/var/lib/openstack-dashboard'#g" \
- -e "s#^SECRET_KEY.*#SECRET_KEY = '4050e76a15dfb7755fe3'#g" \
- -e "s#'enable_ha_router'.*#'enable_ha_router': True,#g" \
- -e 's#TIME_ZONE = .*#TIME_ZONE = "'"Asia/Shanghai"'"#g' \
- /etc/openstack-dashboard/local_settings
- scp /etc/openstack-dashboard/local_settings controller02:/etc/openstack-dashboard/local_settings
- scp /etc/openstack-dashboard/local_settings controller03:/etc/openstack-dashboard/local_settings
- # 在controller02上修改
- sed -i -e 's#OPENSTACK_HOST =.*#OPENSTACK_HOST = "'"192.168.10.102"'"#g' /etc/openstack-dashboard/local_settings
- # 在controiller03上修改
- sed -i -e 's#OPENSTACK_HOST =.*#OPENSTACK_HOST = "'"192.168.10.103"'"#g' /etc/openstack-dashboard/local_settings
- # [所有控制节点]
- echo "COMPRESS_OFFLINE = True" >> /etc/openstack-dashboard/local_settings
- python /usr/share/openstack-dashboard/manage.py compress
- # [所有控制节点] 设置HTTPD在特定的IP上监听
- sed -i -e 's/^Listen.*/Listen '"$(ip addr show dev br-ex scope global | grep "inet " | sed -e 's#.*inet ##g' -e 's#/.*##g'|head -n 1)"':80/g' /etc/httpd/conf/httpd.conf
- vim /etc/haproxy/haproxy.cfg
- listen dashboard_cluster
- bind 192.168.10.100:
- balance source
- option tcpka
- option httpchk
- option tcplog
- server controller01 192.168.10.101: check inter rise fall
- server controller02 192.168.10.102: check inter rise fall
- server controller03 192.168.10.103: check inter rise fall
- scp /etc/haproxy/haproxy.cfg controller02:/etc/haproxy/haproxy.cfg
- scp /etc/haproxy/haproxy.cfg controller03:/etc/haproxy/haproxy.cfg
- ```
- ##### 、安装配置cinder
- ```
- # 所有控制节点
- yum install -y openstack-cinder
- # [任一节点]创建数据库
- mysql -uroot -popenstack -e "CREATE DATABASE cinder;
- GRANT ALL PRIVILEGES ON cinder.* TO 'cinder'@'localhost' IDENTIFIED BY '"cinder"';
- GRANT ALL PRIVILEGES ON cinder.* TO 'cinder'@'%' IDENTIFIED BY '"cinder"';
- FLUSH PRIVILEGES;"
- # [任一节点]创建用户等
- . /root/keystonerc_admin
- # [任一节点]创建用户等
- openstack user create --domain default --password cinder cinder
- openstack role add --project service --user cinder admin
- openstack service create --name cinder --description "OpenStack Block Storage" volume
- openstack service create --name cinderv2 --description "OpenStack Block Storage" volumev2
- #创建cinder服务的API endpoints
- openstack endpoint create --region RegionOne volume public http://controller:8776/v1/%\(tenant_id\)s
- openstack endpoint create --region RegionOne volume internal http://controller:8776/v1/%\(tenant_id\)s
- openstack endpoint create --region RegionOne volume admin http://controller:8776/v1/%\(tenant_id\)s
- openstack endpoint create --region RegionOne volumev2 public http://controller:8776/v2/%\(tenant_id\)s
- openstack endpoint create --region RegionOne volumev2 internal http://controller:8776/v2/%\(tenant_id\)s
- openstack endpoint create --region RegionOne volumev2 admin http://controller:8776/v2/%\(tenant_id\)s
- #[所有控制节点] 修改/etc/cinder/cinder.conf
- openstack-config --set /etc/cinder/cinder.conf database connection mysql+pymysql://cinder:cinder@controller/cinder
- openstack-config --set /etc/cinder/cinder.conf database max_retries -
- openstack-config --set /etc/cinder/cinder.conf DEFAULT auth_strategy keystone
- openstack-config --set /etc/cinder/cinder.conf keystone_authtoken auth_uri http://controller:5000
- openstack-config --set /etc/cinder/cinder.conf keystone_authtoken auth_url http://controller:35357
- openstack-config --set /etc/cinder/cinder.conf keystone_authtoken memcached_servers controller01:,controller02:,controller03:
- openstack-config --set /etc/cinder/cinder.conf keystone_authtoken auth_type password
- openstack-config --set /etc/cinder/cinder.conf keystone_authtoken project_domain_name default
- openstack-config --set /etc/cinder/cinder.conf keystone_authtoken user_domain_name default
- openstack-config --set /etc/cinder/cinder.conf keystone_authtoken project_name service
- openstack-config --set /etc/cinder/cinder.conf keystone_authtoken username cinder
- openstack-config --set /etc/cinder/cinder.conf keystone_authtoken password cinder
- openstack-config --set /etc/cinder/cinder.conf DEFAULT rpc_backend rabbit
- openstack-config --set /etc/cinder/cinder.conf oslo_messaging_rabbit rabbit_hosts controller01:,controller02:,controller03:
- openstack-config --set /etc/cinder/cinder.conf oslo_messaging_rabbit rabbit_ha_queues true
- openstack-config --set /etc/cinder/cinder.conf oslo_messaging_rabbit rabbit_retry_interval
- openstack-config --set /etc/cinder/cinder.conf oslo_messaging_rabbit rabbit_retry_backoff
- openstack-config --set /etc/cinder/cinder.conf oslo_messaging_rabbit rabbit_max_retries
- openstack-config --set /etc/cinder/cinder.conf oslo_messaging_rabbit rabbit_durable_queues true
- openstack-config --set /etc/cinder/cinder.conf oslo_messaging_rabbit rabbit_userid openstack
- openstack-config --set /etc/cinder/cinder.conf oslo_messaging_rabbit rabbit_password openstack
- openstack-config --set /etc/cinder/cinder.conf DEFAULT control_exchange cinder
- openstack-config --set /etc/cinder/cinder.conf DEFAULT osapi_volume_listen $(ip addr show dev br-ex scope global | grep "inet " | sed -e 's#.*inet ##g' -e 's#/.*##g'|head -n )
- openstack-config --set /etc/cinder/cinder.conf DEFAULT my_ip $(ip addr show dev br-ex scope global | grep "inet " | sed -e 's#.*inet ##g' -e 's#/.*##g'|head -n )
- openstack-config --set /etc/cinder/cinder.conf oslo_concurrency lock_path /var/lib/cinder/tmp
- openstack-config --set /etc/cinder/cinder.conf DEFAULT glance_api_servers http://controller:9292
- # [任一节点]生成数据库
- su -s /bin/sh -c "cinder-manage db sync" cinder
- # 所有控制节点修改计算节点配置
- openstack-config --set /etc/nova/nova.conf cinder os_region_name RegionOne
- # 重启计算节点 nova-api
- # pcs resource restart openstack-nova-api-clone
- # 安装配置存储节点 ,存储节点和控制节点复用
- # 所有节点
- yum install lvm2 -y
- systemctl enable lvm2-lvmetad.service
- systemctl start lvm2-lvmetad.service
- pvcreate /dev/sdb
- vgcreate cinder-volumes /dev/sdb
- yum install openstack-cinder targetcli python-keystone -y
- # 所有控制节点修改部分配置文件
- openstack-config --set /etc/cinder/cinder.conf lvm volume_driver cinder.volume.drivers.lvm.LVMVolumeDriver
- openstack-config --set /etc/cinder/cinder.conf lvm volume_group cinder-volumes
- openstack-config --set /etc/cinder/cinder.conf lvm iscsi_protocol iscsi
- openstack-config --set /etc/cinder/cinder.conf lvm iscsi_helper lioadm
- openstack-config --set /etc/cinder/cinder.conf DEFAULT enabled_backends lvm
- # 增加haproxy.cfg配置文件
- vim /etc/haproxy/haproxy.cfg
- listen cinder_api_cluster
- bind 192.168.10.100:
- balance source
- option tcpka
- option httpchk
- option tcplog
- server controller01 192.168.10.101: check inter rise fall
- server controller02 192.168.10.102: check inter rise fall
- server controller03 192.168.10.103: check inter rise fall
- scp /etc/haproxy/haproxy.cfg controller02:/etc/haproxy/haproxy.cfg
- scp /etc/haproxy/haproxy.cfg controller03:/etc/haproxy/haproxy.cfg
- # [任一节点]添加pacemaker资源
- pcs resource create openstack-cinder-api systemd:openstack-cinder-api --clone interleave=true
- pcs resource create openstack-cinder-scheduler systemd:openstack-cinder-scheduler --clone interleave=true
- pcs resource create openstack-cinder-volume systemd:openstack-cinder-volume
- pcs constraint order start openstack-keystone-clone then openstack-cinder-api-clone
- pcs constraint order start openstack-cinder-api-clone then openstack-cinder-scheduler-clone
- pcs constraint colocation add openstack-cinder-scheduler-clone with openstack-cinder-api-clone
- pcs constraint order start openstack-cinder-scheduler-clone then openstack-cinder-volume
- pcs constraint colocation add openstack-cinder-volume with openstack-cinder-scheduler-clone
- # 重启集群
- bash restart-pcs-cluster.sh
- # [任一节点]测试
- . /root/keystonerc_admin
- cinder service-list
- ```
- #### 、安装配置ceilometer和aodh集群
- ##### 7.1 安装配置ceilometer集群
- 实在无力吐槽这个项目,所以不想写了
- ##### 7.2 安装配置aodh集群
- 实在无力吐槽这个项目,所以不想写了
- #### 四、安装配置计算节点
- ##### 4.1 OpenStack Compute service
- ```
- # 所有计算节点
- yum install -y openstack-nova-compute
- # 修改配置文件/etc/nova/nova.conf
- openstack-config --set /etc/nova/nova.conf DEFAULT my_ip $(ip addr show dev ens160 scope global | grep "inet " | sed -e 's#.*inet ##g' -e 's#/.*##g')
- openstack-config --set /etc/nova/nova.conf DEFAULT use_neutron True
- openstack-config --set /etc/nova/nova.conf DEFAULT firewall_driver nova.virt.firewall.NoopFirewallDriver
- openstack-config --set /etc/nova/nova.conf DEFAULT memcached_servers controller01:,controller02:,controller03:
- openstack-config --set /etc/nova/nova.conf DEFAULT rpc_backend rabbit
- openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_hosts controller01:,controller02:,controller03:
- openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_ha_queues true
- openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_retry_interval
- openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_retry_backoff
- openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_max_retries
- openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_durable_queues true
- openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_userid openstack
- openstack-config --set /etc/nova/nova.conf oslo_messaging_rabbit rabbit_password openstack
- openstack-config --set /etc/nova/nova.conf DEFAULT auth_strategy keystone
- openstack-config --set /etc/nova/nova.conf keystone_authtoken auth_uri http://controller:5000
- openstack-config --set /etc/nova/nova.conf keystone_authtoken auth_url http://controller:35357
- openstack-config --set /etc/nova/nova.conf keystone_authtoken memcached_servers controller01:,controller02:,controller03:
- openstack-config --set /etc/nova/nova.conf keystone_authtoken auth_type password
- openstack-config --set /etc/nova/nova.conf keystone_authtoken project_domain_name default
- openstack-config --set /etc/nova/nova.conf keystone_authtoken user_domain_name default
- openstack-config --set /etc/nova/nova.conf keystone_authtoken project_name service
- openstack-config --set /etc/nova/nova.conf keystone_authtoken username nova
- openstack-config --set /etc/nova/nova.conf keystone_authtoken password nova
- openstack-config --set /etc/nova/nova.conf oslo_concurrency lock_path /var/lib/nova/tmp
- openstack-config --set /etc/nova/nova.conf vnc enabled True
- openstack-config --set /etc/nova/nova.conf vnc vncserver_listen 0.0.0.0
- openstack-config --set /etc/nova/nova.conf vnc vncserver_proxyclient_address $(ip addr show dev ens160 scope global | grep "inet " | sed -e 's#.*inet ##g' -e 's#/.*##g')
- openstack-config --set /etc/nova/nova.conf vnc novncproxy_base_url http://192.168.10.100:6080/vnc_auto.html
- openstack-config --set /etc/nova/nova.conf glance api_servers http://controller:9292
- openstack-config --set /etc/nova/nova.conf libvirt virt_type $(count=$(egrep -c '(vmx|svm)' /proc/cpuinfo); if [ $count -eq ];then echo "qemu"; else echo "kvm"; fi)
- # 打开虚拟机迁移的监听端口
- sed -i -e "s#\#listen_tls *= *0#listen_tls = 0#g" /etc/libvirt/libvirtd.conf
- sed -i -e "s#\#listen_tcp *= *1#listen_tcp = 1#g" /etc/libvirt/libvirtd.conf
- sed -i -e "s#\#auth_tcp *= *\"sasl\"#auth_tcp = \"none\"#g" /etc/libvirt/libvirtd.conf
- sed -i -e "s#\#LIBVIRTD_ARGS *= *\"--listen\"#LIBVIRTD_ARGS=\"--listen\"#g" /etc/sysconfig/libvirtd
- #启动服务
- systemctl enable libvirtd.service openstack-nova-compute.service
- systemctl start libvirtd.service openstack-nova-compute.service
- ```
- ##### 4.2 OpenStack Network service
- ```
- # 安装组件
- yum install -y openstack-neutron-openvswitch ebtables ipset
- yum install -y openstack-neutron openstack-neutron-ml2 openstack-neutron-openvswitch
- # 修改/etc/neutron/neutron.conf
- openstack-config --set /etc/neutron/neutron.conf DEFAULT rpc_backend rabbit
- openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_hosts controller01:,controller02:,controller03:
- openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_ha_queues true
- openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_retry_interval
- openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_retry_backoff
- openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_max_retries
- openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_durable_queues true
- openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_userid openstack
- openstack-config --set /etc/neutron/neutron.conf oslo_messaging_rabbit rabbit_password openstack
- openstack-config --set /etc/neutron/neutron.conf DEFAULT auth_strategy keystone
- openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_uri http://controller:5000
- openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_url http://controller:35357
- openstack-config --set /etc/neutron/neutron.conf keystone_authtoken memcached_servers controller01:,controller02:,controller03:
- openstack-config --set /etc/neutron/neutron.conf keystone_authtoken auth_type password
- openstack-config --set /etc/neutron/neutron.conf keystone_authtoken project_domain_name default
- openstack-config --set /etc/neutron/neutron.conf keystone_authtoken user_domain_name default
- openstack-config --set /etc/neutron/neutron.conf keystone_authtoken project_name service
- openstack-config --set /etc/neutron/neutron.conf keystone_authtoken username neutron
- openstack-config --set /etc/neutron/neutron.conf keystone_authtoken password neutron
- openstack-config --set /etc/neutron/neutron.conf oslo_concurrency lock_path /var/lib/neutron/tmp
- ### 配置Open vSwitch agent,/etc/neutron/plugins/ml2/openvswitch_agent.ini,注意,此处填写第二块网卡
- openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini securitygroup enable_security_group True
- openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini securitygroup enable_ipset True
- openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini securitygroup firewall_driver iptables_hybrid
- openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini ovs local_ip $(ip addr show dev ens192 scope global | grep "inet " | sed -e 's#.*inet ##g' -e 's#/.*##g')
- openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini agent tunnel_types vxlan
- openstack-config --set /etc/neutron/plugins/ml2/openvswitch_agent.ini agent l2_population True
- ### 配置nova和neutron集成,/etc/nova/nova.conf
- openstack-config --set /etc/nova/nova.conf neutron url http://controller:9696
- openstack-config --set /etc/nova/nova.conf neutron auth_url http://controller:35357
- openstack-config --set /etc/nova/nova.conf neutron auth_type password
- openstack-config --set /etc/nova/nova.conf neutron project_domain_name default
- openstack-config --set /etc/nova/nova.conf neutron user_domain_name default
- openstack-config --set /etc/nova/nova.conf neutron region_name RegionOne
- openstack-config --set /etc/nova/nova.conf neutron project_name service
- openstack-config --set /etc/nova/nova.conf neutron username neutron
- openstack-config --set /etc/nova/nova.conf neutron password neutron
- ln -s /etc/neutron/plugins/ml2/ml2_conf.ini /etc/neutron/plugin.ini
- systemctl restart openstack-nova-compute.service
- systemctl start openvswitch.service
- systemctl restart neutron-openvswitch-agent.service
- systemctl enable openvswitch.service
- systemctl enable neutron-openvswitch-agent.service
- ```
- #### 五 修补
- 控制节点:
- ```
- GRANT ALL PRIVILEGES ON *.* TO 'root'@'controller01' IDENTIFIED BY "openstack";
- GRANT ALL PRIVILEGES ON *.* TO 'root'@'controller02' IDENTIFIED BY "openstack";
- GRANT ALL PRIVILEGES ON *.* TO 'root'@'controller03' IDENTIFIED BY "openstack";
- GRANT ALL PRIVILEGES ON *.* TO 'root'@'192.168.10.101' IDENTIFIED BY "openstack";
- GRANT ALL PRIVILEGES ON *.* TO 'root'@'192.168.10.102' IDENTIFIED BY "openstack";
- GRANT ALL PRIVILEGES ON *.* TO 'root'@'192.168.10.103' IDENTIFIED BY "openstack";
- ```
- rabbitmq集群相关:
- ```
- /sbin/service rabbitmq-server stop
- /sbin/service rabbitmq-server start
- ```
- ```
- # 设置资源超时时间
- pcs resource op defaults timeout=90s
- # 清除错误
- pcs resource cleanup openstack-keystone-clone
- ```
- ##### mariadb集群排错
- ```
- 报错描述如下:节点启动不了,查看 tailf /var/log/messages日志发现如下报错:
- [ERROR] WSREP: gcs/src/gcs_group.cpp:group_post_state_exchange():
- 解决错误: rm -f /var/lib/mysql/grastate.dat
- 然后重启服务即可
- ```
- #### 六 增加dvr功能
- ##### 6.1 控制节点配置
- ```
- vim /etc/neutron/neutron.conf
- [DEFAULT]
- router_distributed = true
- vim /etc/neutron/plugins/ml2/ml2_conf.ini
- mechanism_drivers = openvswitch,linuxbridge,l2population
- vim /etc/neutron/plugins/ml2/openvswitch_agent.ini
- [DEFAULT]
- enable_distributed_routing = true
- [agent]
- l2_population = True
- vim /etc/neutron/l3_agent.ini
- [DEFAULT]
- agent_mode = dvr_snat
- vim /etc/openstack-dashboard/local_settings
- 'enable_distributed_router': True,
- 重启控制节点neutron相关服务,重启httpd服务
- ```
- ##### 6.2 计算节点配置
- ```
- vim /etc/neutron/plugins/ml2/ml2_conf.ini
- [ml2]
- mechanism_drivers = openvswitch,l2population
- vim /etc/neutron/plugins/ml2/openvswitch_agent.ini
- [DEFAULT]
- enable_distributed_routing = true
- [agent]
- l2_population = True
- vim /etc/neutron/l3_agent.ini
- [DEFAULT]
- interface_driver = openvswitch
- external_network_bridge =
- agent_mode = dvr
- 重启neutron相关服务
- ovs-vsctl add-br br-ex
- ovs-vsctl add-port br-ex ens160
- openstack-service restart neutron
- ```
- 关于rabbitmq连接数限制问题:
- ```
- [root@controller01 ~]# cat /etc/security/limits.d/-nproc.conf
- # Default limit for number of user's processes to prevent
- # accidental fork bombs.
- # See rhbz # for reasoning.
- * soft nproc
- root soft nproc unlimited
- * soft nofile
- * hard nofile
- [root@controller01 ~]#ulimit -n
- [root@controller01 ~]#cat /usr/lib/systemd/system/rabbitmq-server.service
- [Service]
- LimitNOFILE= #在启动脚本中添加此参数
- [root@controller01 ~]#systemctl daemon-reload
- [root@controller01 ~]#systemctl restart rabbitmq-server.service
- [root@controller01 ~]#rabbitctl status
- {file_descriptors,[{total_limit,},
- {total_used,},
- {sockets_limit,},
- {sockets_used,}]}
- ```
- #### 关于高可用路由器
- 只能在系统管理员页面上创建高可用或者DVR分布式路由器
- # 关于image镜像共享
- 把控制节点中的/var/lib/glance/images 镜像目录共享出来。
- yum -y install nfs-utils rpcbind -y
- mkdir /opt/glance/images/ -p
- vim /etc/exports
- /opt/glance/images/ 10.128.246.0/(rw,no_root_squash,no_all_squash,sync)
- exportfs -r
- systemctl enable rpcbind.service
- systemctl start rpcbind.service
- systemctl enable nfs-server.service
- systemctl start nfs-server.service
- 、2个nova节点查看
- showmount -e 10.128.247.153
- # 三个控制节点挂载
- mount -t nfs 10.128.247.153:/opt/glance/images/ /var/lib/glance/iamges/
- chown -R glance.glance /opt/glance/images/
- ##########
- 普通用户创建HA路由器
- ```
- neutron router-create router_demo --ha True
- ```
OpenStack Mitaka HA部署方案(随笔)的更多相关文章
- Neutron 物理部署方案 - 每天5分钟玩转 OpenStack(68)
前面我们讨论了 Neutron 的架构,本节讨论 Neutron 的物理部署方案:不同节点部署不同的 Neutron 服务组件. 方案1:控制节点 + 计算节点 在这个部署方案中,OpenStack ...
- openstack项目【day24】:OpenStack mitaka部署
前言: openstack的部署非常简单,简单的前提建立在扎实的理论功底,本人一直觉得,玩技术一定是理论指导实践,网上遍布个种搭建方法都可以实现一个基本的私有云环境,但是诸位可曾发现,很多配置都是重复 ...
- OpenStack Mitaka安装
http://egon09.blog.51cto.com/9161406/1839667 前言: openstack的部署非常简单,简单的前提建立在扎实的理论功底,本人一直觉得,玩技术一定是理论指导实 ...
- 在Openstack H版部署Nova Cell 时 ,终端输入nova service-list 和 nova host-list 命令将报错
关于Cell的基本介绍,可以参考贤哥的一篇文章: [OpenStack]G版中关于Nova的Cell http://blog.csdn.net/lynn_kong/article/details/8 ...
- openstack controller ha测试环境搭建记录(一)——操作系统准备
为了初步了解openstack controller ha的工作原理,搭建测试环境进行学习. 在学习该方面知识时,当前采用的操作系统版本是centos 7.1 x64.首先在ESXi中建立2台用于测试 ...
- Hadoop2.0 Namenode HA实现方案
Hadoop2.0 Namenode HA实现方案介绍及汇总 基于社区最新release的Hadoop2.2.0版本,调研了hadoop HA方面的内容.hadoop2.0主要的新特性(Hadoop2 ...
- eql高可用部署方案
运行环境 服务器两台(后面的所有配置案例都是以10.96.0.64和10.96.0.66为例) 操作系统CentOS release 6.2 必须要有共同的局域网网段 两台服务器都要安装keepali ...
- Win10+VirtualBox+Openstack Mitaka
首先VirtualBox安装的话,没有什么可演示的,去官网(https://www.virtualbox.org/wiki/Downloads)下载,或者可以去(https://www.virtual ...
- openstack高可用集群21-生产环境高可用openstack集群部署记录
第一篇 集群概述 keepalived + haproxy +Rabbitmq集群+MariaDB Galera高可用集群 部署openstack时使用单个控制节点是非常危险的,这样就意味着单个节 ...
随机推荐
- Python--(并发编程之线程Part2)
GIL只能保证垃圾回收机制的安全,进程中的数据安全还是需要自定义锁 线程执行代码首先要抢到GIL全局锁,假设线程X首先抢到,以为要抢到自定义锁要执行代码,所以这个线程在执行代码的时候就很容抢到了自定义 ...
- Codeforce 475 C. Kamal-ol-molk's Painting
从最左上的点開始枚举长宽.... C. Kamal-ol-molk's Painting time limit per test 2 seconds memory limit per test 256 ...
- oracle入门(1)——安装oracle 11g x64 for windows
[本文简介] 最近因为一个项目的需要,从零学习起了oracle,现在把学到的东西记录分享一下. 首先是安装篇,在win8 装10G 一直失败,网上各种方法都试过了,最后不得不放弃,选择了11G. 11 ...
- zz Qt下 QString转char*和char []
以下内容为转载:http://www.cnblogs.com/Romi/archive/2012/03/12/2392478.html -------------------------------- ...
- mysql第二天作业
create database 数据库名 default charset utf8;use 数据库名;1.创建成绩表,字段包括:学生姓名,语文成绩,数学成绩,英语成绩create table resu ...
- CodeMatic动软自动生成Nhibernate
前两天调查了下自动生成工具MyGeneration和codesmith前一个版本已经不更新了后面一个太高级生成 的代码包含了太多东西,没整明白.不过生成的xmlmapping很强大.所以干脆整合一下c ...
- Word 中设置图、表、公式、代码要与正文之间行间距
一.概述 在撰写论文等文档时,常常对图.表.公式.代码要与正文之间行间距有要求.例如: (5)图.表.公式.代码要与正文之间有6磅的行间距. 二.设置方式 选中 图/表/公式/代码 与 图题/表头/- ...
- POJ_2533 Longest Ordered Subsequence【DP】【最长上升子序列】
POJ_2533 Longest Ordered Subsequence[DP][最长递增子序列] Longest Ordered Subsequence Time Limit: 2000MS Mem ...
- 移动端1px的border
移动端浏览器解决1px的底部border问题 1.使用border:1px solid #e0e0e0. 在不同设备下由于devicePixelRatio不同导致1px实际显示的长度不同.所以在移动端 ...
- Windows 2008下系统网站运行环境的搭建
1.右击[计算机]-->[管理],进入到”服务器管理器” 界面,如图所示: 2.依次展开[角色]-->[Web服务器(IIS)]-->[Internet 信息服务(IIS)管理器], ...