多puppetmaster，多ca，keepalived+haproxy（nginx）puppet集群搭建

多puppetmaster，多ca，keepalived+haproxy（nginx）puppet集群搭建

一、服务器详情

192.168.122.111 pm01.jq.com pm01 #（puppetmaster服务器）

192.168.122.112 pm02.jq.com pm02 #（puppetmaster服务器）

192.168.122.121 ag01.jq.com ag01 #（puppet agent服务器）

192.168.122.122 ag02.jq.com ag02 #（puppet agent服务器）

192.168.122.131 ca01.jq.com ca01 #（puppet ca服务器）

192.168.122.132 ca02.jq.com ca02 #（puppet ca服务器）

192.168.122.141 lvs01.jq.com lvs01 #（puppet 负载服务器）

192.168.122.142 lvs02.jq.com lvs02 #（puppet 负载服务器）

#vip暂时用于测试，使用ip addr的方式绑定，后续用高可用软件实现bind

192.168.122.130 pc.jq.com pc #（ca服务器的vip，前期绑定在ca01服务器上）

192.168.122.115 lvs.jq.com lvs #（负载服务器的vip，前期绑定在puppetmaster上，后期需要绑定在lvs服务器上）

二、CA服务器部署

CA服务器单独用于签署和撤销证书，当puppetCA服务不可用时，新的客户端将不能获得证书，从而会影响使用，而已签发证书的客户端缺不受影响。因此将CA进行独立队架构，这对容错性而言是非常有必要的。

2.1 安装软件包

[root@ca01 ~]# groupadd -g 3000 puppet

[root@ca01 ~]# useradd -u 3000 -g 3000 puppet

[root@ca01 ~]# yum install puppet puppet-server –y

2.2 bind vip

绑定ca的vip 192.168.122.130到ca01服务器

[root@ca01 ~]#ip addr add 192.168.122.130/24 dev eth0

2.3 生成证书

使用puppet cert命令生成CA服务器与服务器域名证书。生成pc.jq.com和lvs.jq.com两个域名的授权证书文件,前面是证书别名，后面是证书名称。

[root@ca01 ssl]# puppet  cert --generate --dns_alt_names pc pc.jq.com

[root@ca01 ssl]# puppet  cert --generate --dns_alt_names lvs lvs.jq.com [root@ca01 ssl]# puppet cert --list --all

+ "lvs.jq.com" (SHA256) D6:5B:51:D6:6E:35:61:A4:45:D8:37:17:5B:85:A1:1B:34:BB:2F:D7:48:E8:44:57:B7:1D:42:8E:11:18:81:34 (alt names: "DNS:lvs", "DNS:lvs.jq.com")

+ "pc.jq.com"  (SHA256) A7:71:E1:46:1E:F0:F1:70:72:E3:B5:16:03:91:17:6D:68:5B:55:39:B6:79:6B:30:DD:41:ED:10:21:27:2A:33 (alt names: "DNS:pc", "DNS:pc.jq.com")

2.4 配置puppet.conf，添加标签[master]

[root@ca01 ~]# cat /etc/puppet/puppet.conf  | grep -v "#"

[main]

logdir = /var/log/puppet

rundir = /var/run/puppet

ssldir = $vardir/ssl

pluginsync = false

[agent]

classfile = $vardir/classes.txt

localconfig = $vardir/localconfig

server = lvs.jq.com

ca_server = pc.jq.com

environment = jqprd

[master]

confdir = /etc/puppet

certname = pc.jq.com

ca = true #开启CA认证

2.5 启动puppetmaster，CA部署完成

[root@ca01 ~]# /etc/init.d/puppetmaster start

[root@ca01 ~]# chkconfig puppetmaster on

2.6 ca02服务部署

ca02跟ca01的部署方式完全一致，证书是从ca01 拷贝过来的。直接copy /var/lib/ssl/puppet目录

三、PuppetMaster服务器部署

PuppetMaster服务器部署可采用默认的WebRick方式，也可以采用apache+passenger或nginx+passenger方式。

3.1 WebRick方式：

3.1.1 安装软件包

[root@pm01 ~]# groupadd -g 3000 puppet

[root@pm01 ~]# useradd -u 3000 -g 3000 puppet

[root@pm01 ~]# yum install puppet puppet-server -y

3.1.2 设置hosts文件

[root@pm01 ~]# vim /etc/hosts

192.168.122.111 pm01.jq.com pm01

192.168.122.112 pm02.jq.com pm02

192.168.122.121 ag01.jq.com ag01

192.168.122.122 ag02.jq.com ag02

192.168.122.131 ca01.jq.com ca01

192.168.122.132 ca02.jq.com ca02

192.168.122.141 lvs01.jq.com lvs01

192.168.122.142 lvs02.jq.com lvs02

192.168.122.130 pc.jq.com pc

192.168.122.115 lvs.jq.com lvs

3.1.3 bind master vip

绑定LVS的vip 192.168.122.115到pm01服务器，测试用，在没有负载之前，绑定在master上。

ip addr add 192.168.122.115/24 dev eth0

3.1.4 创建证书目录

[root@pm01 ~]# mkdir /var/lib/puppet/ssl/{certs,ca,private_keys} –p

3.1.5 将puppetca上生成的puppetmaster公钥、私钥和根证书复制到pm01

scp -r root@192.168.122.130:/var/lib/puppet/ssl/ca/signed/lvs.jq.com.pem /var/lib/puppet/ssl/certs/lvs.jq.com.pem

scp -r root@192.168.122.130:/var/lib/puppet/ssl/ca/ca_crt.pem /var/lib/puppet/ssl/certs/ca.pem

scp -r root@192.168.122.130:/var/lib/puppet/ssl/private_keys/lvs.jq.com.pem /var/lib/puppet/ssl/private_keys/lvs.jq.com.pem

scp -r root@192.168.122.130:/var/lib/puppet/ssl/ca/ca_crl.pem /var/lib/puppet/ssl/ca/ca_crl.pem

3.1.6 配置puppet.conf，添加标签[master]，关闭ca

[root@pm01 ~]# grep -v "#" /etc/puppet/puppet.conf

[main]

logdir = /var/log/puppet

rundir = /var/run/puppet

ssldir = $vardir/ssl

privatekeydir = $ssldir/private_keys { group = service }

hostprivkey = $privatekeydir/$certname.pem { mode = 640 }

[agent]

classfile = $vardir/classes.txt

localconfig = $vardir/localconfig

server = lvs.jq.com #puppetmaster域名，应该与之前手动生成的证书匹配

ca_server = pc.jq.com #ca证书服务器

[master]

certname = lvs.jq.com #puppetmaster的域名，应该与之前手动生成的证书匹配

ca = false #关闭ca验证

3.1.7 启动puppetmaster服务，puppetmaster部署完成

[root@pm01 ssl]# /etc/init.d/puppetmaster restart

3.1.8 运行puppet命令进行本地证书申请

[root@pm01 ssl]# puppet  agent -t

Info: Creating a new SSL key for pm01.jq.com

Info: csr_attributes file loading from /etc/puppet/csr_attributes.yaml

Info: Creating a new SSL certificate request for pm01.jq.com

Info: Certificate Request fingerprint (SHA256): 2C:09:32:E1:13:CA:0F:44:3B:93:4B:0F:0E:2D:46:19:3A:37:E1:47:C7:D3:E8:2C:A6:83:44:B3:D3:94:63:D6

Exiting; no certificate found and waitforcert is disabled

3.1.9 登录puppetca进行证书签发

[root@ca01 ~]# puppet cert --sign pm01.jq.com

Notice: Signed certificate request for pm01.jq.com

Notice: Removing file Puppet::SSL::CertificateRequest pm01.jq.com at '/var/lib/puppet/ssl/ca/requests/pm01.jq.com.pem'

3.1.10 再次运行puppet命令进行测试连通性

[root@pm01 ssl]# puppet  agent -t

Info: Caching certificate for pm01.jq.com

Info: Caching certificate_revocation_list for ca

Info: Caching certificate for pm01.jq.com

Info: Retrieving pluginfacts

Info: Retrieving plugin

Info: Caching catalog for pm01.jq.com

Info: Applying configuration version '1425526708'

Notice: Finished catalog run in 0.17 seconds

3.1.11 在kspupt-ca上申请本地证书

[root@ca01 ~]# vim /etc/puppet/puppet.conf

[agent]

server    = lvs.jq.com

ca_server = pc.jq.com

[root@ca01 ~]# puppet agent -t

[root@ca01 ~]# puppet cert --sign ca01.jq.com

[root@ca01 ~]# puppet agent –t

3.2 Nginx+Passenger方式：

注：可参考 http://kisspuppet.com/2014/10/20/puppet_learning_ext4/

3.2.1、配置nginx

[root@pm01 ssl]# cat /usr/local/nginx/conf/vhosts/passenger.conf

server {

listen 8140                ssl;

server_name                puppetmaster;

passenger_enabled          on;

passenger_set_cgi_param    HTTP_X_CLIENT_DN $ssl_client_s_dn;

passenger_set_cgi_param    HTTP_X_CLIENT_VERIFY $ssl_client_verify;

proxy_buffer_size 4000k;

proxy_buffering on;

proxy_buffers 32 1280k;

proxy_busy_buffers_size 17680k;

client_max_body_size 10m;

client_body_buffer_size 4096k;

access_log /var/log/nginx/puppet_access.log;

error_log /var/log/nginx/puppet_error.log;

root /etc/puppet/rack/public;

ssl off;

ssl_session_timeout 5m;

ssl_certificate /var/lib/puppet/ssl/certs/lvs.jq.com.pem;

ssl_certificate_key /var/lib/puppet/ssl/private_keys/lvs.jq.com.pem;

ssl_client_certificate /var/lib/puppet/ssl/certs/ca.pem;

ssl_crl /var/lib/puppet/ssl/ca/ca_crl.pem;

ssl_verify_client optional;

ssl_ciphers SSLv2:-LOW:-EXPORT:RC4+RSA;

ssl_prefer_server_ciphers on;

ssl_verify_depth 1;

ssl_session_cache shared:SSL:128m;

# File sections

location /production/file_content/files/ {

types { }

default_type application/x-raw;

alias /etc/puppet/files/;

}

}

3.2.2、配置puppet.conf

[root@pm01 ssl]# grep -v "#" /etc/puppet/puppet.conf

[main]

logdir = /var/log/puppet

rundir = /var/run/puppet

ssldir = $vardir/ssl

privatekeydir = $ssldir/private_keys { group = service }

hostprivkey = $privatekeydir/$certname.pem { mode = 640 }

[agent]

classfile = $vardir/classes.txt

localconfig = $vardir/localconfig

server = lvs.jq.com

ca_server = pc.jq.com

[master]

certname = lvs.jq.com

ca = false

ssl_client_verify_header = HTTP_X_CLIENT_VERIFY

ssl_client_header = HTTP_X_CLIENT_DN

3.3 master02服务器部署

master02的部署与master01的完全一致，包括拷贝证书部分

4 Puppet LB负载均衡器部署

4.1 puppet认证建立

4.1.1、安装软件包

[root@lvs01 ~]# groupadd -g 3000 puppet

[root@lvs01 ~]# useradd -u 3000 -g 3000 puppet

[root@lvs01 ~]# yum install puppet

4.1.2、编辑hosts文件

[root@lvs01 ~]# vim /etc/hosts

192.168.122.111 pm01.jq.com pm01

192.168.122.112 pm02.jq.com pm02

192.168.122.121 ag01.jq.com ag01

192.168.122.122 ag02.jq.com ag02

192.168.122.131 ca01.jq.com ca01

192.168.122.132 ca02.jq.com ca02

192.168.122.141 lvs01.jq.com lvs01

192.168.122.142 lvs02.jq.com lvs02

192.168.122.130 pc.jq.com pc

192.168.122.115 lvs.jq.com lvs

 

4.1.3、创建证书目录

[root@lvs01 ~]# mkdir /var/lib/puppet/ssl/{certs,ca,private_keys} –p

4.1.4、将ca上生成的puppetmaster公钥、私钥和根证书复制到lvs01

scp -r root@192.168.122.130:/var/lib/puppet/ssl/ca/signed/lvs.jq.com.pem /var/lib/puppet/ssl/certs/lvs.jq.com.pem

scp -r root@192.168.122.130:/var/lib/puppet/ssl/ca/ca_crt.pem /var/lib/puppet/ssl/certs/ca.pem

scp -r root@192.168.122.130:/var/lib/puppet/ssl/private_keys/lvs.jq.com.pem /var/lib/puppet/ssl/private_keys/lvs.jq.com.pem

scp -r root@192.168.122.130:/var/lib/puppet/ssl/ca/ca_crl.pem /var/lib/puppet/ssl/ca/ca_crl.pem

4.1.5、配置puppet.conf，修改标签[agent]，增加server和ca_server字段

[root@lvs01 ~]# vim /etc/puppet/puppet.conf

[agent]

server      = lvs.jq.com

ca_server   = pc.jq.com

4.1.6、运行puppet命令进行本地证书申请

[root@lvs01 ~]# puppet  agent -t

4.1.7、登录ca进行证书签发

[root@ca01 ~]# puppet  cert --sign lvs01.jq.com

4.1.8、再次运行puppet命令进行测试连通性

[root@lvs01 ~]# puppet agent -t

Info: Caching certificate for lvs01.jq.com

Info: Caching certificate_revocation_list for ca

Info: Caching certificate for lvs01.jq.com

Info: Loading facts

Info: Caching catalog for lvs01.jq.com

Info: Applying configuration version '1425527450'

Notice: Finished catalog run in 0.24 seconds

4.2 安装并配置nginx负载均衡器

4.2.1、安装nginx软件

[root@lvs01 ~]# groupadd -g 3001 nginx

[root@lvs01 ~]# useradd -u 3001 -g 3001 nginx

[root@lvs01 ~]# yum install nginx

4.2.2、临时设置VIP地址（后面通过高可用软件代替）

[root@lvs01 ~]# ip addr add 192.168.122.115/24 dev eth0

此处请将之前bind到pm01的vip取消

4.2.3、配置nginx虚拟主机，添加upstrem

[root@lvs01 ~]# cat /etc/nginx/conf.d/puppetmaster.conf

upstream puppet-master {

server 192.168.122.111:8140;

server 192.168.122.112:8140;

}

server {

listen         8140 ssl;

server_name    puppetmaster;

access_log     /var/log/nginx/puppet_access.log;

error_log      /var/log/nginx/puppet_error.log;

ssl_protocols SSLv3 TLSv1;

ssl_ciphers ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP;

proxy_set_header             X-SSL-Subject  $ssl_client_s_dn;

proxy_set_header             X-Client-DN  $ssl_client_s_dn;

proxy_set_header             X-Client-Verify  $ssl_client_verify;

client_max_body_size 100m;

client_body_buffer_size 1024k;

proxy_buffer_size 100m;

proxy_buffers 8 100m;

proxy_busy_buffers_size 100m;

proxy_temp_file_write_size 100m;

proxy_read_timeout 500;

ssl                     on;

ssl_session_timeout     5m;

ssl_certificate         /var/lib/puppet/ssl/certs/lvs.jq.com.pem;

ssl_certificate_key     /var/lib/puppet/ssl/private_keys/lvs.jq.com.pem;

ssl_client_certificate  /var/lib/puppet/ssl/certs/ca.pem;

ssl_crl                 /var/lib/puppet/ssl/ca/ca_crl.pem;

ssl_verify_client       optional;

ssl_prefer_server_ciphers  on;

ssl_verify_depth           1;

ssl_session_cache          shared:SSL:128m;

location / {

proxy_redirect    off;

proxy_pass        https://puppet-master;

}

}

4.2.4、编辑hosts文件，puppetmaster解析指向VIP

[root@lvs01 ~]# vim /etc/hosts

192.168.122.111 pm01.jq.com pm01

192.168.122.112 pm02.jq.com pm02

192.168.122.121 ag01.jq.com ag01

192.168.122.122 ag02.jq.com ag02

192.168.122.131 ca01.jq.com ca01

192.168.122.132 ca02.jq.com ca02

192.168.122.141 lvs01.jq.com lvs01

192.168.122.142 lvs02.jq.com lvs02

192.168.122.130 pc.jq.com pc

192.168.122.115 lvs.jq.com lvs

4.2.5、修改ca01和pm01的hosts文件puppetmaster解析

[root@kspupt-ca1 ~]# vim /etc/hosts

192.168.122.111 pm01.jq.com pm01

192.168.122.112 pm02.jq.com pm02

192.168.122.121 ag01.jq.com ag01

192.168.122.122 ag02.jq.com ag02

192.168.122.131 ca01.jq.com ca01

192.168.122.132 ca02.jq.com ca02

192.168.122.141 lvs01.jq.com lvs01

192.168.122.142 lvs02.jq.com lvs02

192.168.122.130 pc.jq.com pc

192.168.122.115 lvs.jq.com lvs

[root@pm01 ~]# vim /etc/hosts

192.168.122.111 pm01.jq.com pm01

192.168.122.112 pm02.jq.com pm02

192.168.122.121 ag01.jq.com ag01

192.168.122.122 ag02.jq.com ag02

192.168.122.131 ca01.jq.com ca01

192.168.122.132 ca02.jq.com ca02

192.168.122.141 lvs01.jq.com lvs01

192.168.122.142 lvs02.jq.com lvs02

192.168.122.130 pc.jq.com pc

192.168.122.115 lvs.jq.com lvs

4.2.6、启动nginx服务器

[root@lvs01 ~]# /etc/init.d/nginx start

4.2.7、再次运行puppet命令进行测试连通性

[root@kspupt-ca1 ~]# puppet  agent -t

[root@pm01 ~]# puppet  agent -t

[root@lvs01 ~]# puppet  agent -t

[root@pm01 ~]# tailf  /var/log/nginx/puppet_access.log

[root@lvs01 ~]# tailf /var/log/nginx/puppet_access.log

4.3 安装配置Haproxy负载均衡

安装haproxy和keepalived过程略去，网上很多教程

[root@lvs01 keepalived]# cat /etc/haproxy/haproxy.cfg

global

maxconn         40000

ulimit-n        500000

log             127.0.0.1 local0

uid             99

gid             99

chroot          /tmp

#       nbproc          4

daemon

defaults

log     global

retries 2

option redispatch

option dontlognull

balance roundrobin

timeout connect 30000ms

timeout client 30000ms

timeout server 30000ms

timeout check 2000

listen admin_stats

bind 0.0.0.0:8080

mode http

stats refresh 5s

stats enable

stats hide-version

stats realm Haproxy\ Statistics

stats uri /haproxy

stats auth admin:password

listen puppetmaster *:8140

mode tcp

option ssl-hello-chk

#    option tcplog

#balance source

#    balance roundrobin

balance source

server pm01 pm01.jq.com:8140 check inter 2000 fall 3

server pm02 pm02.jq.com:8140 check inter 2000 fall 3

4.4 配置keepalived，取消vip 通过ip addr 绑定

Keepalived的备机配置略去，网上也可以搜索到，需要修改的地方很少。

[root@lvs01 ~]# cat /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {

notification_email {

test@gmail.com

}

notification_email_from Alexandre.Cassen@firewall.loc

smtp_server 127.0.0.1

smtp_connect_timeout 30

router_id LVS_DEVEL

}

vrrp_script chk_http_port {

script "/etc/keepalived/check_haproxy.sh"

interval 2

weight 2

}

vrrp_instance VI_1 {

state MASTER

interface eth0

virtual_router_id 51

priority 100

advert_int 1

authentication {

auth_type PASS

auth_pass 1111

}

track_script {

chk_http_port

}

virtual_ipaddress {

192.168.122.115 #此处为负载的VIP，配置keepalived之后，切记去掉之前ip addr的绑定

}

}

4.4.1 Keepalived监控haproxy脚本

[root@lvs01 ~]# cat /etc/keepalived/check_haproxy.sh

#!/bin/bash

. /etc/profile

A=`ps -C haproxy --no-header |wc -l`

if [ $A -eq 0 ];then

/etc/init.d/haproxy start

sleep 3

if [ `ps -C haproxy --no-header |wc -l` -eq 0 ];then

/etc/init.d/keepalived stop

fi

fi

4.5 Lvs02服务器部署

Lvs02的配置与lvs01的配置完全一致，将此服务器作为lvs01的备机，包括keepalived+haproxy。

几乎照搬了http://kisspuppet.com/2014/10/21/puppet_learning_ext6/ 的文章，非常感谢kisspuppet！