ansible playbook部署ELK集群系统

一、介绍

总共4台机器，分别为

192.168.1.99

192.168.1.100

192.168.1.210

192.168.1.211

服务所在机器为：

redis：192.168.1.211

logstash: 192.168.210 192.168.1.211

elasticsearch: 分别布置在四台（211不为主、不存储数据）

kibana：192.168.1.211

nginx：192.168.1.211

二、elk集群架构图

三、为每一个软件或集群创建一个单独的角色。

cd /etc/ansible
mkdir -pv elk/roles/{jdk,redis,logstash,es,es_head,kibana,nginx}/{tasks,files,templates,meta,handlers,vars}

四、配置hosts

在/etc/ansible/hosts添加如下内容：

[jdk_servers]
192.168.1.99
192.168.1.100
192.168.1.210
192.168.1.211

[redis_servers]
192.168.1.211

[es_servers]
192.168.1.99  node_name=node-99  node_master=true node_data=true network_host=192.168.1.99
192.168.1.100 node_name=node-100 node_master=true node_data=true network_host=192.168.1.100
192.168.1.210 node_name=node-210 node_master=true node_data=true network_host=192.168.1.210
192.168.1.211 node_name=node-211 node_master=true node_data=true network_host=192.168.1.211

[es_plugin_head]
192.168.1.211

[logstash_servers]
192.168.1.210
192.168.1.211

[kibana_servers]
192.168.1.211

[nginx_servers]
192.168.1.211

PS：es_servers组里面，后面定义了变量，因为es配置文件中，会根据服务器的不同，对应的配置也不同，在这里定义，可以直接在tasks里面去引用

五、JDK roles

1.把jdk-8u91-linux-x64.tar.gz和Java环境变量脚本传到files目录下

2.编写vars

[root@master jdk]# cat vars/main.yml
jdk_package_name: jdk-8u91-linux-x64.tar.gz
env_dir: /etc/profile.d

3.编写tasks

[root@master jdk]# cat tasks/main.yml
- name: copy jdk software
  copy: src={{ jdk_package_name }} dest=/tmp/ owner=root group=root
- name: uncompression jdk software
  shell: tar -zxf /tmp/{{ jdk_package_name }} -C /usr/local/
- name: copy jdk env
  copy: src=java.sh dest={{ env_dir }}/ owner=root group=root mode=0644
- name: enforce env
  shell: source {{ env_dir }}/java.sh
- name: delete jdk compressin files
  shell: rm -f /tmp/{{ jdk_package_name }}

4.使用角色

在roles同级目录，创建一个jdk.yml文件，里面定义好你的playbook。

[root@master elk]# cat jdk.yml
- hosts: 127.0.0.1
  remote_user: root
  roles:
    - jdk

5. 运行playbook安装jdk

[root@master elk]# ansible-playbook jdk.yml

六、redis roles

1.把redis.tar.gz传到files目录下，redis配置文件和启动脚本传到templates目录下

2.编写vars

[root@master redis]# cat vars/main.yml
redis_package_file: redis.tar.gz
redis_dir: /usr/local/redis
exec_dir: /etc/init.d

3.编写tasks

[root@master redis]# cat tasks/main.yml
- name: copy redis software
  copy: src={{ redis_package_file }} dest=/tmp/ owner=root group=root
- name: uncompression redis software
  shell: tar zxf /tmp/{{ redis_package_file }} -C /usr/local
- name: copy redis start script
  template: src=redis dest={{ exec_dir }}/ owner=root group=root mode=0755
- name: copy redis config
  template: src=redis.conf dest={{ redis_dir }}/etc/ owner=redis group=redis
- name: create redis user
  user: name=redis shell=/sbin/nologin
- name: change owner and group
  file: path={{ redis_dir }} owner=redis group=redis #recurse=yes
- name: mkdir directory for redis data
  file: dest=/data/redis mode=0755 state=directory owner=redis group=redis
- name: start redis service
  shell: /etc/init.d/redis start
- name: add boot start redis start
  shell: chkconfig --level 345 redis on
- name: delete redis compression files
  shell: rm -f /tmp/{{ redis_package_file }}

4.使用角色

在roles同级目录，创建一个redis.yml文件，里面定义好你的playbook

[root@master elk]# cat redis.yml
- hosts: redis_servers
  remote_user: root
  roles:
    - redis

5. 运行playbook安装redis

[root@master elk]# ansible-playbook redis.yml

七、logstash roles

1.把logstash.tar.gz传到files目录下，logstash自定义配置文件和启动脚本传到templates目录下

2.编写vars

[root@master logstash]# cat vars/main.yml
logstash_package_file: logstash.tar.gz
logstash_dir: /usr/local/logstash

3.编写tasks

[root@master logstash]# cat tasks/main.yml
- name: copy logstash software
  copy: src={{ logstash_package_file }} dest=/tmp/ owner=root group=root
- name: uncompression logstash software
  shell: tar zxf /tmp/{{ logstash_package_file }} -C /usr/local
- name: mkdir config files dir
  file: dest={{ logstash_dir }}/conf.d mode=0755 state=directory owner=es group=es
- name: copy logstash start script
  template: src=start.sh dest={{ logstash_dir }}/ owner=es group=es mode=0755
- name: copy logstash config
  template: src=visit.conf dest={{ logstash_dir }}/conf.d/ owner=es group=es
- name: create es user
  user: name=es
- name: change owner and group
  file: path={{ logstash_dir }} owner=es group=es
- name: start logstash service
  shell: su - es -c "{{ logstash_dir }}/start.sh"
- name: delete logstash compression files
  shell: rm -f /tmp/{{ logstash_package_file }}

4.使用角色

在roles同级目录，创建一个logstash.yml文件，里面定义好你的playbook

[root@master elk]# cat logstash.yml
- hosts: logstash_servers
  remote_user: root
  roles:
    - logstash

5. 运行playbook安装logstash

[root@master elk]# ansible-playbook logstash.yml

八、es roles

1.把elasticsearch.tar.gz传到files目录下，es配置文件和启动脚本传到templates目录下

2.编写vars

[root@master es]# cat vars/main.yml
es_package_file: elasticsearch.tar.gz
es_dir: /usr/local/elasticsearch
config_name: elasticsearch.yml

3.编写tasks

[root@master es]# cat tasks/main.yml
- name: copy es software
  copy: src={{ es_package_file }} dest=/tmp/ owner=root group=root
- name: uncompression es software
  shell: tar zxf /tmp/{{ es_package_file }} -C /usr/local
- name: copy es start script
  template: src=start.sh dest={{ es_dir }}/ owner=es group=es mode=0755
- name: copy es config
  template: src=elasticsearch.yml dest={{ es_dir }}/config/ owner=es group=es
- name: update es config
  shell: sed -i "{{ item }}" {{ es_dir }}/config/{{ config_name }}
  with_items:
    - 's/#node.name/node.name: {{ node_name }}/'
    - 's/#node.master/node.master: {{ node_master }}/'
    - 's/#node.data/node.data: {{ node_data }}/'
    - 's/#network.host/network.host: {{ network_host }}/'
- name: create es user
  user: name=es
- name: change owner and group
  file: path={{ es_dir }} owner=es group=es #recurse=yes
- name: mkdir directory for es data
  file: dest=/data/elasticsearch mode=0755 state=directory owner=es group=es
  notify: mkdir Subdirectory
- name: start es service
  shell: su es -c 'cd {{ es_dir }} && ./start.sh'
- name: delete es compression files
  shell: rm -f /tmp/{{ es_package_file }}

4.编写handlers

[root@master es]# cat handlers/main.yml
- name: mkdir Subdirectory
  file: dest=/data/elasticsearch/{{ item }} mode=0755 state=directory owner=es group=es
  with_items:
    - logs
    - data

5.使用角色

在roles同级目录，创建一个es.yml文件，里面定义好你的playbook

[root@master elk]# cat es.yml
- hosts: es_servers
  remote_user: root
  roles:
    - es

5. 运行playbook安装es

[root@master elk]# ansible-playbook es.yml

九、es-head roles

1.把node-v4.4.7-linux-x64.tar.gz和elasticsearch-head.tar.gz传到files目录下，把_site/app.js传到templates目录下

2.编写vars

[root@master es_head]# cat vars/main.yml
node_package_file: node-v4.4.7-linux-x64.tar.gz
node_dir: /usr/local/node-v4.4.7-linux-x64
head_package_file: elasticsearch-head.tar.gz
head_dir: /usr/local/elasticsearch/elasticsearch-head

3.编写tasks

[root@master es_head]# cat tasks/main.yml
- name: copy node software
  copy: src={{ node_package_file }} dest=/tmp/ owner=root group=root
- name: uncompression node software
  shell: tar zxf /tmp/{{ node_package_file }} -C /usr/local/
- name: copy node to env
  shell: rsync -a {{ node_dir  }}/bin/* /usr/bin/
- name: copy head plugin software
  copy: src={{ head_package_file }} dest=/tmp/ owner=root group=root
- name: uncopmression head plugin software
  shell: tar zxf /tmp/{{ head_package_file }} -C /usr/local/elasticsearch/
- name: copy head config
  template: src=app.js dest={{ head_dir }}/_site/ owner=es group=es
- name: change owner and group
  file: path={{ head_dir }} owner=es group=es #recurse=yes
- name: start es service
  shell: su es -c 'cd {{ head_dir }} && npm run start > {{ head_dir }}/run.log &'
- name: delete node compression files
  shell: rm -f /tmp/{{ node_package_file }}
- name: delete head compression files
  shell: rm -f /tmp/{{ head_package_file }}

4.使用角色

在roles同级目录，创建一个es_head.yml文件，里面定义好你的playbook

[root@master elk]# cat es_head.yml
- hosts: es_plugin_head
  remote_user: root
  roles:
    - es_head

5. 运行playbook安装es-head

ansible-playbook es_head.yml

十、kibana roles

1.把kibana.tar.gz传到files目录下，kibana配置文件和启动脚本传到templates目录下

2.编写vars

[root@master kibana]# cat vars/main.yml
kibana_package_file: kibana.tar.gz
kibana_dir: /usr/local/kibana

3.编写tasks

[root@master kibana]# cat tasks/main.yml
- name: copy kibana software
  copy: src={{ kibana_package_file }} dest=/tmp/ owner=root group=root
- name: uncompression kibana software
  shell: tar zxf /tmp/{{ kibana_package_file }} -C /usr/local
- name: copy kibana start script
  template: src=start.sh dest={{ kibana_dir }}/ owner=es group=es mode=0755
- name: copy kibana config
  template: src=kibana.yml dest={{ kibana_dir }}/config/ owner=es group=es
- name: change owner and group
  file: path={{ kibana_dir }} owner=es group=es
- name: start kibana service
  shell: su - es -c "{{ kibana_dir }}/start.sh"
- name: delete kibana compression files
  shell: rm -f /tmp/{{ kibana_package_file }}

4.使用角色

在roles同级目录，创建一个kibana.yml文件，里面定义好你的playbook

[root@master elk]# cat kibana.yml
- hosts: kibana_servers
  remote_user: root
  roles:
    - kibana

5. 运行playbook安装kibana

ansible-playbook kibana.yml

十一、nginx roles（211本身就已经安装nginx,如果目标机器没安装，请参考我之前的文章）

1.把htpasswd生成密码的对应文件传到files目录下，nginx配置文件传到templates目录下

2.编写vars

[root@master nginx]# cat vars/main.yml
nginx_basedir: /usr/local/nginx

3.编写tasks

[root@master nginx]# cat tasks/main.yml
- name: copy conf file
  template: src={{ item.src }} dest={{ nginx_basedir }}/{{ item.dest }} backup=yes owner=root group=root mode=0644
  with_items:
    - { src: es.conf, dest: conf/vhosts/ }
    - { src: kibana.conf, dest: conf/vhosts/ }
- name: copy passwd file
  copy: src={{ item }} dest={{ nginx_basedir }}/conf/htpasswd/ owner=root group=root mode=0644
  with_items:
    - es
    - kibana
  notify: restart nginx

4.编写handlers

[root@master nginx]# cat handlers/main.yml
- name: restart nginx
  shell: /usr/local/nginx/sbin/nginx -s reload

5.使用角色

在roles同级目录，创建一个nginx.yml文件，里面定义好你的playbook

[root@master elk]# cat nginx.yml
---
- hosts: nginx_servers
  user: root
  roles:
    - nginx

5. 运行playbook加载nginx

ansible-playbook nginx.yml