一.部署前环境介绍:

es集群5台(es01,es02,es03,es04,es05),logstash服务器1台(logstash2),kibana服务器1台(kibana2),模拟apache服务及filebeat(收集日志工具)1台(web2);以上均由虚拟机模拟实现;

ip分配如下:

192.168.1.11 es01  

192.168.1.12 es02

192.168.1.13 es03

192.168.1.14 es04

192.168.1.15 es05

192.168.1.21 logstash2

192.168.1.22 kibana2

192.168.1.31 web2

真机:192.168.1.254

通过ftp共享真机yum源在/var/ftp/elk和centos-1804

二.ansible-playbook应用

ansible服务器ip:192.168.1.40

配置ansible:

 echo "[es]
es01
es02
es03
es04
es05" >> /etc/ansible/hosts

1.部署脚本elk.yml

 ---
- name: 环境部署
hosts: es,logstash2,kibana2,web2
tasks:
- name: 环境部署
script: /root/elk.sh --some-arguments - name: es集群部署
hosts: es
tasks:
- name: 安装jdk,es
yum:
name: 'java-1.8.0-openjdk'
state: latest
- yum:
name: 'elasticsearch'
state: latest
- name: 修改配置文件
lineinfile:
path: /etc/elasticsearch/elasticsearch.yml
regexp: "{{ item.old }}"
line: "{{ item.new }}"
with_items:
- {old: '# cluster.name',new: 'cluster.name: myelk' }
- {old: '# network.host',new: 'network.host: 0.0.0.0' }
- {old: '# discovery.zen.ping.unicast.hosts',new:'discovery.zen.ping.unicast.hosts: ["es01", "es02","es03"]' }
- {old: '# node.name',new: 'node.name: {{ ansible_nodename }}' }
- name: reload es
service:
name: elasticsearch
state: restarted
enabled: yes
#必须在es部署之后执行
- name: es01的head和kopf插件安装
hosts: es01
tasks:
- name: 安装head插件
shell: '/usr/share/elasticsearch/bin/plugin install ftp://192.168.1.254/elk/elasticsearch-head-master.zip'
- name: 安装kopf插件
shell: '/usr/share/elasticsearch/bin/plugin install ftp://192.168.1.254/elk/elasticsearch-kopf-master.zip' - name: logstash部署
hosts: logstash2
tasks:
- name: 安装jdk,logstash
yum:
name: 'java-1.8.0-openjdk'
state: latest
- yum:
name: 'logstash'
state: latest
- name: 方便apache日志读取
script: /root/elk2.sh --some-arguments - name: kibana部署
hosts: kibana2
tasks:
- name: 安装kibana
yum:
name: 'kibana'
state: latest
- name: 修改配置文件
lineinfile:
path: /opt/kibana/config/kibana.yml
regexp: "{{ item.old2 }}"
line: "{{ item.new2 }}"
with_items:
- {old2: 'server.port',new2: ' server.port: 5601' }
- {old2: 'server.host',new2: ' server.host: "0.0.0.0"' }
- {old2: 'elasticsearch.url',new2: ' elasticsearch.url: "http://192.168.1.11:9200"' }
- {old2: 'kibana.index',new2: ' kibana.index: ".kibana"' }
- {old2: 'kibana.defaultAppId',new2: ' kibana.defaultAppId: "discover"' }
- {old2: 'elasticsearch.pingTimeout',new2: ' elasticsearch.pingTimeout: 1500' }
- {old2: 'elasticsearch.requestTimeout',new2: ' elasticsearch.requestTimeout: 30000' }
- {old2: 'elasticsearch.startupTimeout',new2: ' elasticsearch.startupTimeout: 5000' }
- name: reload kibana
service:
name: kibana
state: restarted
enabled: yes - name: web服务和filebeat部署
hosts: web2
tasks:
- name: 安装apache,filebeat
yum:
name: 'httpd'
state: latest
- yum:
name: 'filebeat'
state: latest
- name: 修改配置文件
lineinfile:
path: /etc/filebeat/filebeat.yml
regexp: "{{ item.old3 }}"
line: "{{ item.new3 }}"
with_items:
- {old3: 'elasticsearch:',new3: '# elasticsearch:' }
- {old3: 'localhost:9200"',new3: '#hosts: ["localhost:9200"]' }
- {old3: '#logstash:',new3: ' logstash:' }
- {old3: 'localhost:5044"',new3: ' hosts: ["192.168.1.21:5044"]' }
- replace:
path: /etc/filebeat/filebeat.yml
regexp: '{{ item.old4 }}'
replace: '{{ item.new4 }}'
backup: yes
with_items:
- {old4: '\*\.',new4: 'access_' }
- name: reload http,filebeat
service:
name: 'httpd'
state: restarted
enabled: yes
- service:
name: 'filebeat'
state: restarted
enabled: yes

2.调用的shell脚本

/root/elk.sh

 #!/bin/bash
echo "127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
192.168.1.11 es01
192.168.1.12 es02
192.168.1.13 es03
192.168.1.14 es04
192.168.1.15 es05
192.168.1.21 logstash2
192.168.1.22 kibana2" > /etc/hosts
mkdir /var/ftp/elk
echo "[local_repo]
name=CentOS-$releasever - Base
baseurl="ftp://192.168.1.254/centos-1804"
enabled=
gpgcheck=
[elk]
name=elk
baseurl="ftp://192.168.1.254/elk"
enabled=
gpgcheck=
" > /etc/yum.repos.d/local.repo #elasticsearch,logstash,kibana,filebeat安装包
yum clean all
yum repolist

/root/elk2.sh

 #!/bin/bash
touch /etc/logstash/logstash.conf
echo 'input{
stdin{codec => "json"}
beats{
port =>
}
file{
path => ["/tmp/c.log"]
type => "test"
start_position => "beginning"
sincedb_path => "/var/lib/logstash/sincedb"
}
}
filter{
if [type] == "apache_log"{
grok{
match => {"message" => "%{COMBINEDAPACHELOG}"}
}}
}
output{
stdout{ codec => "rubydebug" }
if [type] == "apache_log"{
elasticsearch{
hosts => ["192.168.1.51:9200","192.168.1.52:9200"]
index => "filelog"
flush_size =>
idle_flush_time =>
}}
}
' > /etc/logstash/logstash.conf

利用ansible-playbook一键部署ELK(ElasticSearch,logstash and kibana)的更多相关文章

  1. 使用ELK(Elasticsearch + Logstash + Kibana) 搭建日志集中分析平台实践--转载

    原文地址:https://wsgzao.github.io/post/elk/ 另外可以参考:https://www.digitalocean.com/community/tutorials/how- ...

  2. (转)开源分布式搜索平台ELK(Elasticsearch+Logstash+Kibana)入门学习资源索引

    Github, Soundcloud, FogCreek, Stackoverflow, Foursquare,等公司通过elasticsearch提供搜索或大规模日志分析可视化等服务.博主近4个月搜 ...

  3. ELk(Elasticsearch, Logstash, Kibana)的安装配置

    目录 ELk(Elasticsearch, Logstash, Kibana)的安装配置 1. Elasticsearch的安装-官网 2. Kibana的安装配置-官网 3. Logstash的安装 ...

  4. 开源分布式搜索平台ELK(Elasticsearch+Logstash+Kibana)入门学习资源索引

    from:  http://www.w3c.com.cn/%E5%BC%80%E6%BA%90%E5%88%86%E5%B8%83%E5%BC%8F%E6%90%9C%E7%B4%A2%E5%B9%B ...

  5. CentOS 6.x ELK(Elasticsearch+Logstash+Kibana)

    CentOS 6.x ELK(Elasticsearch+Logstash+Kibana) 前言 Elasticsearch + Logstash + Kibana(ELK)是一套开源的日志管理方案, ...

  6. 基于CentOS6.5或Ubuntu14.04下Suricata里搭配安装 ELK (elasticsearch, logstash, kibana)(图文详解)

    前期博客 基于CentOS6.5下Suricata(一款高性能的网络IDS.IPS和网络安全监控引擎)的搭建(图文详解)(博主推荐) 基于Ubuntu14.04下Suricata(一款高性能的网络ID ...

  7. 键盘侠Linux干货| ELK(Elasticsearch + Logstash + Kibana) 搭建教程

    前言 Elasticsearch + Logstash + Kibana(ELK)是一套开源的日志管理方案,分析网站的访问情况时我们一般会借助 Google / 百度 / CNZZ 等方式嵌入 JS ...

  8. (转)How to Use Elasticsearch, Logstash, and Kibana to Manage MySQL Logs

    A comprehensive log management and analysis strategy is vital, enabling organizations to understand ...

  9. ELK (Elasticsearch+Logstash+Kibana)部署

    部署机器: 服务端:dev-server    X.X.X.X      ( logstash-1.5.4,elasticsearch-1.7.1,kibana-4.1.1 ) 客户端:dev-cli ...

随机推荐

  1. Codeforces Round #616 (Div. 2) C. Mind Control

    题目链接:http://codeforces.com/contest/1291/problem/C 思路: 我们可以很容易想到,只有前m-1个人才能影响m的选择的大小,后面的人无法影响. 如果所有人都 ...

  2. OpenCV3.2.0+VS2015开发环境配置

    vs2015安装可参考:https://www.jianshu.com/p/391e67529bd3 OpenCV3.2.0配置可参考:https://www.jianshu.com/p/026093 ...

  3. OrCAD Capture CIS绘制原理图、Allegro PCB Design XL 绘制PCB

    1.OrCAD Capture CIS绘制原理图 1.1.快捷键 (1)放置连线         w (2)放置net名称      n     放下一个时再按n可以编辑名字 (3)编辑属性      ...

  4. 创建登录WEB UI页面的Business role.

    1: Define business role 2:  business role 中可以指定 config key,  该config key可以用于UI configurationo determ ...

  5. uabntu命令行

    1.命令行    命令行中令字体大小变大:ctrl+shift+"+"    命令行中令字体大小变小:ctrl+'-'    不执行:ctrl+c    下一页:f    上一页: ...

  6. Could not connect to SMTP host: smtp.qq.com, port: 465, response: -1 SpringBoot发送邮件

    解决方案 换端口 QQ邮箱可以把端口换成587 设置属性 spring.mail.properties.mail.smtp.ssl.enable=true 原因 465端口是为SMTPS(SMTP-o ...

  7. linux - 异常:安装包冲突 conflicts with

    问题描述 解决方案 删除冲突的包 命令格式:yum -y remove 包名 yum -y remove httpd24u yum -y remove httpd24u-tools

  8. 【Linux】解决Linux服务器内存不足问题

    在Linux服务器上使用Jmeter进行压测的时候,遇到了一个错误: 根据这里的提示: Java HotSpot(TM) -Bit Server VM warning: INFO: os::commi ...

  9. JS高级---识别正则表达式是否匹配

    识别正则表达式是否匹配 console.log(/[a-zA-Z]+/.test("hello")); console.log(/./.test("除了回车换行以为的任意 ...

  10. 【Unity|C#】基础篇(7)——属性(Property)/ 索引器(Indexer)

    [学习资料] <C#图解教程>(第6章):https://www.cnblogs.com/moonache/p/7687551.html 电子书下载:https://pan.baidu.c ...