ELK (Elasticsearch+Logstash+Kibana)部署
部署机器:
服务端:
dev-server X.X.X.X ( logstash-1.5.4,elasticsearch-1.7.1,kibana-4.1.1 )
客户端:
dev-client X.X.X.X (logstash-forwarder-0.4.0-1)
需求:将客户端访问日志(nginx log)在ELK中展示。
安装ELK:
服务端:
设置FQDN(创建SSL证书的时候需要配置FQDN):
[root@dev-client ~]# hostname
dev-client
[root@dev-client ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4
X.X.X.X elk.test.com elk
安装Java 1.8:
[root@dev-server elk]# cat /etc/redhat-release
CentOS release 6.6 (Final)
[root@dev-server elk]# yum install java-1.8.0-openjdk.x86_64
[root@dev-server elk]# java -version
openjdk version "1.8.0_65"
OpenJDK Runtime Environment (build 1.8.0_65-b17)
OpenJDK 64-Bit Server VM (build 25.65-b01, mixed mode)
安装 elasticsearch-1.7.1:
#下载安装
[root@dev-server elk]# wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.1.noarch.rpm
#启动相关服务
[root@dev-server elk]# /etc/init.d/elasticsearch start
[root@dev-server elk]# /etc/init.d/elasticsearch stop
#查看elasticsearch配置文件
[root@dev-server elk]# rpm -qc elasticsearch
/etc/elasticsearch/elasticsearch.yml
/etc/elasticsearch/logging.yml
/etc/init.d/elasticsearch
/etc/sysconfig/elasticsearch
/usr/lib/sysctl.d/elasticsearch.conf
/usr/lib/systemd/system/elasticsearch.service
/usr/lib/tmpfiles.d/elasticsearch.conf
#查看端口使用情况
[root@dev-server elk]# netstat -tunlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 :::9300 :::* LISTEN 14585/java
tcp 0 0 :::9200 :::* LISTEN 14585/java
安装Kibana 4.1.1:
#下载tar包
[root@dev-server elk]# wget https://download.elastic.co/kibana/kibana/kibana-4.1.1-linux-x64.tar.gz
#解压
[root@dev-server elk]# pwd
/data1/elk
[root@dev-server elk]# tar xf kibana-4.1.1-linux-x64.tar.gz
[root@dev-server elk]# ln -s /data1/elk/kibana-4.1.1-linux-x64 kibana
#创建kibana服务
[root@dev-server elk]# cat /etc/init.d/kibana
#!/bin/bash
### BEGIN INIT INFO
# Provides: kibana
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Runs kibana daemon
# Description: Runs the kibana daemon as a non-root user
### END INIT INFO # Process name
NAME=kibana
DESC="Kibana4"
PROG="/etc/init.d/kibana" # Configure location of Kibana bin
KIBANA_BIN=/data1/elk/kibana/bin #注意路径 # PID Info
PID_FOLDER=/var/run/kibana/
PID_FILE=/var/run/kibana/$NAME.pid
LOCK_FILE=/var/lock/subsys/$NAME
PATH=/bin:/usr/bin:/sbin:/usr/sbin:$KIBANA_BIN
DAEMON=$KIBANA_BIN/$NAME # Configure User to run daemon process
DAEMON_USER=root
# Configure logging location
KIBANA_LOG=/var/log/kibana.log # Begin Script
RETVAL=0 if [ `id -u` -ne 0 ]; then
echo "You need root privileges to run this script"
exit 1
fi # Function library
. /etc/init.d/functions start() {
echo -n "Starting $DESC : " pid=`pidofproc -p $PID_FILE kibana`
if [ -n "$pid" ] ; then
echo "Already running."
exit 0
else
# Start Daemon
if [ ! -d "$PID_FOLDER" ] ; then
mkdir $PID_FOLDER
fi
daemon --user=$DAEMON_USER --pidfile=$PID_FILE $DAEMON 1>"$KIBANA_LOG" 2>&1 &
sleep 2
pidofproc node > $PID_FILE
RETVAL=$?
[[ $? -eq 0 ]] && success || failure
echo
[ $RETVAL = 0 ] && touch $LOCK_FILE
return $RETVAL
fi
} reload()
{
echo "Reload command is not implemented for this service."
return $RETVAL
} stop() {
echo -n "Stopping $DESC : "
killproc -p $PID_FILE $DAEMON
RETVAL=$?
echo
[ $RETVAL = 0 ] && rm -f $PID_FILE $LOCK_FILE
} case "$1" in
start)
start
;;
stop)
stop
;;
status)
status -p $PID_FILE $DAEMON
RETVAL=$?
;;
restart)
stop
start
;;
reload)
reload
;;
*)
# Invalid Arguments, print the following message.
echo "Usage: $0 {start|stop|status|restart}" >&2
exit 2
;;
esac
#修改启动权限
[root@dev-server elk]# chmod +x /etc/init.d/kibana
#启动kibana服务
[root@dev-server elk]# /etc/init.d/kibana start
[root@dev-server elk]# /etc/init.d/kibana status
#查看端口
[root@dev-server elk]# netstat -tunlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:5601(默认,可以改为80) 0.0.0.0: LISTEN 15128/node
安装logstash 1.5.4
#下载安装
[root@dev-server elk]# wget https://download.elastic.co/logstash/logstash/packages/centos/logstash-1.5.4-1.noarch.rpm
[root@dev-server elk]# yum localinstall logstash-1.5.4-1.noarch.rpm
#设置ssl,之前设置的FQDN是elk.test.com
[root@dev-server tls]# pwd
/etc/pki/tls
[root@dev-server tls]# openssl req -subj '/CN=elk.test.com/' -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt [root@dev-server certs]# pwd
/etc/pki/tls/certs
[root@dev-server certs]# ls -l logstash-forwarder.crt
-rw-r--r-- 1 root root 1103 Nov 23 22:46 logstash-forwarder.crt
#创建一个01-logstash-initial.conf文件
[root@dev-server conf.d]# cat 01-logstash-initial.conf
input {
lumberjack {
port => 5000
type => "logs"
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
} filter {
if [type] == "nginx" {
grok {
match => { "message" => "%{NGINXACCESS}" }
}
}
} output {
elasticsearch {
index => "zabbix-access-%{+YYYY.MM.dd}"
host => localhost
}
stdout { codec => rubydebug }
}
#启动logstash服务
[root@dev-server conf.d]# /etc/init.d/logstash start
[root@dev-server conf.d]# /etc/init.d/logstash stop
#查看端口
[root@dev-server conf.d]# netstat -tunlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 :::9301 :::* LISTEN 4381/java
tcp 0 0 :::5000 :::* LISTEN 4381/java
#启动客户端logstash(后面会讲解客户端)
[root@dev-client ~]# /etc/init.d/logstash-forwarder start
[root@dev-client ~]# /etc/init.d/logstash-forwarder status
#访问kibana
http://XXXXX.XXX
#增加节点和客户端配置一样,注意同步证书
/etc/pki/tls/certs/logstash-forwarder.crt
客户端安装logstash-forwarder :
#安装客户端
[root@dev-client opt]# wget https://download.elastic.co/logstash-forwarder/binaries/logstash-forwarder-0.4.0-1.x86_64.rpm
[root@dev-client opt]# yum localinstall logstash-forwarder-0.4.0-1.x86_64.rpm
#查看配置文件
[root@dev-client opt]# rpm -qc logstash-forwarder
/etc/logstash-forwarder.conf
#备份配置文件
[root@dev-client opt]# cp /etc/logstash-forwarder.conf /etc/logstash-forwarder.conf.save
#编辑配置文件
[root@dev-client opt]# cat /etc/logstash-forwarder.conf
{
"network": {
"servers": [ "elk.test.com:5000" ], "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt", "timeout": 15
}, "files": [
{
"paths": [
"/var/log/nginx/log/zabbix.access.log"
],
"fields": { "type": "nginx" }
}
]
}
配置日志规则
#服务端增加patterns
[root@dev-server ]# mkdir /opt/logstash/patterns/ [root@dev-server patterns]# cat nginx
NGUSERNAME [a-zA-Z\.\@\-\+_%]+
NGUSER %{NGUSERNAME}
NGINXACCESS %{IPORHOST:remote_addr} - - \[%{HTTPDATE:time_local}\] "%{WORD:method} %{URIPATH:path}(?:%{URIPARAM:param})? HTTP/%{NUMBER:httpversion}" %{INT:status} %{INT:body_bytes_sent} %{QS:http_referer} %{QS:http_user_agent}
#修改logstash权限
[root@dev-server patterns]# chown -R logstash:logstash /opt/logstash/patterns/
#修改服务端配置
[root@dev-server patterns]# cat /etc/logstash/conf.d/01-logstash-initial.conf
input {
lumberjack {
port => 5000
type => "logs"
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
} filter {
if [type] == "nginx" {
grok {
match => { "message" => "%{NGINXACCESS}" }
}
}
} output {
elasticsearch {
index => "zabbix-access-%{+YYYY.MM.dd}"
host => localhost
}
stdout { codec => rubydebug }
}
修改Kibana端口
[root@dev-server config]# pwd
/data1/elk/kibana/config
[root@dev-server config]# cat kibana.yml | grep port
# Kibana is served by a back end server. This controls which port to use.
#port: 5601
port: 80
访问Kibana
http://XXX.XXX
ELK (Elasticsearch+Logstash+Kibana)部署的更多相关文章
- elasticsearch+logstash+kibana部署
这篇博客讲的是elasticsearch+logstash+kibana部署的方法. 内容大纲: 1.elasticsearch+logstash+kibana部署 2.收集Tomcat日志 3.收集 ...
- 使用ELK(Elasticsearch + Logstash + Kibana) 搭建日志集中分析平台实践--转载
原文地址:https://wsgzao.github.io/post/elk/ 另外可以参考:https://www.digitalocean.com/community/tutorials/how- ...
- ELk(Elasticsearch, Logstash, Kibana)的安装配置
目录 ELk(Elasticsearch, Logstash, Kibana)的安装配置 1. Elasticsearch的安装-官网 2. Kibana的安装配置-官网 3. Logstash的安装 ...
- CentOS 6.x ELK(Elasticsearch+Logstash+Kibana)
CentOS 6.x ELK(Elasticsearch+Logstash+Kibana) 前言 Elasticsearch + Logstash + Kibana(ELK)是一套开源的日志管理方案, ...
- 基于CentOS6.5或Ubuntu14.04下Suricata里搭配安装 ELK (elasticsearch, logstash, kibana)(图文详解)
前期博客 基于CentOS6.5下Suricata(一款高性能的网络IDS.IPS和网络安全监控引擎)的搭建(图文详解)(博主推荐) 基于Ubuntu14.04下Suricata(一款高性能的网络ID ...
- 键盘侠Linux干货| ELK(Elasticsearch + Logstash + Kibana) 搭建教程
前言 Elasticsearch + Logstash + Kibana(ELK)是一套开源的日志管理方案,分析网站的访问情况时我们一般会借助 Google / 百度 / CNZZ 等方式嵌入 JS ...
- 【转】ELK(ElasticSearch, Logstash, Kibana)搭建实时日志分析平台
[转自]https://my.oschina.net/itblog/blog/547250 摘要: 前段时间研究的Log4j+Kafka中,有人建议把Kafka收集到的日志存放于ES(ElasticS ...
- ELK(ElasticSearch+Logstash+ Kibana)搭建实时日志分析平台
一.简介 ELK 由三部分组成elasticsearch.logstash.kibana,elasticsearch是一个近似实时的搜索平台,它让你以前所未有的速度处理大数据成为可能. Elastic ...
- 【Big Data - ELK】ELK(ElasticSearch, Logstash, Kibana)搭建实时日志分析平台
摘要: 前段时间研究的Log4j+Kafka中,有人建议把Kafka收集到的日志存放于ES(ElasticSearch,一款基于Apache Lucene的开源分布式搜索引擎)中便于查找和分析,在研究 ...
随机推荐
- [Redis] redis在centos下安装测试
下载软件,使用命令wget xxx,参数:url 例如: wget http://download.redis.io/releases/redis-3.0.0.tar.gz 解压缩,使用命令tar,参 ...
- 【Dubbo&&Zookeeper】1、Dubbo与Zookeeper、SpringMVC整合和使用(负载均衡、容错)
转自:http://blog.csdn.net/congcong68/article/details/41113239 互联网的发展,网站应用的规模不断扩大,常规的垂直应用架构已无法应对,分布式服务架 ...
- intellij idea 2016.3.5 控制台取消行数限制
有时候我们要输出大量的信息放到控制台显示,但是多了之后就出现最上面的信息被覆盖删除, 因此就需要设置控制台的显示行数,但在idea7之后的版本中,取消了对控制台行数设置选项, 只能通过更改配置文件进行 ...
- JS中undefined和null的区别,以及出现原因
区别:null是一个表示无的对象,转换为数值为0: undefined表示一个无的原始值,转化为数值为NAN(与任何数字相加也为NAN) undefined出现原因:(口诀:一变量二函数一对象) 1. ...
- 盲刷bios
本帖最后由 evayh 于 2011-12-17 13:09 编辑 先看看是否是insyde的bios,如果是的话,可以救回来 insyde BIOS在损坏时,会自动进入CRISIS MODE试图刷回 ...
- TNS-12549问题分析及解决办法
该服务器启动监听时候报错因为最后一句是Linux Error:No space left on device 因为是LINUX Error,所以可以到/var/log/messages里查看具体报错信 ...
- Nginx的介绍
1.什么是NGINX: Nginx是一个开源的,支持高性能的.支持高并发的web服务软件和代理服务软件,它是由俄罗斯人lgor Sysoev开发的 Nginx具有高并发(特别是对于静态资源)占用系统资 ...
- Nginx的防爬虫优化
我们可以根据客户端的 user-agents 首部字段来阻止指定的爬虫爬取我们的网站: 虚拟主机配置如下:(红色标记为添加或者修改内容) [root@Nginx www_date]# cat bria ...
- React Native八大Demo
参考资料:http://www.cnblogs.com/shaoting/p/7148240.html 下一个项目公司也打算使用react native.大致看了下原型设计,写几个小demo先试试水. ...
- 使用Webpack对Css文件压缩处理的思考
问题的起因: 使用 bulma.css ,通过webpack打包后样式出错,查看压缩代码,发现代码从css的 long hand 属性被压缩为 short hand(PS: 什么是long hand ...