ELK日志管理搭建

目录：

一、介绍

二、安装JDK

三、安装Elasticsearch

四、安装Kibana

五、安装Nginx

六、安装Logstash

七、安装Logstash-forwarder

八、测试

系统环境：CentOS Linux release 7.4.1708 (Core)

软件版本：

elasticsearch-5.6.10

kibana-5.6.10

logstash-5.6.10

当前问题状况

1. 开发人员不能登录线上服务器查看详细日志。
2. 各个系统都有日志，日志数据分散难以查找。
3. 日志数据量大，查询速度慢，或者数据不够实时。

一、介绍

1、组成

ELK由Elasticsearch、Logstash和Kibana三部分组件组成；
Elasticsearch是个开源分布式搜索引擎，它的特点有：分布式，零配置，自动发现，索引自动分片，索引副本机制，restful风格接口，多数据源，自动搜索负载等。
Logstash是一个完全开源的工具，它可以对你的日志进行收集、分析，并将其存储供以后使用
kibana 是一个开源和免费的工具，它可以为 Logstash 和 ElasticSearch 提供的日志分析友好的 Web 界面，可以帮助您汇总、分析和搜索重要数据日志。

2、四大组件
Logstash: logstash server端用来搜集日志；
Elasticsearch: 存储各类日志；
Kibana: web化接口用作查寻和可视化日志；
Logstash Forwarder: logstash client端用来通过lumberjack 网络协议发送日志到logstash server；

3、工作流程

在需要收集日志的所有服务上部署logstash，作为logstash
agent（logstash shipper）用于监控并过滤收集日志，将过滤后的内容发送到Redis，然后logstash
indexer将日志收集在一起交给全文搜索服务ElasticSearch，可以用ElasticSearch进行自定义搜索通过Kibana
来结合自定义搜索进行页面展示。

二、安装JDK

配置阿里源：wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
yum clean all
yum makecache
Logstash的运行依赖于Java运行环境，Elasticsearch 要求至少 Java 7。
[root@controller ~]# yum install java-1.8.0-openjdk -y
[root@controller ~]# java -version
openjdk version "1.8.0_151"
OpenJDK Runtime Environment (build 1.8.0_151-b12)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)

三、安装Elasticsearch

官方下载地址：https://www.elastic.co/downloads/past-releases/elasticsearch-5-6-10

#创建用户elasticsearch-5.6.10
[root@elk-node1 local]# useradd elasticsearch
#解压
[root@elk-node1 application]# tar -xf elasticsearch-5.6.10.tar.gz -C /usr/local/
#创建软链接
[root@elk-node1 local]# ln -s elasticsearch-5.6.10/ elasticsearch
#授权
[root@elk-node1 local]# chown -R elasticsearch elasticsearch-5.6.10/
[root@elk-node1 local]# chown -R elasticsearch elasticsearch
#修改配置文件
[root@elk-node1 local]# cd elasticsearch/config/
[root@elk-node1 config]# pwd
/usr/local/elasticsearch/config
[root@elk-node1 config]# grep -Ev "^#|^$" elasticsearch.yml
cluster.name: pcidata-elk
node.name: elk-node1
bootstrap.memory_lock: true
network.host: 0.0.0.0
http.port: 9200
http.cors.enabled: true
http.cors.allow-origin: "*"
#修改打开文件句柄数，添加limits.conf内容
[root@elk-node1 config]# ulimit -SHn 65536
[root@elk-node1 config]# tail /etc/security/limits.conf
elasticsearch hard nofile 131072
elasticsearch soft nproc 2048
elasticsearch hard nproc 4096
elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited
 
#切换到elasticsearch启动服务
[root@elk-node1 config]# su - elasticsearch
Last login: Mon Aug  6 18:42:34 CST 2018 on pts/2
[elasticsearch@elk-node1 ~]$ ll
total 0
[elasticsearch@elk-node1 ~]$ cd /usr/local/elasticsearch/bin/
[elasticsearch@elk-node1 bin]$ ./elasticsearch -d
#报错则根据日志信息填坑
[elasticsearch@elk-node1 logs]$ pwd
/usr/local/elasticsearch/logs
[elasticsearch@elk-node1 logs]$ tail pcidata-elk.log
[elasticsearch@elk-node1 logs]$ ss -lntp|grep 9200
LISTEN     0      128         :::9200                    :::*                   users:(("java",pid=4189,fd=170))
#输出以下信息说明安装ok
[elasticsearch@elk-node1 logs]$ curl 'http://localhost:9200/?pretty'
{
  "name" : "elk-node1",
  "cluster_name" : "pcidata-elk",
  "cluster_uuid" : "GrfwFbeOQAmATCqZnvsq8Q",
  "version" : {
    "number" : "5.6.10",
    "build_hash" : "b727a60",
    "build_date" : "2018-06-06T15:48:34.860Z",
    "build_snapshot" : false,
    "lucene_version" : "6.6.1"
  },
  "tagline" : "You Know, for Search"
}
安装elasticsearch-head插件
1、安装git
[root@elk-node1 config]# yum install -y git
2、下载elasticsearch-head插件源码
[root@elk-node1 config]# git clone git://github.com/mobz/elasticsearch-head.git
Cloning into 'elasticsearch-head'...
remote: Counting objects: 4224, done.
remote: Total 4224 (delta 0), reused 0 (delta 0), pack-reused 4224
Receiving objects: 100% (4224/4224), 2.16 MiB | 542.00 KiB/s, done.
Resolving deltas: 100% (2328/2328), done.
3.安装node
由于head插件本质上还是一个nodejs的工程，因此需要安装node，使用npm来安装依赖的包，npm可以理解为maven
[root@elk-node1 elasticsearch]# wget https://nodejs.org/dist/v8.11.3/node-v8.11.3-linux-x64.tar.xz
4.解压node
[root@elk-node1 elasticsearch]# mv node-v8.11.3-linux-x64.tar.xz /usr/local/application/
[root@elk-node1 application]# tar -xf node-v8.11.3-linux-x64.tar.xz
[root@elk-node1 application]# ln -s node-v8.11.3-linux-x64/ node
5.配置node环境变量
[root@elk-node1 application]# vim /etc/profile
export NODE_HOME=/usr/local/application/node
export PATH=$NODE_HOME/bin:$PATH
[root@elk-node1 application]# source /etc/profile
 
6.测试node是否生效
[root@elk-node1 application]# node -v
v8.11.3
[root@elk-node1 application]# npm -v
5.6.0
初始化
[root@elk-node1 application]# npm init
This utility will walk you through creating a package.json file.
It only covers the most common items, and tries to guess sensible defaults.
 
See `npm help json` for definitive documentation on these fields
and exactly what they do.
 
Use `npm install <pkg>` afterwards to install a package and
save it as a dependency in the package.json file.
 
Press ^C at any time to quit.
package name: (application)
version: (1.0.0)
description:
entry point: (index.js)
test command:
git repository:
keywords:
author:
license: (ISC)
About to write to /usr/local/application/package.json:
 
{
  "name": "application",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "dependencies": {
    "grunt-cli": "^1.2.0"
  },
  "devDependencies": {},
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC"
}
 
Is this ok? (yes) yes
 
7.安装grunt
grunt是一个很方便的构建工具，可以进行打包压缩、测试、执行等等的工作，5.6.10里的head插件就是通过grunt启动的。因此需要安装一下
[root@elk-node1 elasticsearch-head]# pwd
/usr/local/elasticsearch/elasticsearch-head
[root@elk-node1 elasticsearch-head]# npm install -g grunt-cli
 
8.检查安装是否成功
[root@elk-node1 elasticsearch-head]# grunt -version
grunt-cli v1.2.0
 
9.修改服务器监听地址
vim elasticsearch-head/Gruntfile.js
 
connect: {
    server: {
        options: {
            port: 9100,
            hostname: '*',
            base: '.',
            keepalive: true
        }
    }
}
增加hostname属性设置为*
 
10.修改head的连接地址:
 
vim elasticsearch-head/_site/app.js
 
this.base_uri = this.config.base_uri || this.prefs.get("app-base_uri") || "http://localhost:9200";
 
把localhost修改成你es的服务器地址，如:
 
this.base_uri = this.config.base_uri || this.prefs.get("app-base_uri") || "http://172.20.10.198:9200";
 
11.运行head， 在head目录中，执行npm install
[root@elk-node1 elasticsearch-head]# npm install phantomjs-prebuilt@5.6.106 --ignore-scripts
[root@elk-node1 elasticsearch-head]# npm install
12.启动nodejs
grunt server &
 
13.访问target：9100
 
http://172.20.10.198:9100/

四、安装Kibana

官方下载地址：https://www.elastic.co/downloads/past-releases/kibana-5-6-10

#解压kibana-5.6.10
[root@elk-node1 local]# pwd
/usr/local
[root@elk-node1 local]# tar -xf kibana-5.6.10-linux-x86_64.tar.gz
#创建软链接
[root@elk-node1 local]# ln -s kibana-5.6.10-linux-x86_64/ kibana
#修改配置文件
[root@elk-node1 config]# pwd
/usr/local/kibana/config
[root@elk-node1 config]# grep -Ev "^#|^$" kibana.yml
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.url: "http://172.20.10.198:9200"
kibana.index: ".kibana"
#启动
[root@elk-node1 config]# /usr/local/kibana/bin/kibana
#输出以下信息说明安装ok
[root@elk-node1 config]# curl localhost:5601
<script>var hashRoute = '/app/kibana';
var defaultRoute = '/app/kibana';
 
var hash = window.location.hash;
if (hash.length) {
  window.location = hashRoute + hash;
} else {
  window.location = defaultRoute;
}</script>

五、安装Nginx

直接yum安装：
yum install nginx httpd-tools -y
修改配置文件：
vim /etc/nginx/nginx.conf
查看http{...}中是否有include /etc/nginx/default.d/*.conf;若无则添加
添加kibana.conf文件：
[root@controller ~]# cat /etc/nginx/conf.d/kibana.conf
server {
 listen 80;
 
 server_name example.com;
 
 location / {
 proxy_pass http://172.20.10.198:5601;
 proxy_http_version 1.1;
 proxy_set_header Upgrade $http_upgrade;
 proxy_set_header Connection 'upgrade';
 proxy_set_header Host $host;
 proxy_cache_bypass $http_upgrade;
 }
｝
启动：
[root@controller ~]# systemctl start nginx

六、安装Logstash

下载安装Logstash：
[root@controller elk]# wget https://download.elastic.co/logstash/logstash/logstash-5.6.10.tar.gz
[root@controller elk]# tar xf logstash-5.1.1.tar.gz
[root@controller elk]# cd logstash-5.6.10
验证：
[root@controller logstash-5.6.10]# ./bin/logstash  -e 'input { stdin { } } output { stdout {} }'
Settings: Default filter workers: 2
Logstash startup completed
2017-11-17T03:32:02.825Z controller
配置openssl:
[root@controller tls]# vim /etc/pki/tls/openssl.cnf
[root@controller tls]# echo " subjectAltName=IP: 172.20.10.198">>/etc/pki/tls/openssl.cnf
生成logstash-forwarder.crt文件：
[root@controller tls]# openssl req -config /etc/pki/tls/openssl.cnf -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt
配置logstash：
[root@controller logstash-5.6.10]# pwd
/usr/local/src/elk/logstash-5.6.10
[root@controller logstash-5.6.10]# mkdir conf
[root@controller logstash-5.6.10]# cd conf/
[root@controller conf]# cat simple.conf
input {
 lumberjack {
 port => 5043
 type => "logs"
 ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
 ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
 }
}
filter {
 grok {
 match => { "message" => "%{COMBINEDAPACHELOG}" }
 }
 date {
 match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
 }
}
output {
 elasticsearch { hosts => ["localhost:9200"] }
 stdout { codec => rubydebug }
}
启动：
[root@controller tls]# cd /usr/local/src/elk/logstash-5.6.10/bin/
[root@controller bin]# ./logstash -f ../conf/simple.conf

七、安装Logstash-forwarder(客户端)

安装源：
[root@controller ~]# rpm --import http://packages.elastic.co/GPG-KEY-elasticsearch
[root@controller ~]#  cat /etc/yum.repos.d/logstash-forwarder.repo
name=logstash-forwarder repository
baseurl=http://packages.elastic.co/logstashforwarder/centos
gpgcheck=1
gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch
enabled=1
[root@controller ~]# yum -y install logstash-forwarder
修改配置文件：
添加服务端ssl文件
mkdir -p /etc/pki/tls/certs
scp user@logstash_server:/etc/pki/tls/certs/logstash_forwarder.crt /etc/pki/tls/certs/
[root@controller bin]# grep -Ev '#|^$' /etc/logstash-forwarder.conf
{
  "network": {
    "servers": [ "172.20.10.198:5043" ],
    "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt",
    "timeout": 15
  },
  "files": [
    {
      "paths": [
        "/var/log/syslog",
        "/var/log/auth.log"
      ],
      "fields": { "type": "syslog" }
    }, {
    }, {
    }
  ]
}
启动：
service logstash-forwarder start

效果图：