ELK+Redis 解析Nginx日志

一、ELK简介

Elk是指logstash,elasticsearch,kibana三件套，我们一般使用它们做日志分析。

ELK工作原理图:

简单来讲ELK具体的工作流程就是客户端的logstash agent（shipper）从日志里取出数据推送到服务端的redis里，服务端的logstash从redis里取出数据推送到elasticsearch并产生索引，然后使用Kibana进行页面展示。

二.ELK准备环境配置

1.搭建环境(都是在Centos6.8系统下操作完成)：
  Ip地址          　　　　  节点              　　　　部署服务        
  192.168.100.10        ELK-node1                elasticsearch + logstrsh
  192.168.100.20        ELK-node2                elasticsearch + redis + kibana
  192.168.100.30       nginx-agent               nginx + logstash

2.软件下载地址及版本:

cd /usr/local/src/
wget https://download.elasticsearch.org/elasticsearch/release/org/elasticsearch/distribution/tar/elasticsearch/2.3.0/elasticsearch-2.3.0.tar.gz
wget https://download.elastic.co/logstash/logstash/logstash-1.5.4.tar.gz
curl -L -O https://download.elastic.co/kibana/kibana/kibana-4.5.1-linux-x64.tar.gz
wget http://download.redis.io/releases/redis-3.0.7.tar.gz

3.各服务器安装jdk，jdk要1.7以上的版本.

rpm -ivh jre-8u91-linux-x64.rpm

node1 node2 节点部署elasticsearch + logstrsh.

1.安装elasticsearch

cd /usr/local/src/
wget https://download.elasticsearch.org/elasticsearch/release/org/elasticsearch/distribution/tar/elasticsearch/2.3.0/elasticsearch-2.3.0.tar.gz

tar -zxvf elasticsearch-2.3.0.tar.gz -C /usr/local/
ln -s elasticsearch-2.3.0 elasticsearch
mkdir -pv /usr/local/elasticsearch/data
mkdir -pv /usr/local/elasticsearch/logs		//日志文件目录

#主节点配置如下：
grep '^[a-z]' elasticsearch.yml
cluster.name: ELK-elasticsearch
node.name: node-1
path.data: /usr/local/elasticsearch/data
path.logs: /usr/local/elasticsearch/logs
bootstrap.mlockall: true
network.host: 192.168.100.10
http.port: 9200
node.master: true
node.data: true

#集群节点参数详解:
https://my.oschina.net/liuxundemo/blog/688736?p={{page}}

启动elasticsearch:
elasticsearch只能以普通用户运行

创建elasticsearch运行用户,es只能用普通用户启动.
groupadd elasticsearch
useradd -g elasticsearch  elasticsearch
chown -R elasticsearch:elasticsearch /usr/local/elasticsearch
/usr/local/elasticsearch/bin/elasticsearch  >/dev/null 2>&1 &

服务管理的插件：
elasticsearch的插件应用
http://www.cnblogs.com/xing901022/p/5962722.html

安装集群管理插件：
/usr/local/elasticsearch/bin/plugin install mobz/elasticsearch-head
http://192.168.100.10:9200/_plugin/head/

health状况：

curl '192.168.100.10:9200/_cat/health?v'
epoch      timestamp cluster       status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1488389751 01:35:51  elasticsearch green           1         1      0   0    0    0        0             0                  -                100.0% 

curl -X GET http://192.168.100.10:9200     //获取网页内容
curl -I GET http://192.168.100.10:9200  //获取网页头部信息,200正常

从节点192.168.100.20 node2 elasticsearch配置如下:

#从节点配置如下：
cluster.name: ELK-elasticsearch
node.name: node-2
path.data: /usr/local/elasticsearch/data
path.logs: /usr/local/elasticsearch/logs
bootstrap.mlockall: true
network.host: 192.168.100.20
http.port: 9200
discovery.zen.ping.unicast.hosts: ["192.168.100.10:9300"]

LogStash部署和使用(主节点和从节点一样，只要服务启动正常即可)：

cd /usr/local/src/
wget https://download.elastic.co/logstash/logstash/logstash-1.5.4.tar.gz
[root@localhost local]# tar -zxvf logstash-1.5.4.tar.gz -C /usr/local/
[root@localhost local]#ln -s logstash-1.5.4 logstash
标准输入输出:
/usr/local/logstash/bin/logstash -e 'input { stdin{} } output { stdout{}}'
#加东西可以改变输出:
[root@localhost ~]# /usr/local/logstash/bin/logstash -e 'input { stdin{} } output { stdout{codec => rubydebug}}'
#标准输出到elasticsearch中，定义host和协议
/usr/local/logstash/bin/logstash -e 'input { stdin{} } output { elasticsearch { host => "192.168.100.10" protocol => "http"}}'
#多重输出
/usr/local/logstash/bin/logstash -e 'input { stdin{} } output { elasticsearch { host => "192.168.100.10" protocol => "http"} stdout{ codec => rubydebug }}'
启动logstrash
/usr/local/logstash/bin/logstash -f /usr/local/logstash/config/logstrash.conf
#logstash解析nginx时间字段
http://blog.csdn.net/jianblog/article/details/54585043

logstash配置文件编写:
#文件输入到文件输出.
logstash.conf
input {
  file{
      path => "/var/log/messages"
  }
}
output {
  file {
    path => "/tmp/%{+YYYY-MM-dd}.messages.gz"
    gzip => true
  }
}
#文件输入到文件和es中
input {

  file{
      path => "/var/log/messages"
  }
}
output {

  file {
    path => "/tmp/%{+YYYY-MM-dd}.messages.gz"
    gzip => true
  }
  elasticsearch {
    host => "192.168.100.10"
    protocol => "http"
    index => "sysem-messages-%{+YYYY.MM.dd}"
  }

}
启动logstrash
/usr/local/logstash/bin/logstash -f /usr/local/logstash/config/logstrash.conf

安装redis保证服务启动正常即可:

wget http://download.redis.io/releases/redis-3.0.7.tar.gz
tar -zxvf redis-3.0.7
make && make install

[root@localhost ~]#nohup redis-server 2>&1 &
[root@localhost ~]# redis-cli -h 192.168.100.20 -p 6379
192.168.21.128:6379> select 1
OK
192.168.100.20:6379[1]> keys *
1) "sys-messages"
192.168.100.20:6379[1]> LLEN sys-messages  #查看redis 列表的长度
(integer) 42120
192.168.100.20:6379[1]> LINDEX sys-messages -1  #查看列表最后的一行
"{\"message\":\"hello logstrash to redis sucess\",\"@version\":\"1\",\"@timestamp\":\"2017-02-19T02:35:44.082Z\",\"host\":\"localhost.localdomain\",\"path\":\"/var/log/messages\"}"
192.168.100.20:6379[1]>

安装kibana:

安装kibana
curl -L -O https://download.elastic.co/kibana/kibana/kibana-4.5.1-linux-x64.tar.gz
tar -zxvf kibana-4.5.1-linux-x64.tar.gz -C /usr/local
ln -s kibana-4.5.1-linux-x64 kibana
vim kibana.yml
elasticsearch_url: "http://192.168.100.10:9200"    #es主节点的ip地址
启动kibana服务
/usr/local/kibana/bin/kibana &
访问地址：
http://192.168.100.20:5601/app/kibana

在192.168.100.30 上安装nginx和logstash作为nginx日志输出端.

ELK收集Nginx日志有两种方式:

1.使用codec的json插件将日志的域进行分段，使用key-value的方式，使日志格式更清晰，易于搜索，还可以降低cpu的负载 ,更改nginx的配置文件的日志格式，使用json

2.不用修改Nginx的日志格式,通过filter过滤器来改变日志的格式.

我们这里先讲第一种方式：Nginx日志改成json输出。

logstash_nginx端的日志格式如下：

[root@logstash_nginx ~]#sed -n '15,33p' /etc/nginx/nginx.conf 

log_format logstash_json '{ "@timestamp": "$time_local", '
                         '"@fields": { '
                         '"remote_addr": "$remote_addr", '
                         '"remote_user": "$remote_user", '
                         '"body_bytes_sent": "$body_bytes_sent", '
                         '"request_time": "$request_time", '
                         '"status": "$status", '
                         '"request": "$request", '
                         '"request_method": "$request_method", '
                         '"http_referrer": "$http_referer", '
                         '"body_bytes_sent":"$body_bytes_sent", '
			 '"http_user_agent": "$http_user_agent" , '
 			 '"http_x_forwarded_for": "$http_x_forwarded_for"} }';

    access_log  /var/log/nginx/access.log  logstash_json;
启动nginx
[root@logstash_nginx ~]# /usr/sbin/nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

[root@logstash_nginx ~]# /usr/sbin/nginx

logstash配置如下：

#使用logstash收集nginx的访问日志
#vim /usr/local/logstash/logstash-agent.conf
input {
  file {
    type => "nginx-access-log"
    path => "/var/log/nginx/access.log"
    codec => "json"
  }
}

filter {

}
output {
  redis {
    data_type => "list"
    key => "nginx-access-log"
    host => "192.168.100.20"
    port => "6379"
    db => "1"
 }

}

#logstash indexer端文件,从redis里面读取数据然后在es中

vim /usr/local/logstash/logstash_indexer.conf

input{
  redis{
    data_type => "list"
    key => "nginx-access-log"
    host => "redis-ip"
    port => "6379"
    db => "1"
  }
}

output {

  elasticsearch {
    host => "192.168.100.10"
    protocol => "http"
    index => "logstash-nginx-access-log-%{+YYYY.MM.dd}"
  }
}

确定没有问题后，重新启动logstash

/usr/local/logstash/bin/logstash -f /usr/local/logstash/bin/logstash-agent.conf
/usr/local/logstash/bin/logstash -f /usr/local/logstash/bin/logstash_indexer.conf

我们可以访问http://192.168.100.10:9200/_plugin/head/,如果配置无误可以看到nginx-access-log这个索引，说明logstash已经将日志发送到es端了.

然后在kibana里面创建你的索引即可.

这样就可以收集到日志了。

elasticsearch的插件应用:

http://www.cnblogs.com/xing901022/p/5962722.html

使用Kibana 分析Nginx 日志并在 Dashboard上展示

http://www.cnblogs.com/hanyifeng/p/5860731.html

参考文档：

　　　　kibana画图参考文档：

　　　　　　　　　　http://blog.csdn.net/ming_311/article/details/50619859

　　　　　　　　　　http://www.cnblogs.com/hanyifeng/p/5857875.html
　　　　　　　　　　http://blog.oldboyedu.com/elk/
　　　　　　　　　　http://www.cnblogs.com/galengao/p/5780588.html
　　　　　　　　　　http://blog.csdn.net/wanglipo/article/details/50739820
　　　　　　　　　　http://www.jianshu.com/p/66e03eb6d95a
　　　　　　　　　　http://www.cnblogs.com/skyblue/p/5504595.html

Kibana反向代理配置:

1.Nginx安装(略):

2.Nginx配置 (kibana_proxy.conf)

#The default server

upstream kibana_proxy {
      server kibana-ip-address:5601; 

        }

server {
   listen    80;
   server_name elk.xxx.com; 

   location / {
      index index.html index.htm;
      auth_basic "welcome to kibana";
      auth_basic_user_file /etc/nginx/passwd.db;
      proxy_pass http://kibana_proxy;
    }
    location /status {
        stub_status on;
        access_log /var/log/nginx/kibana_status.log;
        auth_basic "NginxStatus";

 }

}

#生成认证登录的用户名和密码：

[root@elk-node conf.d]# htpasswd -c /etc/nginx/passwd.db admin
New password:
Re-type new password:
Adding password for user admin

# chmod 400 /etc/nginx/passwd.db      //修改网站认证数据库权限
# chown nginx. /etc/nginx/passwd.db   //修改网站认证数据库属主和属组
# cat /etc/nginx/passwd.db            //可以看到通过htpasswd生成的密码为加密格式
admin:8eZAz7BqcrXmY