debian安装filebeat5.5收集nginx日志

先贴一下我的BELK架构

1、Download and install the Public Signing Key:

# wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add -

2、You may need to install the apt-transport-https package on Debian before proceeding:

# aptitude install -y apt-transport-https

3、Save the repository definition to /etc/apt/sources.list.d/elastic-5.x.list:

# echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | tee -a /etc/apt/sources.list.d/elastic-5.x.list

4、Run aptitude update, and the repository is ready for use. For example, you can install Filebeat by running:

# aptitude update
# aptitude install -y filebeat

5、To configure the Beat to start automatically during boot, run:

# update-rc.d filebeat defaults 95 10

6、为nginx添加json日志格式

# vim /usr/local/nginx/conf/nginx.conf
log_format json '{"@timestamp":"$time_iso8601",'
                  '"host":"$server_addr",'
                  '"clientip":"$remote_addr",'
                  '"remote_user":"$remote_user",'
                  '"request":"$request",'
                  '"http_user_agent":"$http_user_agent",'
                  '"size":$body_bytes_sent,'
                  '"responsetime":$request_time,'
                  '"upstreamtime":"$upstream_response_time",'
                  '"upstreamhost":"$upstream_addr",'
                  '"http_host":"$host",'
                  '"url":"$uri",'
                  '"domain":"$host",'
                  '"xff":"$http_x_forwarded_for",'
                  '"referer":"$http_referer",'
                  '"status":"$status"}';
access_log /var/log/nginx/access.log json;

7、重载nginx服务

# /usr/local/nginx/sbin/nginx -t
# /usr/local/nginx/sbin/nginx -s reload

8、修改filebeat的配置文件。如果同一台机器上要收集多个日志文件，而且每个日志要输出到不同的索引，那么可以把每个prospector单独定义一个document_type，然后在logstash上通过 if 判断输出到不同的索引。

# vim /etc/filebeat/filebeat.yml
filebeat.prospectors:
- input_type: log
    paths:
        - /usr/local/nginx/logs/zixun.oupeng.com.access.log
    document_type: zixun-nginx-access 
- input_type: log
  paths:
    - /usr/local/nginx/logs/water.oupeng.com.access.log
  document_type: water-nginx-access
- input_type: log
  paths:
    - /usr/local/nginx/logs/nav.oupeng.com.access.log
  document_type: nav-nginx-access
- input_type: log
  paths:
    - /usr/local/nginx/logs/wood.oupeng.com.access.log
  document_type: wood-nginx-access
- input_type: log
  paths:
    - /usr/local/nginx/logs/redir.oupeng.com.access.log
  document_type: redir-nginx-access
- input_type: log
  paths:
    - /usr/local/nginx/logs/default.access.log
  document_type: default-nginx-access
- input_type: log
  paths:
    - /usr/local/nginx/logs/kibana.oupeng.com.access.log
  document_type: kibana-nginx-access 
output.logstash:
  hosts: ["192.168.3.56:5044","192.168.3.49:5044","192.168.3.57:5044"]
  loadbalance: true

9、启动filebeat服务

启动之前可以测试一下配置是否正确

# filebeat.sh --help
-configtest：Test configuration and exit.
-e：Log to stderr and disable syslog/file output
# filebeat.sh -configtest -e
2017/07/09 17:36:59.623072 beat.go:285: INFO Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2017/07/09 17:36:59.623123 beat.go:186: INFO Setup Beat: filebeat; Version: 5.5.0
2017/07/09 17:36:59.623211 logstash.go:90: INFO Max Retries set to: 3
2017/07/09 17:36:59.623218 metrics.go:23: INFO Metrics logging every 30s
2017/07/09 17:36:59.623493 outputs.go:108: INFO Activated logstash as output plugin.
2017/07/09 17:36:59.623683 publish.go:295: INFO Publisher name: uy05-09
2017/07/09 17:36:59.625146 async.go:63: INFO Flush Interval set to: 1s
2017/07/09 17:36:59.625176 async.go:64: INFO Max Bulk Size set to: 2048
Config OK

# /etc/init.d/filebeat start

10、编写logstash pipeline配置文件。这里通过 if 判断将不同type的日志输出到不同的索引。

# vim /etc/logstash/conf.d/nginx.conf
input {
  beats {
    port => 5044
    codec => "json"
  }
}
output {
    if [type] == "zixun-nginx-access" {
    elasticsearch {
        hosts => ["192.168.3.56:9200","192.168.3.49:9200","192.168.3.57:9200"]
        index => "zixun-nginx-access-%{+YYYY.MM.dd}"
        document_type => "%{[@metadata][type]}"
        template_overwrite => true
        user => elastic
        password => Monkey
    }}
    if [type] == "water-nginx-access" {
    elasticsearch {
        hosts => ["192.168.3.56:9200","192.168.3.49:9200","192.168.3.57:9200"]
        index => "water-nginx-access-%{+YYYY.MM.dd}"
        document_type => "%{[@metadata][type]}"
        template_overwrite => true
        user => elastic
        password => Monkey
    }}
    if [type] == "nav-nginx-access" {
    elasticsearch {
        hosts => ["192.168.3.56:9200","192.168.3.49:9200","192.168.3.57:9200"]
        index => "nav-nginx-access-%{+YYYY.MM.dd}"
        document_type => "%{[@metadata][type]}"
        template_overwrite => true
        user => elastic
        password => Monkey
    }}
    if [type] == "wood-nginx-access" {
    elasticsearch {
        hosts => ["192.168.3.56:9200","192.168.3.49:9200","192.168.3.57:9200"]
        index => "wood-nginx-access-%{+YYYY.MM.dd}"
        document_type => "%{[@metadata][type]}"
        template_overwrite => true
        user => elastic
        password => Monkey
    }}
    if [type] == "redir-nginx-access" {
    elasticsearch {
        hosts => ["192.168.3.56:9200","192.168.3.49:9200","192.168.3.57:9200"]
        index => "redir-nginx-access-%{+YYYY.MM.dd}"
        document_type => "%{[@metadata][type]}"
        template_overwrite => true
        user => elastic
        password => Monkey
    }}
    if [type] == "default-nginx-access" {
    elasticsearch {
        hosts => ["192.168.3.56:9200","192.168.3.49:9200","192.168.3.57:9200"]
        index => "default-nginx-access-%{+YYYY.MM.dd}"
        document_type => "%{[@metadata][type]}"
        template_overwrite => true
        user => elastic
        password => Monkey
    }}
    if [type] == "zx-opgirl-cn-nginx-access" {
    elasticsearch {
        hosts => ["192.168.3.56:9200","192.168.3.49:9200","192.168.3.57:9200"]
        index => "zx-opgirl-cn-nginx-access-%{+YYYY.MM.dd}"
        document_type => "%{[@metadata][type]}"
        template_overwrite => true
        user => elastic
        password => Monkey
    }}
    if [type] == "www-oupeng-nginx-access" {
    elasticsearch {
        hosts => ["192.168.3.56:9200","192.168.3.49:9200","192.168.3.57:9200"]
        index => "www-oupeng-nginx-access-%{+YYYY.MM.dd}"
        document_type => "%{[@metadata][type]}"
        template_overwrite => true
        user => elastic
        password => Monkey
    }}
    if [type] == "kibana-nginx-access" {
    elasticsearch {
        hosts => ["192.168.3.56:9200","192.168.3.49:9200","192.168.3.57:9200"]
        index => "kibana-nginx-access-%{+YYYY.MM.dd}"
        document_type => "%{[@metadata][type]}"
        template_overwrite => true
        user => elastic
        password => Monkey
    }}
}

11、启动logstash

# nohup logstash -f /etc/logstash/conf.d/nginx.conf &

12、在kiaban上添加索引并绘图，绘图方法参考上一篇