elk 日志处理的一点思路

zjtest7-frontend:/usr/local/logstash-2.3.4/bin# ./logstash  -f ../config/logstash_agent.conf 

zjtest7-frontend:/usr/local/logstash-2.3.4/bin# ./logstash  -f ../config/logstash_indexer.conf

你可以在每一个input插件设置一个type，根据type配置不同的filter....这样能节省点资源。logstash起多了，还是不太好的。毕竟每一个都是一个jvm进程....

/*** 写入redis
[elk@zjtest7-frontend config]$ cat logstash_agent.conf
input {
        file {
                type => "zj_nginx_access"
                path => ["/usr/local/nginx/logs/zj_access.log"]
        }

        file {
                type => "wj_nginx_access"
                path => ["/usr/local/nginx/logs/wj_access.log"]
        }
}

filter {
    grok {
        match => {
            "message" => "%{IPORHOST:clientip} \[%{HTTPDATE:time}\] \"%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:http_status_code} %{NUMBER:bytes} \"(?

<http_referer>\S+)\" \"(?<http_user_agent>\S+)\" \"(?<http_x_forwarded_for>\S+)\""
        }
    }
}
output {
        redis {
                host => "192.168.32.67"
                data_type => "list"
                key => "logstash:redis"
                port=>"6379"
                password => "1234567"
        }
}

output {
if [type] == "xxx" {
 redis {xxxxx}
}
}

根除不同的type 写入到redis

/***从redis读取,发送到elasticsearch

[elk@zjtest7-frontend config]$ cat logstash_indexer.conf
input {
        redis {
                host => "192.168.32.67"
                data_type => "list"
                key => "logstash:redis"
                type => "redis-input"
                password => "1234567"
                port =>"6379"
        }
}

output {
        elasticsearch {
                hosts => "192.168.32.80:9200"
                index => "logstash-nginx-%{+YYYY.MM.dd}"
        }
		stdout {
			codec => rubydebug
		}
}

不同的 redis key 怎么发送到不同的elasticsearch 

比如 我同步应用日志到logstash 的  /log/zjzc 下面有nginx-192.168.32.1.log  nginx-192.168.32.2.log  nginx-192.168.32.3.log 

那么我    file {
                type => "zj_nginx_access"
                path => ["/log/zjzc/nginx-*.log"]
        }