ELK之收集Java日志、通过TCP收集日志

1.Java日志收集

使用codec的multiline插件实现多行匹配,这是一个可以将多行进行合并的插件,而且可以使用what指定将匹配到的行与前面的行合并还是和后面的行合并.

语法示例:
input {
  stdin {
    codec => multiline {    #使用multiline插件
      pattern => "pattern, a regexp"   #正则匹配
      negate => "true" or "false"      #匹配是否成功
      what => "previous" or "next"     #和上面的还是和下面的内容合并
    }
  }
}

命令行测试输入输出

logstash -e 'input { stdin {codec => multiline { pattern => "^\[" negate => "true" what => "previous"} }} output { stdout {codec => rubydebug}}'

2.配置logstash

elk集群日志上都是以"["开头并且每一个信息都是如此

tailf /data/logs/elk-cluster.log

vim /etc/logstash/conf.d/java.conf
input {
  file{
    path => "/data/logs/elk-cluster.log"
    type => "elasticsearch-java-log"
    start_position => "beginning"
    stat_interval => "2"
    codec => multiline {
            pattern => "^\["
            negate => "true"
            what => "previous"
        }
    }
}
 
output {
  if [type] == "elasticsearch-java-log" {
    elasticsearch {
      hosts => ["10.0.0.22:9200"]
      index => "elasticsearch-jva-log-%{+YYYY.MM.dd}"
    }
  }
}
 
logstash -f /etc/logstash/conf.d/java.conf -t
systemctl restart logstash

es插件中查看

3.通过TCP收集日志

TCP收集日志使用场景

有一台服务器A只需要收集一个日志,那么我们就可以不需要在这服务器上安装logstash,我们通过在其他logstash上启用tcp模块,监听某个端口,然后在服务器A上把日志通过nc发送到logstash上即可.

a.elk2上配置测试TCP模块

cat /etc/logstash/conf.d/tcp.conf
input {
  tcp{
    port => "5600"     #监听5600端口
    mode => "server"   #模式为server
    type => "tcplog"   #类型为tcplog
    }
}
 
output {
  stdout {
    codec => rubydebug
}
}
 
# elk1节点上安装nc命令,并发送日志到elk2
yum -y install nc
echo "hello world" | nc 10.0.0.33 5600
 
# elk2终端上查看日志输出信息:
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/tcp.conf
{
    "@timestamp" => 2019-02-07T00:59:49.356Z,
          "port" => 57902,
      "@version" => "1",
          "host" => "linux-elk1",
      "@metdata" => {
        "ip_address" => "10.0.0.22"
    },
       "message" => "hello world",
          "type" => "tcplog"
}
# 可以看到linux-elk2上有监听5600端口
netstat -tunlp |grep 5600
 
# 还可以将某个文件发送到nc
nc 10.0.0.33 5600 < /etc/passwd
 
# 也可以通过这种方式伪设备的方式发送日志
echo "222" > /dev/tcp/10.0.0.33/5600

b.配置logstash,通过TCP收集数据,输出到elasticsearch

vim /etc/logstash/conf.d/tcp.conf
input {
  tcp{
    port => "5600"
    mode => "server"
    type => "tcplog"
        }
}
 
output {
  elasticsearch {
    hosts => ["10.0.0.33:9200"]
    index => "tcp-test-%{+YYYY.MM.dd}"
    }
}
systemctl restart logstash

在elk1上向elk2发送数据:nc 10.0.0.33 5600 < /etc/passwd

java日志收集:http://blog.51cto.com/jinlong/2055424

通过TCP收集日志:http://blog.51cto.com/jinlong/2056521