logstash之multiline插件,匹配多行日志
在外理日志时,除了访问日志外,还要处理运行时日志,该日志大都用程序写的,比如log4j。运行时日志跟访问日志最大的不同是,运行时日志是多行,也就是说,连续的多行才能表达一个意思。
在filter中,加入以下代码:
filter {
multiline { }
}
如果能按多行处理,那么把他们拆分到字段就很容易了。
字段属性:
对于multiline插件来说,有三个设置比较重要:negate , pattern 和 what
negate:类型是boolean默认为false
pattern:
必须设置,并且没有默认值,类型为string,要匹配下则表达式
what:
必须设置,并且没有默认值,可以为previous(之前的)或next
下面看看这个例子:
# cat logstash_multiline_shipper.conf
input {
file {
path => "/apps/logstash/conf/test/c.out"
type => "runtimelog"
codec => multiline {
pattern => "^\["
negate => true
what => "previous"
}
start_position => "beginning"
sincedb_path => "/apps/logstash/logs/sincedb-access"
ignore_older =>
}
}
output {
stdout{
codec => rubydebug
}
}
说明:区配以"["开头的行,如果不是,那肯定是属于前一行的
测试数据如下:
[-- :: DEBUG] model.MappingNode:- ['/store/shopclass'] matched over. [-- :: DEBUG] impl.JdbcEntityInserter:- from product_category product_category where product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null order by product_category.ORDERS asc [-- :: DEBUG] model.MappingNode:- ['/store/shopclass'] matched over. [-- :: DEBUG] model.MappingNode:- ['/store/shopclass'] matched over. [-- :: DEBUG] impl.JdbcEntityInserter:- from product_category product_category where product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null order by product_category.ORDERS desc [-- :: DEBUG] impl.JdbcEntityInserter:- from product_category product_category where product_category.PARENT_ID is null and product_category.STATUS = ? and product_category.DEALER_ID is null order by product_category.ORDERS asc [-- :: DEBUG] model.MappingNode:- ['/store/shopclass'] matched over.
启动logstash:
# ./../bin/logstash -f logstash_multiline_shipper.conf
Sending Logstash's logs to /apps/logstash/logs which is now configured via log4j2.properties
[--09T15::,][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>, "pipeline.batch.size"=>, "pipeline.batch.delay"=>, "pipeline.max_inflight"=>}
[--09T15::,][INFO ][logstash.pipeline ] Pipeline main started
[--09T15::,][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>}
加入测试数据到被监控的log后,查看输出:
# ./../bin/logstash -f logstash_multiline_shipper.conf
Sending Logstash's logs to /apps/logstash/logs which is now configured via log4j2.properties
[--09T15::,][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>, "pipeline.batch.size"=>, "pipeline.batch.delay"=>, "pipeline.max_inflight"=>}
[--09T15::,][INFO ][logstash.pipeline ] Pipeline main started
[--09T15::,][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>}
{
"path" => "/apps/logstash/conf/test/c.out",
"@timestamp" => --09T07::.403Z,
"@version" => "",
"host" => "ofs1",
"message" => "# ./../bin/logstash -f logstash_multiline_shipper.conf \nSending Logstash's logs to /apps/logstash/logs which is now configured via log4j2.properties",
"type" => "runtimelog",
"tags" => [
[] "multiline"
]
}
{
"path" => "/apps/logstash/conf/test/c.out",
"@timestamp" => --09T07::.409Z,
"@version" => "",
"host" => "ofs1",
"message" => "[2016-12-09T15:16:59,173][INFO ][logstash.pipeline ] Starting pipeline {\"id\"=>\"main\", \"pipeline.workers\"=>4, \"pipeline.batch.size\"=>125, \"pipeline.batch.delay\"=>5, \"pipeline.max_inflight\"=>500}",
"type" => "runtimelog",
"tags" => []
}
{
"path" => "/apps/logstash/conf/test/c.out",
"@timestamp" => --09T07::.410Z,
"@version" => "",
"host" => "ofs1",
"message" => "[2016-12-09T15:16:59,192][INFO ][logstash.pipeline ] Pipeline main started",
"type" => "runtimelog",
"tags" => []
}
logstash之multiline插件,匹配多行日志的更多相关文章
- Logstash——multiline 插件,匹配多行日志
本文内容 测试数据 字段属性 按多行解析运行时日志 把多行日志解析到字段 参考资料 在处理日志时,除了访问日志外,还要处理运行时日志,该日志大都用程序写的,比如 log4j.运行时日志跟访问日志最大的 ...
- Logstash——multiline 插件,匹配多行日志
本文内容 测试数据 字段属性 按多行解析运行时日志 把多行日志解析到字段 参考资料 在处理日志时,除了访问日志外,还要处理运行时日志,该日志大都用程序写的,比如 log4j.运行时日志跟访问日志最大的 ...
- Logstash-安装logstash-filter-multiline插件(解决logstash匹配多行日志)
ELK-logstash在搬运日志的时候会出现多行日志,普通的搬运会造成保存到ES中日志一条一条的保存,很丑,而且不方便读取,logstash-filter-multiline可以解决该问题. 接下来 ...
- logstash匹配多行日志
在工作中,遇到一个问题就是日志的处理,首选的方案就是ELFK(filebeat+logstash+es+kibana) 因为之前使用过logstash采集日志的时候,非常的消耗系统的资源,所以这里我选 ...
- Logstash之multiline 插件
input { stdin { codec => multiline { pattern => "^\[" negate => true what => & ...
- logstash 中multiline插件的用法
input { stdin { codec =>multiline { charset=>... #可选 字符编码 max_bytes=>... #可选 bytes类型 设置最大的字 ...
- Python正则处理多行日志一例
正则表达式基础知识请参阅<正则表达式基础知识>,本文使用正则表达式来匹配多行日志并从中解析出相应的信息. 假设现在有这样的SQL日志: SELECT * FROM open_app WHE ...
- Python正则处理多行日志一例(可配置化)
正则表达式基础知识请参阅<正则表达式基础知识>,本文使用正则表达式来匹配多行日志并从中解析出相应的信息. 假设现在有这样的SQL日志: SELECT * FROM open_app WHE ...
- 写给大忙人的ELK最新版6.2.4学习笔记-Logstash和Filebeat解析(java异常堆栈下多行日志配置支持)
接前一篇CentOS 7下最新版(6.2.4)ELK+Filebeat+Log4j日志集成环境搭建完整指南,继续对ELK. logstash官方最新文档https://www.elastic.co/g ...
随机推荐
- sublime text3 package control 安装
1.安装 package control 按快捷键ctrl+` 或使用菜单 View->Show Console 打开控制台,然后粘贴如下安装代码 import urllib.request,o ...
- 分享一个快速设置背景的js 自动获取背景图的长宽
我来分享一个快速设置背景的js (需要jq支持!) 快速切图铺页面用---就是不需要手动输入背景图的长宽 自动获取背景图的长宽 : <div class="wrap"> ...
- linux下编译安装boost库
linux下编译安装boost库 linux下编译安装boost库 1.下载并解压boost 1.58 源代码 下载 解压 2.运行bootstrap.sh 3.使用b2进行构建 构建成功的提示 4. ...
- 5.2---小数的二进制表示(CC150)
public static String printBin(double num) { StringBuffer str = new StringBuffer(); str.append('0'); ...
- git 使用入门篇
最近准备给同事培训git,发现了一个不错的资源,在这里:http://www.gitguys.com/topics/creating-a-shared-repository-users-sharing ...
- Quartz表达式详解(转载)
1. CronTrigger时间格式配置说明 CronTrigger配置格式: 格式: [秒] [分] [小时] [日] [月] [周] [年] 序号 说明 是否必填 允许填写的值 允许的通配符 ...
- uniq命令注意事项,检查重复行的时候,只会检查相邻的行。
今天在使用uniq命令统计数量时,uniq -c总是得不到想要的效果,相同的行没有合并,例如 后来在http://ju.outofmemory.cn/entry/78365才看到,原来uniq检查重复 ...
- HTML 返回顶部的样式
.fhdb{ width:50px; height:50px; position:fixed; //可以用来定义元素的固定定位. right:20px; //距离右侧20像素 botto ...
- C#值数值类型转换
1.十进制转16进制 string result=number.ToString("X2"); >>0A //X2表示大写2位 2.字符串转数值类型 "); ...
- ssh免密码登陆设置
服务器端 CentOS 6.5下编辑/etc/ssh/sshd_config MacOSx下编辑/etc/sshd_config #开启公钥验证 RSAAuthentication yes Pubke ...