logstash json和rubydebug 第次重启logstash都会把所有的日志读完 而不是只读入新输入的内容
查看一下agent端的shipper的配置:
# cat logstash_test2.shipper.conf
input {
file {
path => ["/apps/logstash/conf/test/test2_log.txt"]
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
output {
stdout {
#codec => rubydebug
codec => json
}
}
#这个测试主要是看输出的格式为json的
先简测一下刚配好的shipper:
# ./../bin/logstash -f logstash_test2.shipper.conf -t
Sending Logstash's logs to /apps/logstash/logs which is now configured via log4j2.properties
Configuration OK
[--08T18::,][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
可以看到没有报错,接下来启动logstash并指定刚才配置好的配置文件:
# ./../bin/logstash -f logstash_test2.shipper.conf -t
Sending Logstash's logs to /apps/logstash/logs which is now configured via log4j2.properties
Configuration OK
[--08T18::,][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
[root@Appsrv130 conf]# ./../bin/logstash -f logstash_test2.shipper.conf
Sending Logstash's logs to /apps/logstash/logs which is now configured via log4j2.properties
[--08T18::,][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>, "pipeline.batch.size"=>, "pipeline.batch.delay"=>, "pipeline.max_inflight"=>}
[--08T18::,][INFO ][logstash.pipeline ] Pipeline main started
[--08T18::,][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>}
{"path":"/apps/logstash/conf/test/test2_log.txt","@timestamp":"2016-12-08T10:19:13.102Z","@version":"","host":"ofs1","message":"haha------>","tags":[]}{"path":"/apps/logstash/conf/test/test2_log.txt","@timestamp":"2016-12-08T10:19:13.113Z","@version":"","host":"ofs1","message":"haha------>2","tags":[]}{"path":"/apps/logstash/conf/test/test2_log.txt","@timestamp":"2016-12-08T10:19:13.118Z","@version":"","host":"ofs1","message":"haha------>3","tags":[]}{"path":"/apps/logstash/conf/test/test2_log.txt","@timestamp":"2016-12-08T10:19:13.121Z","@version":"","host":"ofs1","message":"haha------>3","tags":[]}
再看看所监控的log日志的内容:
# cat test/test2_log.txt
haha------>
haha------>
haha------>
haha------>
发现 这个shipper启动的时候会从头到尾,把配置文件全读一边(这种效里也是从配置文件中配置好的)
再看一下这个配置文件:
# cat logstash_test2.shipper.conf
input {
file {
path => ["/apps/logstash/conf/test/test2_log.txt"]
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
output {
stdout {
#codec => rubydebug
codec => json
}
}
#要点就是这行sincedb_path =>"/dev/null"了!该参数用来指定sincedb文件名,但是如果我们设置为/dev/null这个linux系统上特殊的空洞文件,
那么logstash每次重启进程的时候,尝试读取sincedb内容,都只会读到空洞,也就可以理解为前不有过运行记录,自然就从初始位置开始读取了!
下面往监控文件里写入内容时,会发生下面变化:
# echo "查看json格式是什么输出-------》">>test/test2_log.txt
再看一下输出的内容:
# ./../bin/logstash -f logstash_test2.shipper.conf -t
Sending Logstash's logs to /apps/logstash/logs which is now configured via log4j2.properties
Configuration OK
[--08T18::,][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
[root@Appsrv130 conf]# ./../bin/logstash -f logstash_test2.shipper.conf
Sending Logstash's logs to /apps/logstash/logs which is now configured via log4j2.properties
[--08T18::,][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>, "pipeline.batch.size"=>, "pipeline.batch.delay"=>, "pipeline.max_inflight"=>}
[--08T18::,][INFO ][logstash.pipeline ] Pipeline main started
[--08T18::,][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>}
{"path":"/apps/logstash/conf/test/test2_log.txt","@timestamp":"2016-12-08T10:19:13.102Z","@version":"","host":"ofs1","message":"haha------>","tags":[]}{"path":"/apps/logstash/conf/test/test2_log.txt","@timestamp":"2016-12-08T10:19:13.113Z","@version":"","host":"ofs1","message":"haha------>2","tags":[]}{"path":"/apps/logstash/conf/test/test2_log.txt","@timestamp":"2016-12-08T10:19:13.118Z","@version":"","host":"ofs1","message":"haha------>3","tags":[]}{"path":"/apps/logstash/conf/test/test2_log.txt","@timestamp":"2016-12-08T10:19:13.121Z","@version":"","host":"ofs1","message":"haha------>3","tags":[]}{"path":"/apps/logstash/conf/test/test2_log.txt","@timestamp":"2016-12-08T11:17:45.060Z","@version":"","host":"ofs1","message":"查看json格式是什么输出-------》","tags":[]}
修改配置文件:
# cat logstash_test2.shipper.conf
input {
file {
path => ["/apps/logstash/conf/test/test2_log.txt"]
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
output {
stdout {
codec => rubydebug #查看这种格式的日志输出
#codec => json
}
}
查看日志:
# echo "查看rubydebug格式是什么输出-------》">>test/test2_log.txt
# ./../bin/logstash -f logstash_test2.shipper.conf
Sending Logstash's logs to /apps/logstash/logs which is now configured via log4j2.properties
[--08T19::,][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>, "pipeline.batch.size"=>, "pipeline.batch.delay"=>, "pipeline.max_inflight"=>}
[--08T19::,][INFO ][logstash.pipeline ] Pipeline main started
[--08T19::,][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>}
{
"path" => "/apps/logstash/conf/test/test2_log.txt",
"@timestamp" => --08T11::.290Z,
"@version" => "",
"host" => "ofs1",
"message" => "haha------>",
"tags" => []
}
{
"path" => "/apps/logstash/conf/test/test2_log.txt",
"@timestamp" => --08T11::.299Z,
"@version" => "",
"host" => "ofs1",
"message" => "haha------>2",
"tags" => []
}
{
"path" => "/apps/logstash/conf/test/test2_log.txt",
"@timestamp" => --08T11::.301Z,
"@version" => "",
"host" => "ofs1",
"message" => "haha------>3",
"tags" => []
}
{
"path" => "/apps/logstash/conf/test/test2_log.txt",
"@timestamp" => --08T11::.302Z,
"@version" => "",
"host" => "ofs1",
"message" => "haha------>3",
"tags" => []
}
{
"path" => "/apps/logstash/conf/test/test2_log.txt",
"@timestamp" => --08T11::.303Z,
"@version" => "",
"host" => "ofs1",
"message" => "查看json格式是什么输出-------》",
"tags" => []
}
{
"path" => "/apps/logstash/conf/test/test2_log.txt",
"@timestamp" => --08T11::.415Z,
"@version" => "",
"host" => "ofs1",
"message" => "查看rubydebug格式是什么输出-------》",
"tags" => []
}
如果去掉上面的两个参数,看一下效果:
# cat logstash_test2.shipper.conf
input {
file {
path => ["/apps/logstash/conf/test/test2_log.txt"]
#start_position => "beginning"
#sincedb_path => "/dev/null"
}
}
output {
stdout {
codec => rubydebug
#codec => json
}
}
从另一个shell可以看到效果:
# ./../bin/logstash -f logstash_test2.shipper.conf
Sending Logstash's logs to /apps/logstash/logs which is now configured via log4j2.properties
[--09T13::,][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>, "pipeline.batch.size"=>, "pipeline.batch.delay"=>, "pipeline.max_inflight"=>}
[--09T13::,][INFO ][logstash.pipeline ] Pipeline main started
[--09T13::,][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>}
先导入数据:
echo '去掉参数start_position => "beginning" sincedb_path => "/dev/null"' >>test/test2_log.txt
下面看一下效果:
# ./../bin/logstash -f logstash_test2.shipper.conf
Sending Logstash's logs to /apps/logstash/logs which is now configured via log4j2.properties
[--09T13::,][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>, "pipeline.batch.size"=>, "pipeline.batch.delay"=>, "pipeline.max_inflight"=>}
[--09T13::,][INFO ][logstash.pipeline ] Pipeline main started
[--09T13::,][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>}
{
"path" => "/apps/logstash/conf/test/test2_log.txt",
"@timestamp" => --09T05::.155Z,
"@version" => "",
"host" => "ofs1",
"message" => "去掉参数start_position => \"beginning\" sincedb_path => \"/dev/null\"",
"tags" => []
}
logstash json和rubydebug 第次重启logstash都会把所有的日志读完 而不是只读入新输入的内容的更多相关文章
- ELK学习笔记之Logstash和Filebeat解析对java异常堆栈下多行日志配置支持
0x00 概述 logstash官方最新文档.假设有几十台服务器,每台服务器要监控系统日志syslog.tomcat日志.nginx日志.mysql日志等等,监控OOM.内存低下进程被kill.ngi ...
- 使用Elasticsearch、Logstash、Kibana与Redis(作为缓冲区)对Nginx日志进行收集(转)
摘要 使用Elasticsearch.Logstash.Kibana与Redis(作为缓冲区)对Nginx日志进行收集 版本 elasticsearch版本: elasticsearch-2.2.0 ...
- 小白都会超详细--ELK日志管理平台搭建教程
目录 一.介绍 二.安装JDK 三.安装Elasticsearch 四.安装Logstash 五.安装Kibana 六.Kibana简单使用 系统环境:CentOS Linux release 7.4 ...
- Logstash Json 过滤器插件
1. Json Filter 功能概述 这是一个JSON解析过滤器.它接受一个包含JSON的现有字段,并将其扩展为Logstash事件中的实际数据结构. 默认情况下,它将把解析过的JSON放在Logs ...
- Logstash:在 Docker 中部署 Logstash
文章转载自:https://elasticstack.blog.csdn.net/article/details/116516923 创建一个目录 docker-logstash.在该目录下,有如下的 ...
- logstash报错401 需要在logstash启动的配置文件中增加es的用户名和密码
- Logstash:如何使用Elasticsearch,Logstash和Kibana管理Apache日志
- 【linux】linux重启tomcat + 实时查看tomcat启动日志
linux重启tomcat命令: http://www.cnblogs.com/plus301/p/6237468.html linux查看toncat实时的启动日志: https://www.cnb ...
- Ajax请求Json数据,报500错误,后台没有错误日志。
post请求:http://localhost:9080/DataDiscoveryWeb/issueformcount/queryIssueTendencyDetail.xhtml?jobId=86 ...
随机推荐
- leetcode 63. Unique Paths II
Follow up for "Unique Paths": Now consider if some obstacles are added to the grids. How m ...
- python traceback 变量值
import sys import traceback import cgitb def handleException(excType, excValue, trace): print 'error ...
- 将 JAR 转为 EXE – JSMOOTH 的使用教程(第二期)(转载)
http://www.iteknical.com/convert-jar-to-exe-phase-ii-jsmooth-use-tutorial/
- zookeeper部署及集群测试
zookeeper部署及集群测试 环境 三台测试机 操作系统: centos7 ; hostname: c1 ; ip: 192.168.1.80 操作系统: centos7 ; hostname: ...
- RecyclerView notifyDataSetChanged不起作用
一般listview设置完data后调用notifyDataSetChanged便可刷新布局界面,然而recycleview调用这个方法却没有任何反应.对于很多不熟悉recycleview的话很容易躺 ...
- hnu10104
AC自动机+DFS #include <cstdio> #include <queue> #include <cstring> using namespace st ...
- apache官网怎样下载apache HTTP Server服务器
我相信有些朋友刚用apache服务器时,都希望从官网上下载,而面对着官网上众多的项目和镜像以及目录,也许有点茫然.下面是具体步骤 第一步:打开apache官网 第二步:点击右上角Download,出现 ...
- malloc原理和内存碎片
当一个进程发生缺页中断的时候,进程会陷入内核态,执行以下操作: 1.检查要访问的虚拟地址是否合法 2.查找/分配一个物理页 3.填充物理页内容(读取磁盘,或者直接置0,或者啥也不干) 4.建立映射关系 ...
- Divide and conquer:Matrix(POJ 3685)
矩阵 题目大意:矩阵里面的元素按i*i + 100000 * i + j*j - 100000 * j + i*j填充(i是行,j是列),求最小的M个数 这一题要用到两次二分,实在是二分法的经典,主要 ...
- 【C语言】指针
错误一: 一种错误的写法: * sizeof(int)); * sizeof(int)); y = x; 没有必要为y开辟内存,因为y在开辟内存时 y内存储的地址时开辟的内存的位置, 但是后面又把x的 ...