使用filebeat发送nginx日志到kafka
1.配置filebeat_nginx.yml
filebeat.modules:
- module: nginx
access:
enabled: true
var.paths: ["/var/log/nginx/access.log*"]
error:
enabled: true
var.paths: ["/var/log/nginx/error.log*"] #----------------------------------Kafka output--------------------------------#
output.kafka:
version: "1.0.1"
enabled: true
hosts: ['xxx:9092', 'xxx:9092', 'xxx:9092']
topic: 'temp'
required_acks: 1 #default
compression: gzip #default
max_message_bytes: 1000000 #default
codec.format:
string: '%{[message]}'
2.启动filebeat
./filebeat -e -c filebeat_nginx.yml
3.访问nginx
tail -f /var/log/nginx/access.log
日志文件输出
{"ts":"2019-10-14 10:53:22","host":"127.0.0.1","clientip":"127.0.0.1","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"localhost","url":"/index.html","domain":"localhost","xff":"-","referer":"-","status":"304"}
{"ts":"2019-10-14 10:53:23","host":"127.0.0.1","clientip":"127.0.0.1","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"localhost","url":"/index.html","domain":"localhost","xff":"-","referer":"-","status":"304"}
{"ts":"2019-10-14 10:53:23","host":"127.0.0.1","clientip":"127.0.0.1","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"localhost","url":"/index.html","domain":"localhost","xff":"-","referer":"-","status":"304"}
{"ts":"2019-10-14 10:53:30","host":"127.0.0.1","clientip":"127.0.0.1","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"localhost","url":"/index.html","domain":"localhost","xff":"-","referer":"-","status":"304"}
{"ts":"2019-10-14 10:53:31","host":"127.0.0.1","clientip":"127.0.0.1","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"localhost","url":"/index.html","domain":"localhost","xff":"-","referer":"-","status":"304"}
kafka输出
{"ts":"2019-10-14 10:53:23","host":"127.0.0.1","clientip":"127.0.0.1","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"localhost","url":"/index.html","domain":"localhost","xff":"-","referer":"-","status":"304"}
{"ts":"2019-10-14 10:53:23","host":"127.0.0.1","clientip":"127.0.0.1","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"localhost","url":"/index.html","domain":"localhost","xff":"-","referer":"-","status":"304"}
{"ts":"2019-10-14 10:53:22","host":"127.0.0.1","clientip":"127.0.0.1","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"localhost","url":"/index.html","domain":"localhost","xff":"-","referer":"-","status":"304"}
{"ts":"2019-10-14 10:53:30","host":"127.0.0.1","clientip":"127.0.0.1","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"localhost","url":"/index.html","domain":"localhost","xff":"-","referer":"-","status":"304"}
{"ts":"2019-10-14 10:53:31","host":"127.0.0.1","clientip":"127.0.0.1","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"localhost","url":"/index.html","domain":"localhost","xff":"-","referer":"-","status":"304"}
使用filebeat发送nginx日志到kafka的更多相关文章
- ELK整合Filebeat监控nginx日志
ELK 日志分析 1. 为什么用到 ELK 一般我们需要进行日志分析场景:直接在日志文件中 grep. awk 就可以获得自己想要的信息.但在规模较大的场景中,此方法效率低下,面临问题包括日志量太大如 ...
- 利用ELK分析Nginx日志生产实战(高清多图)
本文以api.mingongge.com.cn域名为测试对象进行统计,日志为crm.mingongge.com.cn和risk.mingongge.com.cn请求之和(此二者域名不具生产换环境统计意 ...
- 利用ELK分析Nginx日志
本文以api.mingongge.com.cn域名为测试对象进行统计,日志为crm.mingongge.com.cn和risk.mingongge.com.cn请求之和(此二者域名不具生产换环境统计意 ...
- ELK 6安装配置 nginx日志收集 kabana汉化
#ELK 6安装配置 nginx日志收集 kabana汉化 #环境 centos 7.4 ,ELK 6 ,单节点 #服务端 Logstash 收集,过滤 Elasticsearch 存储,索引日志 K ...
- 一次flume exec source采集日志到kafka因为单条日志数据非常大同步失败的踩坑带来的思考
本次遇到的问题描述,日志采集同步时,当单条日志(日志文件中一行日志)超过2M大小,数据无法采集同步到kafka,分析后,共踩到如下几个坑.1.flume采集时,通过shell+EXEC(tail -F ...
- ELK - nginx 日志分析及绘图
1. 前言 先上一张整体的效果图: 上面这张图就是通过 ELK 分析 nginx 日志所得到的数据,通过 kibana 的功能展示出来的效果图.是不是这样对日志做了解析,想要知道的数据一目了然.接下来 ...
- 在kibana中查看nginx日志的Discover,Dashboards
官方的操作: 1.安装filebeat,配置filebeat获取nginx日志,来源有两种: 第一种是使用自带的模块进行收集,在modules.d目录中启用模块配置,运行Filebeat时启用模块,在 ...
- CentOS6.9安装Filebeat监控Nginx的访问日志发送到Kafka
一.下载地址: 官方:https://www.elastic.co/cn/downloads/beats/filebeat 百度云盘:https://pan.baidu.com/s/1dvhqb0 二 ...
- Kafka+Zookeeper+Filebeat+ELK 搭建日志收集系统
ELK ELK目前主流的一种日志系统,过多的就不多介绍了 Filebeat收集日志,将收集的日志输出到kafka,避免网络问题丢失信息 kafka接收到日志消息后直接消费到Logstash Logst ...
随机推荐
- linux 管道,输出重定向,后端执行
1. | (管道) $time|data:"Y-m-d"将前部的输出,作为后部的输入:配合管道使用的命令,典型的就是grep,more find --help | grep cne ...
- PAT甲级1017题解——模拟排序
题目分析: 本题我第一次尝试去做的时候用的是优先队列,但是效率不仅代码量很大,而且还有测试样例过不去,很显然没有找到一个好的数据结构来解决这道题目(随着逐渐的刷PAT甲级的题会发现有时选择一个好的解题 ...
- ansible(二)
软件相关模块 yum rpm和yum的区别 rpm:redhat package manager yum可以解决依赖关系 yum源配置 [epel] name=Extra Packages - $ba ...
- MyBatis框架之入门(三)
使用原始dao层进行开发 UserMapper层接口 public interface UserMapper { /** * 通过id查询用户 * @param id * @return */ Use ...
- css 宽高等比
1.利用js 2.容器里添加图片,让图片的等比缩放撑大容器,图片z-index=负数,
- 模板内部不支持原生js的方法?》》artTemplate模版方法注册window全部事件方法
我的模板中出现了: {{parseInt(hasshakenum)}} 结果报错: Template Error resultPageTMP Render Error parseInt is not ...
- springBoot 利用Idea打包部署
springBoot 打包部署 1 项目如图: 2 依赖打包插件 3 打包操作 4 运行项目:
- 一个Java字符串中到底有多少个字符?
依照Java的文档, Java中的字符内部是以UTF-16编码方式表示的,最小值是 \u0000 (0),最大值是\uffff(65535), 也就是一个字符以2个字节来表示,难道Java最多只能表示 ...
- 创建Maven之后,Java包下无法直接创建servlet的原因:
---恢复内容开始--- 没有导入servlet-api的依赖,导入完依赖就可以直接创建了 依赖如下: <dependency> <groupId>javax.serv ...
- MongoDB shell 2 副本集方法
rs.initiate() rs.addArb() rs.help() rs.printReplicationInfo() 查看到副本集操作日志 rs.remove() 减少副本集节点 r ...