基本环境：
filebeat版本：6.5.4 (Linux,x86-64)
elasticsearch版本：6.54
 
（一）需求说明
在一台服务器上有多个日志需要使用filebeat日志收集到elasticsearch中，以便于查看。对于收集方法，主要有2种：
  • 将同一台服务器上的日志收集到elasticsearch的同一个索引中，这种方式存在一个较大的问题，如果服务器上有多个业务在运行，产生了多个日志，那么将会被收集到elasticsearch的同一个索引中，如图1。
  • 将同一台服务器上的日志收集到elasticsearch的不同索引中，每个索引都存放相关业务的日志，如图2。
 
 
很明显，图2的日志输出是我们想要的，因为它将不同的日志放到了不同的索引中。
 
（二）解决方案
在使用filbeat收集日志输出到elasticsearch数据库时，可以使用indices参数来配置不同的日志输出到不同的索引中，官方文档及其配置例子如下：
 
 
（三）实际测试
（3.1）输入存在fields_under_root: true选项
使用filebeat对3个日志testa.log、testb.log、testc.log进行数据抓取，要求：
  • testa.log日志的数据存放到testa-log索引中
  • testb.log日志的数据存放到testb-log索引中
  • 其它(非testa.log和testb.log)的日志数据存放到test-other-log索引中
 
filebeat输入配置如下：
 
输出配置如下：
 
 
最终测试成功。
 
这里附一份完整的filebeat配置文件：

###################### Filebeat Configuration Example #########################

# This file is an example configuration file highlighting only the most common

# options. The filebeat.reference.yml file from the same directory contains all the

# supported options with more comments. You can use it as a reference.

#

# You can find the full configuration reference here:

# https://www.elastic.co/guide/en/beats/filebeat/index.html

# For more available modules and options, please see the filebeat.reference.yml sample

# configuration file.

#=========================== Filebeat inputs =============================

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so

# you can use different inputs for various configurations.

# Below are the input specific configurations.

# testa.log

- type: log

  enabled: true

  paths:

    - /root/test/testa.log

  fields:

    log_topics: "testa"

  fields_under_root: true

# testb.log

- type: log

  enabled: true

  paths:

    - /root/test/testb.log

  fields:

    log_topics: "testb"

  fields_under_root: true

# testc.log

- type: log

  enabled: true

  paths:

    - /root/test/testc.log

  fields:

    log_topics: "testc"

  fields_under_root: true

  # Exclude lines. A list of regular expressions to match. It drops the lines that are

  # matching any regular expression from the list.

  #exclude_lines: ['^DBG']

  # Include lines. A list of regular expressions to match. It exports the lines that are

  # matching any regular expression from the list.

  #include_lines: ['^ERR', '^WARN']

  # Exclude files. A list of regular expressions to match. Filebeat drops the files that

  # are matching any regular expression from the list. By default, no files are dropped.

  #exclude_files: ['.gz$']

  # Optional additional fields. These fields can be freely picked

  # to add additional information to the crawled log files for filtering

  #fields:

  #  level: debug

  #  review: 1

  ### Multiline options

  # Multiline can be used for log messages spanning multiple lines. This is common

  # for Java Stack Traces or C-Line Continuation

  # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [

  #multiline.pattern: ^\[

  # Defines if the pattern set under pattern should be negated or not. Default is false.

  #multiline.negate: false

  # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern

  # that was (not) matched before or after or as long as a pattern is not matched based on negate.

  # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash

  #multiline.match: after

#============================= Filebeat modules ===============================

filebeat.config.modules:

  # Glob pattern for configuration loading

  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading

  reload.enabled: true

  # Period on which files under path should be checked for changes

  #reload.period: 10s

#==================== Elasticsearch template setting ==========================

setup.template.settings:

  index.number_of_shards: 1

  #index.codec: best_compression

  #_source.enabled: false

setup.template.name: "prod-file*"

setup.template.pattern: "prod-file*"

setup.ilm.enabled: false

#================================ General =====================================

# The name of the shipper that publishes the network data. It can be used to group

# all the transactions sent by a single shipper in the web interface.

#name:

# The tags of the shipper are included in their own field with each

# transaction published.

#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the

# output.

#fields:

#  env: staging

#============================== Dashboards =====================================

# These settings control loading the sample dashboards to the Kibana index. Loading

# the dashboards is disabled by default and can be enabled either by setting the

# options here or by using the `setup` command.

#setup.dashboards.enabled: false

# The URL from where to download the dashboards archive. By default this URL

# has a value which is computed based on the Beat name and version. For released

# versions, this URL points to the dashboard archive on the artifacts.elastic.co

# website.

#setup.dashboards.url:

#============================== Kibana =====================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.

# This requires a Kibana endpoint configuration.

setup.kibana:

  # Kibana Host

  # Scheme and port can be left out and will be set to the default (http and 5601)

  # In case you specify and additional path, the scheme is required: http://localhost:5601/path

  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601

  #host: "localhost:5601"

  # Kibana Space ID

  # ID of the Kibana Space into which the dashboards should be loaded. By default,

  # the Default Space will be used.

  #space.id:

#============================= Elastic Cloud ==================================

# These settings simplify using Filebeat with the Elastic Cloud (https://cloud.elastic.co/).

# The cloud.id setting overwrites the `output.elasticsearch.hosts` and

# `setup.kibana.host` options.

# You can find the `cloud.id` in the Elastic Cloud web UI.

#cloud.id:

# The cloud.auth setting overwrites the `output.elasticsearch.username` and

# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.

#cloud.auth:

#================================ Outputs =====================================

# Configure what output to use when sending the data collected by the beat.

#-------------------------- Elasticsearch output ------------------------------

#output.elasticsearch:

#  hosts: ["192.168.10.30:9200"]

#  index: "testlog-666"

#output.elasticsearch:

#  hosts: ["192.168.10.30:9200"]

#  indices:

#    - index: "testa-log"

#      when.contains:

#        log_topics: "testa"

#    - index: "testb-log"

#      when.contains:

#        log_topics: "testb"

output.elasticsearch:

  hosts: ["192.168.10.100:9200"]

  index: "test-other-log"

  indices:

    - index: "testa-log"

      when.contains:

        log_topics: "testa"

    - index: "testb-log"

      when.contains:

        log_topics: "testb"

#----------------------------- Logstash output --------------------------------

#output.logstash:

  # The Logstash hosts

  #hosts: ["localhost:5044"]

  # Optional SSL. By default is off.

  # List of root certificates for HTTPS server verifications

  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication

  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key

  #ssl.key: "/etc/pki/client/cert.key"

#================================ Processors =====================================

# Configure processors to enhance or manipulate events generated by the beat.

#================================ Logging =====================================

# Sets log level. The default log level is info.

# Available log levels are: error, warning, info, debug

#logging.level: debug

# At debug level, you can selectively enable logging only for some components.

# To enable all selectors use ["*"]. Examples of other selectors are "beat",

# "publish", "service".

#logging.selectors: ["*"]

#============================== Xpack Monitoring ===============================

# filebeat can export internal metrics to a central Elasticsearch monitoring

# cluster.  This requires xpack monitoring to be enabled in Elasticsearch.  The

# reporting is disabled by default.

# Set to true to enable the monitoring reporter.

#monitoring.enabled: false

# Uncomment to send the metrics to Elasticsearch. Most settings from the

# Elasticsearch output are accepted here as well.

# Note that the settings should point to your Elasticsearch *monitoring* cluster.

# Any setting that is not set is automatically inherited from the Elasticsearch

# output configuration, so if you have the Elasticsearch output configured such

# that it is pointing to your Elasticsearch monitoring cluster, you can simply

# uncomment the following line.

#monitoring.elasticsearch:

#================================= Migration ==================================

# This allows to enable 6.7 migration aliases

#migration.6_to_7.enabled: true
 （3.2）输入不存在fields_under_root: true选项
filebeat输入配置如下：
 
输出配置如下：
 
这里附一份完整的filebeat配置文件：

###################### Filebeat Configuration Example #########################

# This file is an example configuration file highlighting only the most common

# options. The filebeat.reference.yml file from the same directory contains all the

# supported options with more comments. You can use it as a reference.

#

# You can find the full configuration reference here:

# https://www.elastic.co/guide/en/beats/filebeat/index.html

# For more available modules and options, please see the filebeat.reference.yml sample

# configuration file.

#=========================== Filebeat inputs =============================

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so

# you can use different inputs for various configurations.

# Below are the input specific configurations.

# testa.log

- type: log

  enabled: true

  paths:

    - /root/test/testa.log

  fields:

    log_topics: "testa"

# testb.log

- type: log

  enabled: true

  paths:

    - /root/test/testb.log

  fields:

    log_topics: "testb"

# testc.log

- type: log

  enabled: true

  paths:

    - /root/test/testc.log

  fields:

    log_topics: "testc"

  # Exclude lines. A list of regular expressions to match. It drops the lines that are

  # matching any regular expression from the list.

  #exclude_lines: ['^DBG']

  # Include lines. A list of regular expressions to match. It exports the lines that are

  # matching any regular expression from the list.

  #include_lines: ['^ERR', '^WARN']

  # Exclude files. A list of regular expressions to match. Filebeat drops the files that

  # are matching any regular expression from the list. By default, no files are dropped.

  #exclude_files: ['.gz$']

  # Optional additional fields. These fields can be freely picked

  # to add additional information to the crawled log files for filtering

  #fields:

  #  level: debug

  #  review: 1

  ### Multiline options

  # Multiline can be used for log messages spanning multiple lines. This is common

  # for Java Stack Traces or C-Line Continuation

  # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [

  #multiline.pattern: ^\[

  # Defines if the pattern set under pattern should be negated or not. Default is false.

  #multiline.negate: false

  # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern

  # that was (not) matched before or after or as long as a pattern is not matched based on negate.

  # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash

  #multiline.match: after

#============================= Filebeat modules ===============================

filebeat.config.modules:

  # Glob pattern for configuration loading

  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading

  reload.enabled: true

  # Period on which files under path should be checked for changes

  #reload.period: 10s

#==================== Elasticsearch template setting ==========================

setup.template.settings:

  index.number_of_shards: 1

  #index.codec: best_compression

  #_source.enabled: false

setup.template.name: "prod-file*"

setup.template.pattern: "prod-file*"

setup.ilm.enabled: false

#================================ General =====================================

# The name of the shipper that publishes the network data. It can be used to group

# all the transactions sent by a single shipper in the web interface.

#name:

# The tags of the shipper are included in their own field with each

# transaction published.

#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the

# output.

#fields:

#  env: staging

#============================== Dashboards =====================================

# These settings control loading the sample dashboards to the Kibana index. Loading

# the dashboards is disabled by default and can be enabled either by setting the

# options here or by using the `setup` command.

#setup.dashboards.enabled: false

# The URL from where to download the dashboards archive. By default this URL

# has a value which is computed based on the Beat name and version. For released

# versions, this URL points to the dashboard archive on the artifacts.elastic.co

# website.

#setup.dashboards.url:

#============================== Kibana =====================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.

# This requires a Kibana endpoint configuration.

setup.kibana:

  # Kibana Host

  # Scheme and port can be left out and will be set to the default (http and 5601)

  # In case you specify and additional path, the scheme is required: http://localhost:5601/path

  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601

  #host: "localhost:5601"

  # Kibana Space ID

  # ID of the Kibana Space into which the dashboards should be loaded. By default,

  # the Default Space will be used.

  #space.id:

#============================= Elastic Cloud ==================================

# These settings simplify using Filebeat with the Elastic Cloud (https://cloud.elastic.co/).

# The cloud.id setting overwrites the `output.elasticsearch.hosts` and

# `setup.kibana.host` options.

# You can find the `cloud.id` in the Elastic Cloud web UI.

#cloud.id:

# The cloud.auth setting overwrites the `output.elasticsearch.username` and

# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.

#cloud.auth:

#================================ Outputs =====================================

# Configure what output to use when sending the data collected by the beat.

#-------------------------- Elasticsearch output ------------------------------

#output.elasticsearch:

#  hosts: ["192.168.10.30:9200"]

#  index: "testlog-666"

#output.elasticsearch:

#  hosts: ["192.168.10.30:9200"]

#  indices:

#    - index: "testa-log"

#      when.contains:

#        log_topics: "testa"

#    - index: "testb-log"

#      when.contains:

#        log_topics: "testb"

output.elasticsearch:

  hosts: ["192.168.10.100:9200"]

  index: "test-other-log"

  indices:

    - index: "testa-log"

      when.contains:

        fields:

          log_topics: "testa"

    - index: "testb-log"

      when.contains:

        fields:

          log_topics: "testb"

#----------------------------- Logstash output --------------------------------

#output.logstash:

  # The Logstash hosts

  #hosts: ["localhost:5044"]

  # Optional SSL. By default is off.

  # List of root certificates for HTTPS server verifications

  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication

  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key

  #ssl.key: "/etc/pki/client/cert.key"

#================================ Processors =====================================

# Configure processors to enhance or manipulate events generated by the beat.

#================================ Logging =====================================

# Sets log level. The default log level is info.

# Available log levels are: error, warning, info, debug

#logging.level: debug

# At debug level, you can selectively enable logging only for some components.

# To enable all selectors use ["*"]. Examples of other selectors are "beat",

# "publish", "service".

#logging.selectors: ["*"]

#============================== Xpack Monitoring ===============================

# filebeat can export internal metrics to a central Elasticsearch monitoring

# cluster.  This requires xpack monitoring to be enabled in Elasticsearch.  The

# reporting is disabled by default.

# Set to true to enable the monitoring reporter.

#monitoring.enabled: false

# Uncomment to send the metrics to Elasticsearch. Most settings from the

# Elasticsearch output are accepted here as well.

# Note that the settings should point to your Elasticsearch *monitoring* cluster.

# Any setting that is not set is automatically inherited from the Elasticsearch

# output configuration, so if you have the Elasticsearch output configured such

# that it is pointing to your Elasticsearch monitoring cluster, you can simply

# uncomment the following line.

#monitoring.elasticsearch:

#================================= Migration ==================================

# This allows to enable 6.7 migration aliases

#migration.6_to_7.enabled: true
 
 
 其它说明：为什么需要特别注意fields_under_root参数
fields_under_root参数定义如下：
  • 如果值为ture，那么fields存储在输出文档的顶级位置，如果与filebeat中字段冲突，自定义字段会覆盖其他字段
  • 如果值为false或者未设置，那么fields存储在输出文档的子位置。
如下：
（1）fields_under_root：true
此时在filebeat的input部分定义的字段log_topics是一个顶级字段。
 
 
 
 
（2）fields_under_root：false或者未设置
此时在filebeat的input部分定义的字段log_topics是fields字段的子字段。
 
 
 
 
【完】

filebeat输出结果到elasticsearch的多个索引的更多相关文章

  1. 使用ElasticSearch赋能HBase二级索引 | 实践一年后总结

    前言:还记得那是2018年的一个夏天,天气特别热,我一边擦汗一边听领导大刀阔斧的讲述自己未来的改革蓝图.会议开完了,核心思想就是:我们要搞一个数据大池子,要把公司能灌的数据都灌入这个大池子,然后让别人 ...

  2. 第三百六十二节，Python分布式爬虫打造搜索引擎Scrapy精讲—elasticsearch(搜索引擎)基本的索引和文档CRUD操作、增、删、改、查

    第三百六十二节,Python分布式爬虫打造搜索引擎Scrapy精讲—elasticsearch(搜索引擎)基本的索引和文档CRUD操作.增.删.改.查 elasticsearch(搜索引擎)基本的索引 ...

  3. (转)ElasticSearch Java Api-检索索引库

    上篇博客记录了如何用java调用api把数据写入索引,这次记录下如何搜索. 一.准备数据 String data1 = JsonUtil.model2Json(new Blog(1, "gi ...

  4. 四十一 Python分布式爬虫打造搜索引擎Scrapy精讲—elasticsearch(搜索引擎)基本的索引和文档CRUD操作、增、删、改、查

    elasticsearch(搜索引擎)基本的索引和文档CRUD操作 也就是基本的索引和文档.增.删.改.查.操作 注意:以下操作都是在kibana里操作的 elasticsearch(搜索引擎)都是基 ...

  5. Elasticsearch之curl创建索引

    前提,是 Elasticsearch之curl创建索引库 [hadoop@djt002 elasticsearch-2.4.3]$ curl -XPUT 'http://192.168.80.200: ...

  6. Elasticsearch之curl创建索引库

    关于curl的介绍,请移步 Elasticsearch学习概念之curl 启动es,请移步 Elasticsearch的前后台运行与停止(tar包方式) Elasticsearch的前后台运行与停止( ...

  7. Elasticsearch之curl删除索引库

    关于curl创建索引库的介绍,请移步 Elasticsearch之curl创建索引库 [hadoop@djt002 elasticsearch-2.4.3]$ curl -XPUT 'http://1 ...

  8. Elasticsearch之curl创建索引库和索引时注意事项

    前提, Elasticsearch之curl创建索引库 Elasticsearch之curl创建索引 注意事项 1.索引库名称必须要全部小写,不能以下划线开头,也不能包含逗号 2.如果没有明确指定索引 ...

  9. Elasticsearch之cur查询索引

    前提, Elasticsearch之curl创建索引库 Elasticsearch之curl创建索引 Elasticsearch之curl创建索引库和索引时注意事项 Elasticsearch之cur ...

随机推荐

  1. CMake编译OpenCV4.0时opencv_ffmpeg.dll等下载失败的解决思路总结

    一.Configure会报一些红色的警告信息,比如: 1 CMake Warning at cmake/OpenCVDownload.cmake:193 (message): 2 FFMPEG: Do ...

  2. 幻读在 InnoDB 中是被如何解决的?(转)

    在MySQL事务初识中,我们了解到不同的事务隔离级别会引发不同的问题,如在 RR 级别下会出现幻读.但如果将存储引擎选为 InnoDB ,在 RR 级别下,幻读的问题就会被解决.在这篇文章中,会先介绍 ...

  3. pycharm 报错及解决方法

    1.报错: AttributeError: 'list' object has no attribute 'click' 原因:应是find_element_by 不是 find_elements_b ...

  4. 在线动态修改ulimit

    前言 系统中有些地方会进行资源的限制,其中的一个就是open file的限制,操作系统默认限制的是1024,这个值可以通过各种方式修改,本篇主要讲的是如何在线修改,生产上是不可能随便重启进程的 实践 ...

  5. Linux下查询外网IP的办法。

    Curl 纯文本格式输出:curl icanhazip.comcurl ifconfig.mecurl curlmyip.comcurl ip.appspot.comcurl ipinfo.io/ip ...

  6. html2image

    测试没有,生产有 也就是写死的显示,配置的不显示

  7. springboot中使用Filter、Interceptor和aop拦截REST服务

    在springboot中使用rest服务时,往往需要对controller层的请求进行拦截或者获取请求数据和返回数据,就需要过滤器.拦截器或者切片. 过滤器(Filter):对HttpServletR ...

  8. 还不懂java类加载机制的，建议看下这份阿里技术官总结的笔记！

    类加载机制 把class文件加载到内存,并对数据进行校验,准备,解析,初始化,形成可以被虚拟机直接使用的字节码 类加载的时机(触发类的初始化) 使用new关键字实例化对象 读取一个类的静态代码块 使用 ...

  9. MathType如何对齐公式

    作为强大的公式编辑器,MathType为我们的学习.工作带来了极大的便利.比如在写论文时,有了它,就可以轻松就把论文里的公式码完:老师在编写试卷时,利用它,可以快速编写出一份试卷.那么在编写公式时,也 ...

  10. Java基础教程——Set

    Set·无序,不重复 HashSet 特点:没有重复数据,数据不按存入的顺序输出. HashSet由Hash表结构支持.不支持set的迭代顺序,不保证顺序. 但是Hash表结构查询速度很快. 创建集合 ...