前言

  刚从事开发那段时间不习惯输出日志,认为那是无用功,徒增代码量,总认为自己的代码无懈可击;老大的叮嘱、强调也都视为耳旁风,最终导致的结果是我加班排查问题,花的时间还挺长的,要复现问题、排查问题等,幸亏那是公司内部员工用的系统,时间长一点也没什么大问题,但是如果是针对客户的,时间就代表很多东西了,那造成的影响就非常大了。自那以后养成了输出日志的习惯。

  但是后来发现,仅仅只是输出日志文件,对于排查问题来说,还是很费时,因为要在一个庞大的日志文件中过滤出我们需要的信息也十分耗时;那么此时基于日志文件的日志系统就被需要了。

  至于需不需要搭建日志系统、以及搭建一个怎样的日志系统,需要根据我们的业务实际情况而定,例如公司内部员工用的一个不重要的系统,那么日志文件可能就够了;而对于针对客户的、直接与公司利益挂钩的,我认为不仅要搭建日志系统,更要输出更详细的日志信息到日志文件,提供运维的效率。

  elk + filebeat 各个组件的功能

    Elasticsearch:分布式搜索和分析引擎,具有高可伸缩、高可靠和易管理等特点。基于 Apache Lucene 构建,能对大容量的数据进行接近实时的存储、搜索和分析操作。通常被用作某些应用的基础搜索引擎,使其具有复杂的搜索功能;
    Logstash:数据收集引擎。它支持动态的从各种数据源搜集数据,并对数据进行过滤、分析、丰富、统一格式等操作,然后存储到用户指定的位置;
    Kibana:数据分析和可视化平台。通常与 Elasticsearch 配合使用,对其中数据进行搜索、分析和以统计图表的方式展示;
    Filebeat:一个轻量级开源日志文件数据搜集器,基于 Logstash-Forwarder 源代码开发,是对它的替代。在需要采集日志数据的 server 上安装 Filebeat,并指定日志目录或日志文件后,Filebeat 就能读取数据,迅速发送到 Logstash 进行解析,亦或直接发送到 Elasticsearch 进行集中式存储和分析;

  本文不会对各个组件做详细的介绍与使用说明,如果想对各个组件有更详细的了解,那么需要大伙自行去学习,官网的资料就很不错。

环境准备

  192.168.1.110:logstash + java

  192.168.1.111:filebeat + redis + mysql + jdk + tomcat8

  192.168.1.112:kibana

  192.168.1.113:elasticsearch + java

日志系统搭建

  相关安装包大家自行准备,去官网下载即可;elk+filebeat我用的都是6.3.2版本,jdk用的1.8版本,mysql是5.7,tomcat是8.5.30

  Elasticsearch

    依赖jdk,jdk的 搭建可参考我的 另一篇博客:virtualBox安装centos,并搭建tomcat

    [root@cent0s7-03 opt]# tar -zxvf elasticsearch-6.3.2.tar.gz

  [root@cent0s7-03 opt]# cd elasticsearch-6.3.2

    修改配置,支持远程访问:

      修改elasticsearch的home目录/config/elasticsearch.yml,打开配置项network.host:,并将其值设置成0.0.0.0;

      但是需要增加系统配置来支持:

        [root@cent0s7-03 bin]# vi /etc/security/limits.conf

        新增如下配置        

  1. * soft nofile 65536
  2. * hard nofile 131072
  3. * soft nproc 2048
  4. * hard nproc 4096

        [root@cent0s7-03 bin]# vi /etc/sysctl.conf

        新增如下配置

  1. vm.max_map_count=262144

        [root@cent0s7-03 bin]# sysctl -p

    启动elasticsearch

      [root@cent0s7-03 bin]# ./elasticsearch

      发现报错,如下

  1. [2018-08-19T10:26:33,685][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
  2. org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: can not run elasticsearch as root
  3. at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:140) ~[elasticsearch-6.3.2.jar:6.3.2]
  4. at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) ~[elasticsearch-6.3.2.jar:6.3.2]
  5. at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.3.2.jar:6.3.2]
  6. at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.3.2.jar:6.3.2]
  7. at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.3.2.jar:6.3.2]
  8. at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.3.2.jar:6.3.2]
  9. at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) ~[elasticsearch-6.3.2.jar:6.3.2]
  10. Caused by: java.lang.RuntimeException: can not run elasticsearch as root
  11. at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:104) ~[elasticsearch-6.3.2.jar:6.3.2]
  12. at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:171) ~[elasticsearch-6.3.2.jar:6.3.2]
  13. at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.3.2.jar:6.3.2]
  14. at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.3.2.jar:6.3.2]
  15. ... 6 more

      这是出于系统安全考虑设置的条件。由于ElasticSearch可以接收用户输入的脚本并且执行,为了系统安全考虑,建议创建一个单独的用户用来运行ElasticSearch。 

    [root@cent0s7-03 bin]# groupadd elk

    [root@cent0s7-03 bin]# useradd elsearch -g elk

    [root@cent0s7-03 bin]# cd /opt

    [root@cent0s7-03 opt]# chown -R elsearch:elk elasticsearch-6.3.2

    [root@cent0s7-03 opt]# su elsearch

    [elsearch@cent0s7-03 opt]$ cd elasticsearch-6.3.2/bin

    [elsearch@cent0s7-03 bin]$ ./elasticsearch (加-d,则表示后端运行)

    访问:http://192.168.1.113:9200,出现如下信息

  1. {
  2. "name" : "8dBt-dz",
  3. "cluster_name" : "elasticsearch",
  4. "cluster_uuid" : "gGH8gMvjTm62yyjob3aeZA",
  5. "version" : {
  6. "number" : "6.3.2",
  7. "build_flavor" : "default",
  8. "build_type" : "tar",
  9. "build_hash" : "053779d",
  10. "build_date" : "2018-07-20T05:20:23.451332Z",
  11. "build_snapshot" : false,
  12. "lucene_version" : "7.3.1",
  13. "minimum_wire_compatibility_version" : "5.6.0",
  14. "minimum_index_compatibility_version" : "5.0.0"
  15. },
  16. "tagline" : "You Know, for Search"
  17. }

    表示单节点的elasticsearch搭建起来了

  Kibana  

    [root@centos7-02 opt]# tar -zxvf kibana-6.3.2-linux-x86_64.tar.gz

    [root@centos7-02 opt]# mv kibana-6.3.2-linux-x86_64 kibana6.3.2

    修改配置文件:kibana.yml

      [root@centos7-02 opt]# vi kibana6.3.2/config/kibana.yml

    主要改改两项:

  1. server.host: "0.0.0.0"
  2. elasticsearch.url: "http://192.168.1.113:9200"

      支援远程访问和从elasticsearch获取数据

    [root@centos7-02 opt]# ./kibana6.3.2/bin/kibana

    启动日志如下

  1. log [11:09:00.993] [info][status][plugin:kibana@6.3.2] Status changed from uninitialized to green - Ready
  2. log [11:09:01.032] [info][status][plugin:elasticsearch@6.3.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  3. log [11:09:01.034] [info][status][plugin:xpack_main@6.3.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  4. log [11:09:01.042] [info][status][plugin:searchprofiler@6.3.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  5. log [11:09:01.045] [info][status][plugin:ml@6.3.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  6. log [11:09:01.106] [info][status][plugin:tilemap@6.3.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  7. log [11:09:01.107] [info][status][plugin:watcher@6.3.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  8. log [11:09:01.123] [info][status][plugin:license_management@6.3.2] Status changed from uninitialized to green - Ready
  9. log [11:09:01.125] [info][status][plugin:index_management@6.3.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  10. log [11:09:01.243] [info][status][plugin:timelion@6.3.2] Status changed from uninitialized to green - Ready
  11. log [11:09:01.245] [info][status][plugin:graph@6.3.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  12. log [11:09:01.248] [info][status][plugin:monitoring@6.3.2] Status changed from uninitialized to green - Ready
  13. log [11:09:01.250] [info][status][plugin:security@6.3.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  14. log [11:09:01.251] [warning][security] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in kibana.yml
  15. log [11:09:01.255] [warning][security] Session cookies will be transmitted over insecure connections. This is not recommended.
  16. log [11:09:01.280] [info][status][plugin:grokdebugger@6.3.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  17. log [11:09:01.300] [info][status][plugin:dashboard_mode@6.3.2] Status changed from uninitialized to green - Ready
  18. log [11:09:01.304] [info][status][plugin:logstash@6.3.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  19. log [11:09:01.326] [info][status][plugin:apm@6.3.2] Status changed from uninitialized to green - Ready
  20. log [11:09:01.330] [info][status][plugin:console@6.3.2] Status changed from uninitialized to green - Ready
  21. log [11:09:01.332] [info][status][plugin:console_extensions@6.3.2] Status changed from uninitialized to green - Ready
  22. log [11:09:01.334] [info][status][plugin:metrics@6.3.2] Status changed from uninitialized to green - Ready
  23. log [11:09:01.644] [warning][reporting] Generating a random key for xpack.reporting.encryptionKey. To prevent pending reports from failing on restart, please set xpack.reporting.encryptionKey in kibana.yml
  24. log [11:09:01.651] [info][status][plugin:reporting@6.3.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  25. log [11:09:01.697] [info][listening] Server running at http://0.0.0.0:5601
  26. log [11:09:01.819] [info][status][plugin:elasticsearch@6.3.2] Status changed from yellow to green - Ready
  27. log [11:09:01.852] [info][license][xpack] Imported license information from Elasticsearch for the [data] cluster: mode: basic | status: active
  28. log [11:09:01.893] [info][status][plugin:xpack_main@6.3.2] Status changed from yellow to green - Ready
  29. log [11:09:01.893] [info][status][plugin:searchprofiler@6.3.2] Status changed from yellow to green - Ready
  30. log [11:09:01.894] [info][status][plugin:ml@6.3.2] Status changed from yellow to green - Ready
  31. log [11:09:01.894] [info][status][plugin:tilemap@6.3.2] Status changed from yellow to green - Ready
  32. log [11:09:01.895] [info][status][plugin:watcher@6.3.2] Status changed from yellow to green - Ready
  33. log [11:09:01.895] [info][status][plugin:index_management@6.3.2] Status changed from yellow to green - Ready
  34. log [11:09:01.895] [info][status][plugin:graph@6.3.2] Status changed from yellow to green - Ready
  35. log [11:09:01.896] [info][status][plugin:security@6.3.2] Status changed from yellow to green - Ready
  36. log [11:09:01.897] [info][status][plugin:grokdebugger@6.3.2] Status changed from yellow to green - Ready
  37. log [11:09:01.897] [info][status][plugin:logstash@6.3.2] Status changed from yellow to green - Ready
  38. log [11:09:01.898] [info][status][plugin:reporting@6.3.2] Status changed from yellow to green - Ready
  39. log [11:09:01.916] [info][kibana-monitoring][monitoring-ui] Starting all Kibana monitoring collectors
  40. log [11:09:01.926] [info][license][xpack] Imported license information from Elasticsearch for the [monitoring] cluster: mode: basic | status: active

      几个警告不影响功能,不出现error就可以正常服务。

    访问:http://192.168.1.112:5601,出现下图

  Logstash

    依赖jdk,jdk的 搭建可参考我的 另一篇博客:virtualBox安装centos,并搭建tomcat

    [root@centos7-01 opt]# tar -zxvf logstash-6.3.2.tar.gz

    新增配置文件:first-pipeline.conf

    [root@centos7-01 opt]# vi logstash-6.3.2/config/first-pipeline.conf

  1. input {
  2. stdin {}
  3. beats {
  4. port => 5044
  5. }
  6. }
  7. output {
  8. elasticsearch {
  9. hosts => ["192.168.1.113:9200"]
  10. }
  11. stdout {
  12. codec => rubydebug
  13. }
  14. }

      监听5044端口,filebeat会从此端口向logstash写入数据;logstash处理数据之后(filter,实例中没有展示)再输出到elasticsearch

    [root@centos7-01 opt]# ./logstash-6.3.2/bin/logstash -f /opt/logstash-6.3.2/config/first-pipeline.conf

      启动日志如下

  1. Sending Logstash's logs to /opt/logstash-6.3.2/logs which is now configured via log4j2.properties
  2. [2018-09-03T20:59:05,050][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
  3. [2018-09-03T20:59:06,072][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.3.2"}
  4. [2018-09-03T20:59:11,487][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
  5. [2018-09-03T20:59:12,222][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://192.168.1.113:9200/]}}
  6. [2018-09-03T20:59:12,230][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://192.168.1.113:9200/, :path=>"/"}
  7. [2018-09-03T20:59:12,574][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://192.168.1.113:9200/"}
  8. [2018-09-03T20:59:12,669][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
  9. [2018-09-03T20:59:12,672][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
  10. [2018-09-03T20:59:12,775][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//192.168.1.113:9200"]}
  11. [2018-09-03T20:59:12,810][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
  12. [2018-09-03T20:59:12,862][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
  13. [2018-09-03T20:59:13,758][INFO ][logstash.inputs.beats ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
  14. The stdin plugin is now waiting for input:
  15. [2018-09-03T20:59:13,852][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x2ff95604 run>"}
  16. [2018-09-03T20:59:13,958][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
  17. [2018-09-03T20:59:14,066][INFO ][org.logstash.beats.Server] Starting server on port: 5044
  18. [2018-09-03T20:59:14,562][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

  Filebeat  

    [root@centos7 opt]# tar -zxvf filebeat-6.3.2-linux-x86_64.tar.gz

  [root@centos7 opt]# mv filebeat-6.3.2-linux-x86_64 filebeat6.3.2

    配置filebeat .yml

      [root@centos7 opt]# vi filebeat6.3.2/filebeat.yml

      配置之后,内容如下

  1. ###################### Filebeat Configuration Example #########################
  2.  
  3. # This file is an example configuration file highlighting only the most common
  4. # options. The filebeat.reference.yml file from the same directory contains all the
  5. # supported options with more comments. You can use it as a reference.
  6. #
  7. # You can find the full configuration reference here:
  8. # https://www.elastic.co/guide/en/beats/filebeat/index.html
  9.  
  10. # For more available modules and options, please see the filebeat.reference.yml sample
  11. # configuration file.
  12.  
  13. #=========================== Filebeat inputs =============================
  14.  
  15. filebeat.inputs:
  16.  
  17. # Each - is an input. Most options can be set at the input level, so
  18. # you can use different inputs for various configurations.
  19. # Below are the input specific configurations.
  20.  
  21. - type: log
  22.  
  23. # Change to true to enable this input configuration.
  24. enabled: true
  25.  
  26. # Paths that should be crawled and fetched. Glob based paths.
  27. paths:
  28. - /log/*.log
  29. #- c:\programdata\elasticsearch\logs\*
  30.  
  31. # Exclude lines. A list of regular expressions to match. It drops the lines that are
  32. # matching any regular expression from the list.
  33. #exclude_lines: ['^DBG']
  34.  
  35. # Include lines. A list of regular expressions to match. It exports the lines that are
  36. # matching any regular expression from the list.
  37. #include_lines: ['^ERR', '^WARN']
  38.  
  39. # Exclude files. A list of regular expressions to match. Filebeat drops the files that
  40. # are matching any regular expression from the list. By default, no files are dropped.
  41. #exclude_files: ['.gz$']
  42.  
  43. # Optional additional fields. These fields can be freely picked
  44. # to add additional information to the crawled log files for filtering
  45. #fields:
  46. # level: debug
  47. # review: 1
  48.  
  49. ### Multiline options
  50.  
  51. # Mutiline can be used for log messages spanning multiple lines. This is common
  52. # for Java Stack Traces or C-Line Continuation
  53.  
  54. # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
  55. #multiline.pattern: ^\[
  56.  
  57. # Defines if the pattern set under pattern should be negated or not. Default is false.
  58. #multiline.negate: false
  59.  
  60. # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
  61. # that was (not) matched before or after or as long as a pattern is not matched based on negate.
  62. # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
  63. #multiline.match: after
  64.  
  65. #============================= Filebeat modules ===============================
  66.  
  67. filebeat.config.modules:
  68. # Glob pattern for configuration loading
  69. path: ${path.config}/modules.d/*.yml
  70.  
  71. # Set to true to enable config reloading
  72. reload.enabled: false
  73.  
  74. # Period on which files under path should be checked for changes
  75. #reload.period: 10s
  76.  
  77. #==================== Elasticsearch template setting ==========================
  78.  
  79. # setup.template.settings:
  80. # index.number_of_shards: 3
  81. #index.codec: best_compression
  82. #_source.enabled: false
  83.  
  84. #================================ General =====================================
  85.  
  86. # The name of the shipper that publishes the network data. It can be used to group
  87. # all the transactions sent by a single shipper in the web interface.
  88. #name:
  89.  
  90. # The tags of the shipper are included in their own field with each
  91. # transaction published.
  92. #tags: ["service-X", "web-tier"]
  93.  
  94. # Optional fields that you can specify to add additional information to the
  95. # output.
  96. #fields:
  97. # env: staging
  98.  
  99. #============================== Dashboards =====================================
  100. # These settings control loading the sample dashboards to the Kibana index. Loading
  101. # the dashboards is disabled by default and can be enabled either by setting the
  102. # options here, or by using the `-setup` CLI flag or the `setup` command.
  103. #setup.dashboards.enabled: false
  104.  
  105. # The URL from where to download the dashboards archive. By default this URL
  106. # has a value which is computed based on the Beat name and version. For released
  107. # versions, this URL points to the dashboard archive on the artifacts.elastic.co
  108. # website.
  109. #setup.dashboards.url:
  110.  
  111. #============================== Kibana =====================================
  112.  
  113. # Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
  114. # This requires a Kibana endpoint configuration.
  115. # setup.kibana:
  116.  
  117. # Kibana Host
  118. # Scheme and port can be left out and will be set to the default (http and 5601)
  119. # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  120. # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  121. #host: "localhost:5601"
  122.  
  123. #============================= Elastic Cloud ==================================
  124.  
  125. # These settings simplify using filebeat with the Elastic Cloud (https://cloud.elastic.co/).
  126.  
  127. # The cloud.id setting overwrites the `output.elasticsearch.hosts` and
  128. # `setup.kibana.host` options.
  129. # You can find the `cloud.id` in the Elastic Cloud web UI.
  130. #cloud.id:
  131.  
  132. # The cloud.auth setting overwrites the `output.elasticsearch.username` and
  133. # `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
  134. #cloud.auth:
  135.  
  136. #================================ Outputs =====================================
  137.  
  138. # Configure what output to use when sending the data collected by the beat.
  139.  
  140. #-------------------------- Elasticsearch output ------------------------------
  141. # output.elasticsearch:
  142. # Array of hosts to connect to.
  143. # hosts: ["localhost:9200"]
  144.  
  145. # Optional protocol and basic auth credentials.
  146. #protocol: "https"
  147. #username: "elastic"
  148. #password: "changeme"
  149.  
  150. #----------------------------- Logstash output --------------------------------
  151. output.logstash:
  152. # The Logstash hosts
  153. hosts: ["192.168.1.110:5044"]
  154.  
  155. # Optional SSL. By default is off.
  156. # List of root certificates for HTTPS server verifications
  157. #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
  158.  
  159. # Certificate for SSL client authentication
  160. #ssl.certificate: "/etc/pki/client/cert.pem"
  161.  
  162. # Client Certificate Key
  163. #ssl.key: "/etc/pki/client/cert.key"
  164.  
  165. #================================ Logging =====================================
  166.  
  167. # Sets log level. The default log level is info.
  168. # Available log levels are: error, warning, info, debug
  169. logging.level: info
  170.  
  171. # At debug level, you can selectively enable logging only for some components.
  172. # To enable all selectors use ["*"]. Examples of other selectors are "beat",
  173. # "publish", "service".
  174. #logging.selectors: ["*"]
  175.  
  176. #============================== Xpack Monitoring ===============================
  177. # filebeat can export internal metrics to a central Elasticsearch monitoring
  178. # cluster. This requires xpack monitoring to be enabled in Elasticsearch. The
  179. # reporting is disabled by default.
  180.  
  181. # Set to true to enable the monitoring reporter.
  182. #xpack.monitoring.enabled: false
  183.  
  184. # Uncomment to send the metrics to Elasticsearch. Most settings from the
  185. # Elasticsearch output are accepted here as well. Any setting that is not set is
  186. # automatically inherited from the Elasticsearch output configuration, so if you
  187. # have the Elasticsearch output configured, you can simply uncomment the
  188. # following line.
  189. #xpack.monitoring.elasticsearch:

      主要是配置filebeat.inputs,采集哪些日志;关闭output.elasticsearch,打开output.logstash,将收集到的信息推送到logstash。

    [root@centos7 opt]# ./filebeat6.3.2/filebeat -e -c ./filebeat6.3.2/filebeat.yml

      启动日志如下

  1. 2018-09-03T21:10:38.748+0800 INFO instance/beat.go:492 Home path: [/opt/filebeat6.3.2] Config path: [/opt/filebeat6.3.2] Data path: [/opt/filebeat6.3.2/data] Logs path: [/opt/filebeat6.3.2/logs]
  2. 2018-09-03T21:10:38.780+0800 INFO instance/beat.go:499 Beat UUID: 07d523d5-68ef-4470-a99d-5476bbc8535d
  3. 2018-09-03T21:10:38.780+0800 INFO [beat] instance/beat.go:716 Beat info {"system_info": {"beat": {"path": {"config": "/opt/filebeat6.3.2", "data": "/opt/filebeat6.3.2/data", "home": "/opt/filebeat6.3.2", "logs": "/opt/filebeat6.3.2/logs"}, "type": "filebeat", "uuid": "07d523d5-68ef-4470-a99d-5476bbc8535d"}}}
  4. 2018-09-03T21:10:38.781+0800 INFO [beat] instance/beat.go:725 Build info {"system_info": {"build": {"commit": "45a9a9e1561b6c540e94211ebe03d18abcacae55", "libbeat": "6.3.2", "time": "2018-07-20T04:18:19.000Z", "version": "6.3.2"}}}
  5. 2018-09-03T21:10:38.781+0800 INFO [beat] instance/beat.go:728 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":1,"version":"go1.9.4"}}}
  6. 2018-09-03T21:10:38.800+0800 INFO [beat] instance/beat.go:732 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2018-09-03T15:40:54+08:00","containerized":true,"hostname":"centos7","ips":["127.0.0.1/8","::1/128","192.168.1.111/24","fe80::3928:4541:b030:bea4/64"],"kernel_version":"3.10.0-862.el7.x86_64","mac_addresses":["08:00:27:e9:d7:da"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":5,"patch":1804,"codename":"Core"},"timezone":"CST","timezone_offset_sec":28800,"id":"acc3d28b9c824b55b6cdd5c8c2a46705"}}}
  7. 2018-09-03T21:10:38.803+0800 INFO [beat] instance/beat.go:761 Process info {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"ambient":null}, "cwd": "/opt", "exe": "/opt/filebeat6.3.2/filebeat", "name": "filebeat", "pid": 1579, "ppid": 1454, "seccomp": {"mode":"disabled"}, "start_time": "2018-09-03T21:10:37.710+0800"}}}
  8. 2018-09-03T21:10:38.803+0800 INFO instance/beat.go:225 Setup Beat: filebeat; Version: 6.3.2
  9. 2018-09-03T21:10:38.804+0800 INFO pipeline/module.go:81 Beat name: centos7
  10. 2018-09-03T21:10:38.816+0800 INFO instance/beat.go:315 filebeat start running.
  11. 2018-09-03T21:10:38.816+0800 INFO [monitoring] log/log.go:97 Starting metrics logging every 30s
  12. 2018-09-03T21:10:38.817+0800 INFO registrar/registrar.go:117 Loading registrar data from /opt/filebeat6.3.2/data/registry
  13. 2018-09-03T21:10:38.821+0800 INFO registrar/registrar.go:124 States Loaded from registrar: 1
  14. 2018-09-03T21:10:38.821+0800 WARN beater/filebeat.go:354 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
  15. 2018-09-03T21:10:38.821+0800 INFO crawler/crawler.go:48 Loading Inputs: 1
  16. 2018-09-03T21:10:38.822+0800 INFO log/input.go:118 Configured paths: [/log/*.log]
  17. 2018-09-03T21:10:38.822+0800 INFO input/input.go:88 Starting input of type: log; ID: 8294414020995878211
  18. 2018-09-03T21:10:38.866+0800 INFO crawler/crawler.go:82 Loading and starting Inputs completed. Enabled inputs: 1
  19. 2018-09-03T21:10:38.867+0800 INFO cfgfile/reload.go:122 Config reloader started
  20. 2018-09-03T21:10:38.867+0800 INFO cfgfile/reload.go:214 Loading of config files completed.
  21. 2018-09-03T21:10:38.883+0800 INFO log/harvester.go:228 Harvester started for file: /log/spring-boot-integrate.log.2018-08-21.log
  22. 2018-09-03T21:11:08.819+0800 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":50,"time":{"ms":54}},"total":{"ticks":70,"time":{"ms":83},"value":70},"user":{"ticks":20,"time":{"ms":29}}},"info":{"ephemeral_id":"faaf6d3e-8fff-4670-9dca-c51b48b134c8","uptime":{"ms":30102}},"memstats":{"gc_next":5931008,"memory_alloc":3006968,"memory_total":4960192,"rss":15585280}},"filebeat":{"events":{"added":93,"done":93},"harvester":{"open_files":1,"running":1,"started":1}},"libbeat":{"config":{"module":{"running":0},"reloads":1},"output":{"events":{"acked":91,"batches":1,"total":91},"read":{"bytes":6},"type":"logstash","write":{"bytes":5990}},"pipeline":{"clients":1,"events":{"active":0,"filtered":2,"published":91,"retry":91,"total":93},"queue":{"acked":91}}},"registrar":{"states":{"current":1,"update":93},"writes":{"success":3,"total":3}},"system":{"cpu":{"cores":1},"load":{"1":0.05,"15":0.05,"5":0.03,"norm":{"1":0.05,"15":0.05,"5":0.03}}}}}}

    采集的是/log下的所有的log文件,我们就用工程:spring-boot-integrate来产生log文件(对应的就是我们的项目产生的日志文件);spring-boot-integrate依赖127.0.0.1的redis和mysql,需要把redis和mysql启动起来(注意我的redis是配置了密码的,mysql用的数据库是spring-boot,sql文件在工程中)。

    [root@centos7 redis-3.2.12]# cd /usr/local/redis-3.2.12/

    [root@centos7 redis-3.2.12]# ./src/redis-server redis.conf 

    [root@centos7 local]# service mysqld start

    启动我们的spring-boot-integrate

      用maven生成war包,将spring-boot-integrate.war包拷贝到tomcat的webapps,启动tomcat即可;注意tomcat版本需要8及以上;

      [root@centos7 opt]# cd /usr/local/apache-tomcat-8.5.33/

      [root@centos7 apache-tomcat-8.5.33]# ./bin/startup.sh

    访问:http://192.168.1.111:8080/spring-boot-integrate,如下图

    说明应用启动成功,我们可以参照:spring-boot-2.0.3不一样系列之shiro - 搭建篇,访问应用,多产生一些日志数据。

效果

  最终数据到kibana进行可视化展示,我们看看我们刚刚的日志在kibana中的展示情况

总结

  架构图

    一般而言,架构图如下

    由nginx对外暴露访问接口,并提供负载均衡功能。本文中没有集成nginx,大家可以自己去实现,集成nginx也不难。

    另外也没有集成消息中间件

    这种架构适合于日志规模比较庞大的情况。但由于 Logstash 日志解析节点和 Elasticsearch 的负荷比较重,可将他们配置为集群模式,以分担负荷。引入消息队列,均衡了网络传输,从而降低了网络闭塞,尤其是丢失数据的可能性,但依然存在 Logstash 占用系统资源过多的问题。

  2、基于docker的搭建

    ELK版本迭代非常快,如果能基于docker做成镜像,基于docker搭建,既方便ELK的统一搭建、也方便ELK的升级;有兴趣的小伙伴可以试着搭建。

  3、组件组合

    本文只是简单的实现了ELK + Filebeat各个组件都是单节点的集成,相当于搭建了最基础版本;当然有了这个基础版本,再搭建某些组件的集群版本也不难了。

    另外,组件之间是可以灵活组合的,有些组件也不是必须的,我们可以根据我们业务量的需求来搭建合适的日志系统。

  4、组件详情

    本文只是讲elk+filebeat的搭建,各个组件的详情没有具体介绍,需要大家自己去了解了;各个组件的内容还是挺多的,更好的了解各个组件,对搭建高性能的日志系统有很大帮助。

参考

  谈日志的重要性】运维中被低估的日志

  集中式日志系统 ELK 协议栈详解

  ELK+Filebeat 集中式日志解决方案详解

  从零开始搭建ELK+GPE监控预警系统

elk + filebeat,6.3.2版本简单搭建,实现我们自己的集中式日志系统的更多相关文章

  1. 集中式日志系统 ELK 协议栈详解

    简介 在我们日常生活中,我们经常需要回顾以前发生的一些事情:或者,当出现了一些问题的时候,可以从某些地方去查找原因,寻找发生问题的痕迹.无可避免需要用到文字的.图像的等等不同形式的记录.用计算机的术语 ...

  2. Centos7搭建集中式日志系统

    在CentOS7中,Rsyslong是一个集中式的日志收集系统,可以运行在TCP或者UDP的514端口上.   目录 开始之前 配置接收日志的主机 配置发送日志的主机 日志回滚 附件:创建日志接收模板 ...

  3. ELK——集中式日志系统

    https://www.ibm.com/developerworks/cn/opensource/os-cn-elk/index.html 基本流程是 Shipper 负责从各种数据源里采集数据,然后 ...

  4. ELK+Filebeat 集中式日志解决方案详解

    链接:https://www.ibm.com/developerworks/cn/opensource/os-cn-elk-filebeat/index.html?ca=drs- ELK Stack ...

  5. 中小型研发团队架构实践七:集中式日志ELK

    一.集中式日志 日志可分为系统日志.应用日志以及业务日志,系统日志给运维人员使用,应用日志给研发人员使用,业务日志给业务操作人员使用.我们这里主要讲解应用日志,通过应用日志来了解应用的信息和状态,以及 ...

  6. 集中式日志分析平台 - ELK Stack - 安全解决方案 X-Pack

    大数据之心 关注  0.6 2017.02.22 15:36* 字数 2158 阅读 16457评论 7喜欢 9 简介 X-Pack 已经作为 Elastic 公司单独的产品线,前身是 Shield, ...

  7. 安装logstash+kibana+elasticsearch+redis搭建集中式日志分析平台

    安装logstash+kibana+elasticsearch+redis搭建集中式日志分析平台 2014-01-16 19:40:57|  分类: logstash |  标签:logstash   ...

  8. 搭建Loki、Promtail、Grafana轻量级日志系统(centos7)

    搭建Loki.Promtail.Grafana轻量级日志系统(centos7)--简称PLG 需求 公司项目采用微服务的架构,服务很多,每个服务都有自己的日志,分别存放在不同的服务器上.当查找日志时需 ...

  9. Redhat/CentOS7-环境虚拟机简单搭建Nginx+Tomcat负载均衡集群

    Tomcat服务器是一个免费的开放源代码的web应用服务器,属于轻量级应用服务器,是开发和调试JSP程序的首选.由于Tomcat处理静态HTML的能力运不及Apache或者Nginx,所以Tomcat ...

随机推荐

  1. NC 自定义参照类

    package nc.ui.hzctr.costctr.ref; import nc.ui.bd.ref.AbstractRefModel; import nc.vo.pubapp.pattern.p ...

  2. Java之IO流总结

    IO流·Java流式输入/输出原理·Java流类的分类·输入/输出流类·常见的节点流和处理流·文件流·缓冲流·转换流·数据流·Print流·Object流 ①Java流式输入/输出原理         ...

  3. SLICK基础

    1.sbt添加依赖 "com.typesafe.slick" %% "slick" % "3.2.3", "org.slf4j&q ...

  4. activeMq-1 快速入门

    Activemq 是一款开源的消息中间件,适合中小型应用使用,遵循JMS规范. 具体介绍这里就不再阐述了,这里简单说下消息中间件的好处 1请求结果异步处理 客户端发送请求以后,服务器可以把相关数据放到 ...

  5. 02-jQuery的选择器

    我们以前在CSS中学习的选择器有: 今天来学习一下jQuery 选择器. jQuery选择器是jQuery强大的体现,它提供了一组方法,让我们更加方便的获取到页面中的元素. 1.jQuery 的基本选 ...

  6. (转)web前端知识精简

    Web前端技术由 html.css 和 javascript 三大部分构成,是一个庞大而复杂的技术体系,其复杂程度不低于任何一门后端语言.而我们在学习它的时候往往是先从某一个点切入,然后不断地接触和学 ...

  7. Android Studio导入项目一直卡在Building gradle project info的解决方案

    出现了一个很神奇的现象,Android Studio导入其它项目均正常,但是导入某个项目(两天前还正常打开的项目)却一直卡在Building gradle project info 尝试了重启Andr ...

  8. JavaScript笔记基础版

    1.JavaScript数组: 创建数组  var cars = new Array(); 数组的初始化: 1.1 cars[0]="Saab"; cars[1]="Vo ...

  9. 库函数wordcloud安装的问题

    在对python有了一定的了解之后就对python的第三方库产生了十分浓厚的兴趣,因为python的很多功能都是要依靠第三方库函数来实现的,而且在计算机二级刚刚加入的python考试中也有对第三方库的 ...

  10. H5+.Net Webapi集成微信分享前后端代码 微信JS-SDK wx.onMenuShareTimeline wx.onMenuShareAppMessage

    说明: 1/因为赚麻烦这里没有使用数据库或服务器缓存来存储access_token和jsapi_ticket,为了方便这里使用了本地的xml进行持久化这两个值以及这两个值的创建时间和有限期限. 2/每 ...