一、ELK体系结构

二、系统环境变量

【主机信息】

  1. IP                           主机名                   操作系统版本
  2. 10.10.10.102                 console                 CentOS7.5
  3. 10.10.10.103                 log1                    CentOS7.5
    10.10.10.104                 log2                    CentOS7.5

【软件包版本信息】

  1. elasticsearch-6.4..tar.gz
  2. logstash-6.4..tar.gz
  3. kibana-6.4.-linux-x86_64.tar.gz
    node-v8.11.4-linux-x64.tar.gz
    elasticsearch-head-master.zip

1. 设置主机名和IP映射

分别在上述三台机器的/etc/hosts文件中追加如下内容：

  1. 10.10.10.102 console
  2. 10.10.10.103 log1
  3. 10.10.10.104 log2

2.关于3台机器的防火墙，并设置开机不启动

  1. #关闭防火墙
  2. systemctl stop firewalld
  3.  
  4. #设置防火墙开机不启动
  5. systemctl disable firewalld

3.修改3台机器的系统文件描述符大小

  1. vim /etc/security/limits.conf
  2.  
  3. es         -            nofile

4.增大3台机器的虚拟内存mmap count配置

  1. vim  /etc/sysctl.conf
    vm.max_map_count = 
  1. #使修改生效
  2. sysctl -p

5.在3台机器上分别新建用户es和日志文件目录

  1. useradd es
    mkdir  /esdata
    chown -R es:es /esdata

6.在3台机器上都安装JDK1.8

三、Elasticsearch的安装与配置

1.分别在10.10.10.102、10.10.10.103、10.10.10.104机器上新建Elasticsearch安装目录并修改属主用户和组

  1. mkdir -p  /usr/local/elasticsearch-6.4.
  2. chown -R es:es /usr/local/elasticsearch-6.4.

2.登录10.10.10.102机器并切换到es用户，将elasticsearch-6.4.0.tar.gz解压到 /usr/local/elasticsearch-6.4.0目录下

  1. tar -xf  /home/es/elasticsearch-6.4..tar.gz
  2. cp -r *  /usr/local/elasticsearch-6.4.

3.修改配置文件

console配置文件如下：

  1. [es@console config]$ cat /usr/local/elasticsearch-6.4.0/config/elasticsearch.yml
  2. # ======================== Elasticsearch Configuration =========================
  3. #
  4. # NOTE: Elasticsearch comes with reasonable defaults for most settings.
  5. #       Before you set out to tweak and tune the configuration, make sure you
  6. #       understand what are you trying to accomplish and the consequences.
  7. #
  8. # The primary way of configuring a node is via this file. This template lists
  9. # the most important settings you may want to configure for a production cluster.
  10. #
  11. # Please consult the documentation for further information on configuration options:
  12. # https://www.elastic.co/guide/en/elasticsearch/reference/index.html
  13. #
  14. # ---------------------------------- Cluster -----------------------------------
  15. #
  16. # Use a descriptive name for your cluster:
  17. #
  18. cluster.name: console                                #设置集群的名称为console
  19. #
  20. # ------------------------------------ Node ------------------------------------
  21. #
  22. # Use a descriptive name for the node:
  23. #
  24. node.name: console                                   #设置集群节点名称为console
  25. node.master: true                                    #设置该节点是否为主节点，这里选择true，其他2台机器这里设置为false
  26. #
  27. # Add custom attributes to the node:
  28. #
  29. #node.attr.rack: r1
  30. #
  31. # ----------------------------------- Paths ------------------------------------
  32. #
  33. # Path to directory where to store the data (separate multiple locations by comma):
  34. #
  35. path.data: /esdata                                   #设置数据目录为/esdata
  36. #
  37. # Path to log files:
  38. #
  39. #path.logs: /path/to/logs
  40. #
  41. # ----------------------------------- Memory -----------------------------------
  42. #
  43. # Lock the memory on startup:
  44. #
  45. #bootstrap.memory_lock: true
  46. #
  47. #bootstrap.mlockall: true
  48. #
  49. # Make sure that the heap size is set to about half the memory available
  50. # on the system and that the owner of the process is allowed to use this
  51. # limit.
  52. #
  53. # Elasticsearch performs poorly when the system is swapping the memory.
  54. #
  55. # ---------------------------------- Network -----------------------------------
  56. #
  57. # Set the bind address to a specific IP (IPv4 or IPv6):
  58. #
  59. network.host: 10.10.10.102                     #这里配置的是console机器的IP，其他2台机器分别配置自己的IP
  60. network.bind_host: 10.10.10.102                #同上
  61. network.publish_host: 10.10.10.102             #同上
  62.  
  63. #
  64. # Set a custom port for HTTP:
  65. #
  66. http.port: 9200                                #开启端口
  67. #
  68. # For more information, consult the network module documentation.
  69. #
  70. # --------------------------------- Discovery ----------------------------------
  71. #
  72. # Pass an initial list of hosts to perform discovery when new node is started:
  73. # The default list of hosts is ["127.0.0.1", "[::1]"]
  74. #
  75. discovery.zen.ping.unicast.hosts: ["10.10.10.102:9300"]              #配置自动发现机制，其他2台机器也设置这个值
  76. #
  77. # Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes /  + ):
  78. #
  79. discovery.zen.minimum_master_nodes: 1                                #设置发现的主节点个数为1
  80. #
  81. # For more information, consult the zen discovery module documentation.
  82. #
  83. # ---------------------------------- Gateway -----------------------------------
  84. #
  85. # Block initial recovery after a full cluster restart until N nodes are started:
  86. #
  87. #gateway.recover_after_nodes:
  88. #
  89. # For more information, consult the gateway module documentation.
  90. #
  91. # ---------------------------------- Various -----------------------------------
  92. #
  93. # Require explicit names when deleting indices:
  94. #
  95. #action.destructive_requires_name: true

log1配置文件：

  1. [es@log1 config]$ cat elasticsearch.yml
  2. # ======================== Elasticsearch Configuration =========================
  3. #
  4. # NOTE: Elasticsearch comes with reasonable defaults for most settings.
  5. #       Before you set out to tweak and tune the configuration, make sure you
  6. #       understand what are you trying to accomplish and the consequences.
  7. #
  8. # The primary way of configuring a node is via this file. This template lists
  9. # the most important settings you may want to configure for a production cluster.
  10. #
  11. # Please consult the documentation for further information on configuration options:
  12. # https://www.elastic.co/guide/en/elasticsearch/reference/index.html
  13. #
  14. # ---------------------------------- Cluster -----------------------------------
  15. #
  16. # Use a descriptive name for your cluster:
  17. #
  18. cluster.name: console
  19. #
  20. # ------------------------------------ Node ------------------------------------
  21. #
  22. # Use a descriptive name for the node:
  23. #
  24. node.name: log1
  25. node.master: false
  26. #
  27. # Add custom attributes to the node:
  28. #
  29. #node.attr.rack: r1
  30. #
  31. # ----------------------------------- Paths ------------------------------------
  32. #
  33. # Path to directory where to store the data (separate multiple locations by comma):
  34. #
  35. path.data: /esdata
  36. #
  37. # Path to log files:
  38. #
  39. #path.logs: /path/to/logs
  40. #
  41. # ----------------------------------- Memory -----------------------------------
  42. #
  43. # Lock the memory on startup:
  44. #
  45. #bootstrap.memory_lock: true
  46. #
  47. #bootstrap.mlockall: true
  48. #
  49. # Make sure that the heap size is set to about half the memory available
  50. # on the system and that the owner of the process is allowed to use this
  51. # limit.
  52. #
  53. # Elasticsearch performs poorly when the system is swapping the memory.
  54. #
  55. # ---------------------------------- Network -----------------------------------
  56. #
  57. # Set the bind address to a specific IP (IPv4 or IPv6):
  58. #
  59. network.host: 10.10.10.103
  60. network.bind_host: 10.10.10.103
  61. network.publish_host: 10.10.10.103
  62.  
  63. #
  64. # Set a custom port for HTTP:
  65. #
  66. http.port:
  67. #
  68. # For more information, consult the network module documentation.
  69. #
  70. # --------------------------------- Discovery ----------------------------------
  71. #
  72. # Pass an initial list of hosts to perform discovery when new node is started:
  73. # The default list of hosts is ["127.0.0.1", "[::1]"]
  74. #
  75. discovery.zen.ping.unicast.hosts: ["10.10.10.102:9300"]
  76. #
  77. # Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes /  + ):
  78. #
  79. discovery.zen.minimum_master_nodes:
  80. #
  81. # For more information, consult the zen discovery module documentation.
  82. #
  83. # ---------------------------------- Gateway -----------------------------------
  84. #
  85. # Block initial recovery after a full cluster restart until N nodes are started:
  86. #
  87. #gateway.recover_after_nodes:
  88. #
  89. # For more information, consult the gateway module documentation.
  90. #
  91. # ---------------------------------- Various -----------------------------------
  92. #
  93. # Require explicit names when deleting indices:
  94. #
  95. #action.destructive_requires_name: true

log2配置文件：

  1. [es@log2 config]$ cat elasticsearch.yml
  2. # ======================== Elasticsearch Configuration =========================
  3. #
  4. # NOTE: Elasticsearch comes with reasonable defaults for most settings.
  5. #       Before you set out to tweak and tune the configuration, make sure you
  6. #       understand what are you trying to accomplish and the consequences.
  7. #
  8. # The primary way of configuring a node is via this file. This template lists
  9. # the most important settings you may want to configure for a production cluster.
  10. #
  11. # Please consult the documentation for further information on configuration options:
  12. # https://www.elastic.co/guide/en/elasticsearch/reference/index.html
  13. #
  14. # ---------------------------------- Cluster -----------------------------------
  15. #
  16. # Use a descriptive name for your cluster:
  17. #
  18. cluster.name: console
  19. #
  20. # ------------------------------------ Node ------------------------------------
  21. #
  22. # Use a descriptive name for the node:
  23. #
  24. node.name: log2
  25. node.master: false
  26. #
  27. # Add custom attributes to the node:
  28. #
  29. #node.attr.rack: r1
  30. #
  31. # ----------------------------------- Paths ------------------------------------
  32. #
  33. # Path to directory where to store the data (separate multiple locations by comma):
  34. #
  35. path.data: /esdata
  36. #
  37. # Path to log files:
  38. #
  39. #path.logs: /path/to/logs
  40. #
  41. # ----------------------------------- Memory -----------------------------------
  42. #
  43. # Lock the memory on startup:
  44. #
  45. #bootstrap.memory_lock: true
  46. #
  47. #bootstrap.mlockall: true
  48. #
  49. # Make sure that the heap size is set to about half the memory available
  50. # on the system and that the owner of the process is allowed to use this
  51. # limit.
  52. #
  53. # Elasticsearch performs poorly when the system is swapping the memory.
  54. #
  55. # ---------------------------------- Network -----------------------------------
  56. #
  57. # Set the bind address to a specific IP (IPv4 or IPv6):
  58. #
  59. network.host: 10.10.10.104
  60. network.bind_host: 10.10.10.104
  61. network.publish_host: 10.10.10.104
  62.  
  63. #
  64. # Set a custom port for HTTP:
  65. #
  66. http.port:
  67. #
  68. # For more information, consult the network module documentation.
  69. #
  70. # --------------------------------- Discovery ----------------------------------
  71. #
  72. # Pass an initial list of hosts to perform discovery when new node is started:
  73. # The default list of hosts is ["127.0.0.1", "[::1]"]
  74. #
  75. discovery.zen.ping.unicast.hosts: ["10.10.10.102:9300"]
  76. #
  77. # Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes /  + ):
  78. #
  79. discovery.zen.minimum_master_nodes:
  80. #
  81. # For more information, consult the zen discovery module documentation.
  82. #
  83. # ---------------------------------- Gateway -----------------------------------
  84. #
  85. # Block initial recovery after a full cluster restart until N nodes are started:
  86. #
  87. #gateway.recover_after_nodes:
  88. #
  89. # For more information, consult the gateway module documentation.
  90. #
  91. # ---------------------------------- Various -----------------------------------
  92. #
  93. # Require explicit names when deleting indices:
  94. #
  95. #action.destructive_requires_name: true

 

4.后台启动Elasticsearch

  1. /usr/local/elasticsearch-6.4./bin/elasticsearch -d

启动后显示如下：

5. 安装ElasticSearch head插件

由于ElasticSearch的界面展示的是Json文件，不是很友好。我们可以通过安装插件来解决它。

ElasticSearch_head 下载地址：https://github.com/troub1emaker0911/elasticsearch-head

ElasticSearch_head 需要node.js的支持。我们需要首先安装node.js

【安装node.js】

首先切换到root用户下，将node.js的安装包上传到console机器上。

#将node.js解压到目录/usr/local/node-v8.11.4
tar -xf node-v8.11.4-linux-x64.tar.xz -C /usr/local/node-v8.11.4

#设置符号链接
ln -s /usr/local/node-v8.11.4/bin/node   /usr/local/bin/
ln -s /usr/local/node-v8.11.4/bin/npm    /usr/local/bin/

#检查是否配置成功
node -v
npm -v

【安装ElasticSearch_head插件】

切换到es用户，将安装包上传到console机器上。

  1. #解压文件
  2. unzip  elasticsearch-head-master.zip
  3.  
  4. #将文件包移动到目录/usr/local下
  5. mv  elasticsearch-head-master   /usr/local
  6.  
  7. cd  /usr/local/elasticsearch-head-master
    npm install
    #启动Elasticsearch-head-master
    npm run start  > /dev/null 2>&1  & 

执行上述步骤完成后，在浏览器中输入http://10.10.10.102:9100即可显示如下界面。

但是这样集群健康值是不可用的（截图中是我已经配置完毕的），我们需要在console机器的elasticsearch.yml文件中追加如下配置：

  1. vim  /usr/local/elasticsearch-6.4./config/elasticsearch.yml
  2. http.cors.enabled: true
  3. http.cors.allow-origin: "*"

然后修改“连接”按钮前的地址，将原来的http://localhost:9200/修改为console的地址，即http://10.10.10.102:9200，然后点击“连接”，此时后面的“集群健康值”就变成green了。

6. 新建索引

切换到“索引”选项卡，点击“新建索引”，这里填写“索引名称”为book.

然后点击“概览”，就可以看到刚才新建的索引。

注意上图中的绿色块。有粗边框的为主，细边框的为备。

7.安装插件：中文分词器ik

elasticsearch-analysis-ik 是一款中文的分词插件，支持自定义词库。项目地址为：https://github.com/medcl/elasticsearch-analysis-ik

(1)安装Maven

由于该项目使用了Maven来管理，源代码放到github上。所以要先在服务器上面安装Maven，便可以直接在服务器上面生成项目jar包，部署起来更加方便了。

  1. yum install -y maven

(2)安装分词器ik

这里安装的版本是6.3.0

  1. git clone https://github.com/medcl/elasticsearch-analysis-ik.git
  2. [es@console ~]$ cd elasticsearch-analysis-ik/
  3. [es@console elasticsearch-analysis-ik]$ mvn package

(3)复制和解压

  1. [es@console elasticsearch-analysis-ik]$ mkdir -p /usr/local/elasticsearch/plugins/ik
  2. [es@console elasticsearch-analysis-ik]$ cp target/releases/elasticsearch-analysis-ik-6.3.0.zip /usr/local/elasticsearch/plugins/ik
  3. [es@console ~]$ cd /usr/local/elasticsearch/plugins/ik/
  4. [es@console ik]$ unzip -oq elasticsearch-analysis-ik-6.3.0.zip

(4)重启Elasticsearch

  1. [es@console ik]$ cd /usr/local/elasticsearch/bin/
  2. [es@console bin]$ jps
  3.  Jps
  4.  Elasticsearch
  5. [es@console bin]$ kill -
  6. [es@console elasticsearch]$ bin/elasticsearch -d

注：在浏览器输入如下地址可以查看集群的nodes节点，但结果是json格式，不是很易读，可以将其格式化。

  1. http://10.10.10.102:9200/_nodes

四、Logstash的安装与配置

Logstash 是一款强大的数据处理工具，它可以实现数据传输，格式处理，格式化输出，还有强大的插件功能，常用于日志处理。

Logstash工作的三个阶段：

1.安装Logstash

  1. #切换到es用户下，解压安装包到指定目录下
    tar -xf  logstash-6.4.0.tar.gz  -C  /usr/local/logstash-6.4.0

至此，Logstash安装完成

2.Logstash简介

Logstash是一个开源的、接受来自多种数据源（input）、过滤你想要的数据（filter）、存储到其他设备的日志管理程序。Logstash包含三个基本插件input\filter\output，一个基本的logstash服务必须包含input和output.

Logstash如何工作：

Logstash数据处理有三个阶段，input–>filter–>output.input生产数据，filter根据定义的规则修改数据，output将数据输出到你定义的存储位置。

Inputs:

数据生产商，包含以下几个常用输出：

  • file: 从文件系统中读取文件，类似使用tail -0F

  • syslog: syslog服务，监听在514端口使用RFC3164格式

  • redis:  从redis服务读取，使用redis管道和列表。

  • beats: 一种代理，自己负责收集好数据然后转发给Logstash，常用的如filebeat.

Filters:

filters相当一个加工管道，它会一条一条过滤数据根据你定义的规则，常用的filters如下：

  • grok:  解析无规则的文字并转化为有结构的格式。

  • mutate: 丰富的基础类型处理，包括类型转换、字符串处理、字段处理等。

  • drop: 丢弃一部分events不进行处理，例如： debug events

  • clone: 负责一个event,这个过程中可以添加或删除字段。

  • geoip: 添加地理信息（为前台kibana图形化展示使用）

Outputs:

  • elasticserache elasticserache接收并保存数据，并将数据给kibana前端展示。

  • output 标准输出，直接打印在屏幕上。

3.Logstash举例

  1. bin/logstash -e 'input { stdin { } } output { stdout {} }'

我们现在可以在命令行下输入一些字符，然后我们将看到logstash的输出内容：

  1. [es@console logstash-6.4.]$ bin/logstash -e 'input { stdin { } } output { stdout {} }'
  2. hello world
  3. Sending Logstash logs to /usr/local/logstash-6.4./logs which is now configured via log4j2.properties
  4. [--14T22::,][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
  5. [--14T22::,][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.4.0"}
  6. [--14T22::,][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>, "pipeline.batch.size"=>, "pipeline.batch.delay"=>}
  7. [--14T22::,][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x3f32496 run>"}
  8. The stdin plugin is now waiting for input:
  9. [--14T22::,][INFO ][logstash.agent           ] Pipelines running {:count=>, :running_pipelines=>[:main], :non_running_pipelines=>[]}
  10. {
  11.       "@version" => "",
  12.        "message" => "hello world",
  13.     "@timestamp" => --14T14::.245Z,
  14.           "host" => "console"
  15. }
  16. [--14T22::,][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>}

我们再运行另一个命令：

  1. bin/logstash -e 'input { stdin { } } output { stdout { codec => rubydebug } }'

然后输入helloworld，查看显示的内容：

  1. [es@console logstash-6.4.]$ bin/logstash -e 'input { stdin { } } output { stdout { codec => rubydebug } }'
  2. helloworld
  3. Sending Logstash logs to /usr/local/logstash-6.4./logs which is now configured via log4j2.properties
  4. [--12T03::,][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
  5. [--12T03::,][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.4.0"}
  6. [--12T03::,][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>, "pipeline.batch.size"=>, "pipeline.batch.delay"=>}
  7. [--12T03::,][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x7cefe25 run>"}
  8. The stdin plugin is now waiting for input:
  9. [--12T03::,][INFO ][logstash.agent           ] Pipelines running {:count=>, :running_pipelines=>[:main], :non_running_pipelines=>[]}
  10. {
  11.           "host" => "console",
  12.       "@version" => "",
  13.     "@timestamp" => --11T19::.813Z,
  14.        "message" => "helloworld"
  15. }
  16. [--12T03::,][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>}

以上示例通过重新设置了叫”stdout”的output(添加了”codec”参数)，我们就可以改变Logstash的输出表现。类似的我们可以通过在你的配置文件中添加或者修改inputs、outputs、filters，就可以使随意的格式化日志数据成为可能，从而订制更合理的存储格式为查询提供便利。

前面已经说过Logstash必须有一个输入和一个输出，上面的例子表示从终端上输入并输出到终端。

数据在线程之间以事件的形式流传。不要叫行，因为Logstash可以处理多行事件。

input {

# 输入域，可以使用上面提到的几种输入方式。stdin{} 表示标准输入，file{} 表示从文件读取。

input的各种插件：     https://www.elastic.co/guide/en/logstash/current/input-plugins.html

}

output {

#Logstash的功能就是对数据进行加工，上述例子就是Logstash的格式化输出，当然这是最简单的。

output的各种插件：https://www.elastic.co/guide/en/logstash/current/output-plugins.html

}

Logstash配置文件和命令：

Logstash的默认配置已经够我们使用了，从5.0后使用logstash.yml文件，可以将一些命令行参数直接写到YAML文件即可。

  • –configtest 或 -t    用于测试Logstash的配置语法是否正确，非常好的一个参数。

  • –log 或 -l Logstash默认输出日志到标准输出，指定Logstash的日志存放位置

  • –pipeline-workers 或 -w  指定运行filter和output的pipeline线程数量，使用默认就好。

  • -f 指定规则文件，可以将自己的文件放在同一个路径下，使用-f 即可运行。

一个简单的Logstash从文件中读取配置:

  1. vim file.conf  #file.conf可以放在任意位置
  2. input {
  3.          stdin {
  4.          }
  5. }
  6. output {
  7.          stdout {
  8.                  codec=>rubydebug
  9.          }
  10. }
  11. ~
  12. bin/logstash -f /root/conf/file.conf  #启动即可

3. 插件

(1)grok插件

Grok是Logstash最重要的插件，你可以在grok里自定义好命名规则，然后在grok参数或者其他正则表达式中引用它。

官方给出了120个左右默认的模式：https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns

  1. USERNAME [a-zA-Z0-._-]+
  2. USER %{USERNAME}

第一行，用普通的正则表达式来定义一个grok表达式；第二行，通过打印赋值格式，用前面定义好的grok表达式来定义里一个grok表达式。

正则表达式引格式：

  1. %{SYNTAX:SEMANTIC}

  • SYNTAX：表示你的规则是如何被匹配的，比如3.14将会被NUMBER模式匹配，55.1.1.2将会被IP模式匹配。

  • SEMANTIC：表示被匹配到的唯一标识符，比如3.14被匹配到了后，SEMANTIC就当是3.14。

匹配到的数据默认是strings类型，当然你也可以装换你匹配到的数据，类似这样：

  1. %{NUMBER:num:int}

当前只支持装换为int和float。

例如：

  1. [es@console config]$ more file.conf
  2. input
  3.     {
  4.      stdin {
  5.      }
  6. }
  7. filter {
  8.      grok {
  9.        match => {
  10.           "message" => "%{WORD} %{NUMBER:request_time:float} %{WORD}"
  11.        }
  12.      }
  13. }
  14. output {
  15.      stdout {
  16.              codec=>rubydebug
  17.      }
  18. }

然后运行logstash

  1. [es@console logstash-6.4.]$ bin/logstash -f /usr/local/logstash-6.4./config/file.conf

结果如下：

  1. monkey 12.12 beta
  2. {
  3.          "message" => "monkey 12.12 beta",
  4.         "@version" => "",
  5.       "@timestamp" => --17T08::.416Z,
  6.             "host" => "console",
  7.     "request_time" => 12.12
  8. }

这个我们就匹配到我们想要的值了，并将名字命名为：request_time

在实际生产中为了方便我们不可能在配置文件中一行一行的写表达式，建议把所有的grok表达式统一写到一个地方，使用patterns_dir选项来引用。

  1. grok {
  2.         patterns_dir => "/root/conf/nginx"  #这是你定义的grok表达式文件
  3.         match => { "message" => "%{CDN_FORMAT}" }
  4.         add_tag => ["CDN"]
  5.  }

事实上，我们收集的日志也有很多不需要的地方，我们可以删除一部分field信息，保留我们想要的那一部分。

  1. grok {
  2.        match => {
  3.                 "message" => "%{WORD} %{NUMBER:request_time:float} %{WORD}"
  4.         }
  5.         remove_field => [ "request_time" ]
  6.         overwrite => [ "message" ]
  7.         }
  8. as  as
  9. {
  10.     "@timestamp" => --08T06::.921Z,
  11.       "@version" => "",
  12.           "host" => "0.0.0.0",
  13.        "message" => "as 12 as"
  14. }

已经没有request_time这个field啦~

更多关于grok的用户看官方文档：https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html

最重要的一点：我强烈建议每个人都要使用 Grok Debugger 来调试自己的 grok 表达式。

(2)kv插件

(3)geoip插件

geoip主要是查询IP地址归属地，用来判断访问网站的来源地。

  1. [es@console config]$ more file.conf
  2. input
  3.     {
  4.      stdin {
  5.      }
  6. }
  7. filter {
  8.      grok {
  9.        match => {
  10.           "message" => "%{WORD} %{NUMBER:request_time:float} %{WORD}"
  11.        }
  12.      }
  13.      geoip {
  14.        source => "clientip"
  15.        fields => [ "ip","city_name","country_name","location" ]
  16.      }
  17. }
  18. output {
  19.      stdout {
  20.              codec=>rubydebug
  21.      }
  22. }

参考文档：https://www.cnblogs.com/blogjun/articles/8064646.html

四、Kibana的安装与配置

Kibana是一个开源的分析与可视化平台，设计出来用于和Elasticsearch一起使用的。你可以用kibana搜索、查看、交互存放在Elasticsearch索引里的数据，

使用各种不同的图表、表格、地图等kibana能够很轻易地展示高级数据分析与可视化。

Kibana让我们理解大量数据变得很容易。它简单、基于浏览器的接口使你能快速创建和分享实时展现Elasticsearch查询变化的动态仪表盘。

# 简单来讲他具体的工作流程就是 logstash agent 监控并过滤日志，logstash index将日志收集在一起交给全文搜索服务ElasticSearch 可以用ElasticSearch进行自定义搜索 通过Kibana 来结合 自定义搜索进行页面展示，如上图。

1.安装Kibana

#新建安装目录
mkdir -p /usr/local/kibana-6.4.0

#解压安装包并将解压后的复制到相应目录下
tar -xf kibana-6.4.0.tar.gz

#修改安装目录的属主和用户
cp -r * /root/software/kibana-6.4.0 /usr/local/kibana-6.4.0

2.配置Kibana与启动

修改kibana的配置文件kibana.yml, 配置后的结果如下：

  1. [root@console config]# more /usr/local/kibana-6.4.0/config/kibana.yml
  2. # Kibana is served by a back end server. This setting specifies the port to use.
  3. server.port: 5601                      #配置kibana的端口号
  4.  
  5. # Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
  6. # The default is 'localhost', which usually means remote machines will not be able to connect.
  7. # To allow connections from remote users, set this parameter to a non-loopback address.
  8. server.host: "10.10.10.102"            #配置kibana安装的主机的IP
  9.  
  10. # Enables you to specify a path to mount Kibana at if you are running behind a proxy.
  11. # Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
  12. # from requests it receives, and to prevent a deprecation warning at startup.
  13. # This setting cannot end in a slash.
  14. #server.basePath: ""
  15.  
  16. # Specifies whether Kibana should rewrite requests that are prefixed with
  17. # `server.basePath` or require that they are rewritten by your reverse proxy.
  18. # This setting was effectively always `false` before Kibana 6.3 and will
  19. # default to `true` starting in Kibana 7.0.
  20. #server.rewriteBasePath: false
  21.  
  22. # The maximum payload size in bytes for incoming server requests.
  23. #server.maxPayloadBytes: 
  24.  
  25. # The Kibana server's name.  This is used for display purposes.
  26. server.name: "console"
  27.  
  28. # The URL of the Elasticsearch instance to use for all your queries.
  29. elasticsearch.url: "http://10.10.10.102:9200"   #配置Elasticsearch安装主机的IP地址和端口
  30.  
  31. # When this setting's value is true Kibana uses the hostname specified in the server.host
  32. # setting. When the value of this setting is false, Kibana uses the hostname of the host
  33. # that connects to this Kibana instance.
  34. #elasticsearch.preserveHost: true
  35.  
  36. # Kibana uses an index in Elasticsearch to store saved searches, visualizations and
  37. # dashboards. Kibana creates a new index if the index doesn't already exist.
  38. #kibana.index: ".kibana"
  39.  
  40. # The default application to load.
  41. #kibana.defaultAppId: "home"
  42.  
  43. # If your Elasticsearch is protected with basic authentication, these settings provide
  44. # the username and password that the Kibana server uses to perform maintenance on the Kibana
  45. # index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
  46. # is proxied through the Kibana server.
  47. #elasticsearch.username: "user"
  48. #elasticsearch.password: "pass"
  49.  
  50. # Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
  51. # These settings enable SSL for outgoing requests from the Kibana server to the browser.
  52. #server.ssl.enabled: false
  53. #server.ssl.certificate: /path/to/your/server.crt
  54. #server.ssl.key: /path/to/your/server.key
  55.  
  56. # Optional settings that provide the paths to the PEM-format SSL certificate and key files.
  57. # These files validate that your Elasticsearch backend uses the same key files.
  58. #elasticsearch.ssl.certificate: /path/to/your/client.crt
  59. #elasticsearch.ssl.key: /path/to/your/client.key
  60.  
  61. # Optional setting that enables you to specify a path to the PEM file for the certificate
  62. # authority for your Elasticsearch instance.
  63. #elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]
  64.  
  65. # To disregard the validity of SSL certificates, change this setting's value to 'none'.
  66. #elasticsearch.ssl.verificationMode: full
  67.  
  68. # Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of
  69. # the elasticsearch.requestTimeout setting.
  70. #elasticsearch.pingTimeout: 
  71.  
  72. # Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
  73. # must be a positive integer.
  74. #elasticsearch.requestTimeout: 
  75.  
  76. # List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side
  77. # headers, set this value to [] (an empty list).
  78. #elasticsearch.requestHeadersWhitelist: [ authorization ]
  79.  
  80. # Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten
  81. # by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.
  82. #elasticsearch.customHeaders: {}
  83.  
  84. # Time in milliseconds for Elasticsearch to wait for responses from shards. Set to  to disable.
  85. #elasticsearch.shardTimeout: 
  86.  
  87. # Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying.
  88. #elasticsearch.startupTimeout: 
  89.  
  90. # Logs queries sent to Elasticsearch. Requires logging.verbose set to true.
  91. #elasticsearch.logQueries: false
  92.  
  93. # Specifies the path where Kibana creates the process ID file.
  94. #pid.file: /var/run/kibana.pid
  95.  
  96. # Enables you specify a file where Kibana stores log output.
  97. #logging.dest: stdout
  98.  
  99. # Set the value of this setting to true to suppress all logging output.
  100. #logging.silent: false
  101.  
  102. # Set the value of this setting to true to suppress all logging output other than error messages.
  103. #logging.quiet: false
  104.  
  105. # Set the value of this setting to true to log all events, including system usage information
  106. # and all requests.
  107. #logging.verbose: false
  108.  
  109. # Set the interval in milliseconds to sample system and process performance
  110. # metrics. Minimum is 100ms. Defaults to .
  111. #ops.interval: 
  112.  
  113. # The default locale. This locale can be used in certain circumstances to substitute any missing
  114. # translations.
  115. #i18n.defaultLocale: "en"
  116. #
  117. #

启动kibana：

  1. cd /usr/local/kibana-6.4.
  2. ./bin/kibana

成功启动后，在浏览器输入http://10.10.10.102:5601,界面如下：

如下地址可以查看kibana的状态和资源使用情况：

  1. 10.10.10.102:/status

3.

【ELK】ELK安装与配置的更多相关文章

  1. ELK安装与配置

    ELK介绍 日志主要包括系统日志.应用程序日志和安全日志.系统运维和开发人员可以通过日志了解服务器软硬件信息.检查配置过程中的错误及错误发生的原因.经常分析日志可以了解服务器的负荷,性能安全性,从而及 ...

  2. ELK 6安装配置 nginx日志收集 kabana汉化

    #ELK 6安装配置 nginx日志收集 kabana汉化 #环境 centos 7.4 ,ELK 6 ,单节点 #服务端 Logstash 收集,过滤 Elasticsearch 存储,索引日志 K ...

  3. ELK+SpringBoot+Logback离线安装及配置

    ELK+SpringBoot+Logback 离线安装及配置 版本 v1.0 编写时间 2018/6/11 编写人 xxx     目录 一. ELK介绍2 二. 安装环境2 三. Elasticse ...

  4. ELK日志分析系统之Kibana7.x最新版安装与配置

    3.Kibana的简介 Kibana 让您能够自由地选择如何呈现自己的数据.Kibana 核心产品搭载了一批经典功能:柱状图.线状图.饼图.旭日图等等. 3.1.软件包下载地址:https://www ...

  5. ELK简单安装

    ELK日志分析平台 一.ELK介绍 ELK是三个开源软件的缩写,分别为:Elasticsearch . Logstash以及Kibana,都是开源软件,新增一个beats,(轻量级日志处理工具Agen ...

  6. ELK简单安装测试

    1 介绍组件 Filebeat是一个日志文件托运工具,在你的服务器上安装客户端后,filebeat会监控日志目录或者指定的日志文件,追踪读取这些文件(追踪文件的变化,不停的读). Kafka是一种高吞 ...

  7. ELK Stack (1) —— ELK + Redis安装

    ELK Stack (1) -- ELK + Redis安装 摘要 安装Elasticsearch.Logstash.Kibana与Redis以实现一个日志收集平台 版本 elasticsearch版 ...

  8. ELK 二进制安装并收集nginx日志

    对于日志来说,最常见的需求就是收集.存储.查询.展示,开源社区正好有相对应的开源项目:logstash(收集).elasticsearch(存储+搜索).kibana(展示),我们将这三个组合起来的技 ...

  9. ELK节点安装

    ELK 安装参考链接 https://www.cnblogs.com/xialiaoliao0911/p/9599898.html setenforce 0sed -i s/enforcing/dis ...

随机推荐

  1. vue3.0学习笔记（一）

    一.搭建工作环境环境 1.从node.js官网下载相应版本进行安装即可 https://nodejs.org/zh-cn/download/,安装完成后在命令行输入  node -v 如果可以查询到版 ...

  2. Backbone源码解析系列

    01 编码风格.继承 02 Backbone.Events 03 Backbone.Model 04 Backbone.View 05 Backbone.Router 06 Backbone应用于we ...

  3. asp.net 子域跨域 带cookie

    先来一个老外的解决方案: http://www.asp.net/web-api/overview/security/enabling-cross-origin-requests-in-web-api ...

  4. websocket的加密和解密

    补充个小知识点:按位与运算 按位与运算是同位都为1才为1,有一个不为1就是0 websocket_hand import socket, base64, hashlib import websocke ...

  5. C++编写双向链表

    创建双向链表类,该类有默认构造函数.类的拷贝函数.类的.实现链表添加数据.升序排序.查找链表中某个节点及删除链表中某个节点的操作 代码实现: #include<iostream> #inc ...

  6. HDU3973 线段树 + 字符哈希

    题目链接:http://acm.hdu.edu.cn/showproblem.php?pid=3973 , 线段树 + 字符哈希,好题. 又学了一种新的哈希方法,hhhh~ 解法: 想法是用P进制的数 ...

  7. JIRA Plugin Development——Configurable Custom Field Plugin

    关于JIRA Plugin开发的中文资料相当少,这可能还是由于JIRA Plugin开发在国内比较小众的原因吧,下面介绍下自己的一个JIRA Plugin开发的详细过程. 业务需求 创建JIRA IS ...

  8. 解决linux系统CentOS下调整home和根分区大小《转》

    转自http://www.php114.net/2013/1019/637.html 目标:将VolGroup-lv_home缩小到20G,并将剩余的空间添加给VolGroup-lv_root   1 ...

  9. x5webview 微信H5支付

    mWebView.setWebViewClient(new WebViewClient() { // @Override // public boolean shouldOverrideUrlLoad ...

  10. JS let和const关键字

    ES2015 引入了两个重要的 JavaScript 新关键词:let 和 const. Let关键字 1.用于作用域:块作用域,循环作用域,函数作用域,全局作用域, 在 ES2015 之前,Java ...