Logstash：Data转换，分析，提取，丰富及核心操作

Logstash plugins

Logstash是一个非常容易进行扩张的框架。它可以对各种的数据进行分析处理。这依赖于目前提供的超过200多个plugin。

首先，我们来查看一下目前有哪些plugin：

Input plugins

首先进入到Logstash的安装目录下的bin子目录，并在命令行中打入如下的命令：

$ ./logstash-plugin list --group input

显示：

  1. logstash-input-azure_event_hubs
  2. logstash-input-beats
  3. logstash-input-couchdb_changes
  4. logstash-input-elasticsearch
  5. logstash-input-exec
  6. logstash-input-file
  7. logstash-input-ganglia
  8. logstash-input-gelf
  9. logstash-input-generator
  10. logstash-input-graphite
  11. logstash-input-heartbeat
  12. logstash-input-http
  13. logstash-input-http_poller
  14. logstash-input-imap
  15. logstash-input-jdbc
  16. logstash-input-jms
  17. logstash-input-kafka
  18. logstash-input-pipe
  19. logstash-input-rabbitmq
  20. logstash-input-redis
  21. logstash-input-s3
  22. logstash-input-snmp
  23. logstash-input-snmptrap
  24. logstash-input-sqs
  25. logstash-input-stdin
  26. logstash-input-syslog
  27. logstash-input-tcp
  28. logstash-input-twitter
  29. logstash-input-udp
  30. logstash-input-unix

Filter plugs

在命令行打入如下的命令：

  1. $ ./logstash-plugin list --group filter
  2. logstash-filter-aggregate
  3. logstash-filter-anonymize
  4. logstash-filter-cidr
  5. logstash-filter-clone
  6. logstash-filter-csv
  7. logstash-filter-date
  8. logstash-filter-de_dot
  9. logstash-filter-dissect
  10. logstash-filter-dns
  11. logstash-filter-drop
  12. logstash-filter-elasticsearch
  13. logstash-filter-fingerprint
  14. logstash-filter-geoip
  15. logstash-filter-grok
  16. logstash-filter-http
  17. logstash-filter-jdbc_static
  18. logstash-filter-jdbc_streaming
  19. logstash-filter-json
  20. logstash-filter-kv
  21. logstash-filter-memcached
  22. logstash-filter-metrics
  23. logstash-filter-mutate
  24. logstash-filter-prune
  25. logstash-filter-ruby
  26. logstash-filter-sleep
  27. logstash-filter-split
  28. logstash-filter-syslog_pri
  29. logstash-filter-throttle
  30. logstash-filter-translate
  31. logstash-filter-truncate
  32. logstash-filter-urldecode
  33. logstash-filter-useragent
  34. logstash-filter-uuid
  35. logstash-filter-xml

Output plugins

在命令行打入如下的命令：

  1. $ ./logstash-plugin list --group output
  2. logstash-output-cloudwatch
  3. logstash-output-csv
  4. logstash-output-elastic_app_search
  5. logstash-output-elasticsearch
  6. logstash-output-email
  7. logstash-output-file
  8. logstash-output-graphite
  9. logstash-output-http
  10. logstash-output-lumberjack
  11. logstash-output-nagios
  12. logstash-output-null
  13. logstash-output-pipe
  14. logstash-output-rabbitmq
  15. logstash-output-redis
  16. logstash-output-s3
  17. logstash-output-sns
  18. logstash-output-sqs
  19. logstash-output-stdout
  20. logstash-output-tcp
  21. logstash-output-udp
  22. logstash-output-webhdfs

Codec plugins:

在命令行打入如下的命令：

  1. $ ./logstash-plugin list codec
  2. logstash-codec-avro
  3. logstash-codec-cef
  4. logstash-codec-collectd
  5. logstash-codec-dots
  6. logstash-codec-edn
  7. logstash-codec-edn_lines
  8. logstash-codec-es_bulk
  9. logstash-codec-fluent
  10. logstash-codec-graphite
  11. logstash-codec-json
  12. logstash-codec-json_lines
  13. logstash-codec-line
  14. logstash-codec-msgpack
  15. logstash-codec-multiline
  16. logstash-codec-netflow
  17. logstash-codec-plain
  18. logstash-codec-rubydebug

在这上面显示都是我们在安装Logstash后，已经给我们配置好的plugin。我们可以自己开发自己的plugin，并安装它。我们也可以安装一个别人已经开发好的plugin。

从上面我们可以看出来，因为file都在input及output之中，我们甚至可以做如下的配置：

  1. input {
  2.    file {
  3.       path => "C:/Program Files/Apache Software Foundation/Tomcat 7.0/logs/*access*"
  4.       type => "apache"
  5.    }
  6. }
  7. output {
  8.    file {
  9.       path => "C:/tpwork/logstash/bin/log/output.log"
  10.    }
  11. }

这样我们把input文件读入到Logstash，经过它的处理后，就会变成下面的这种输出：

  1. 0:0:0:0:0:0:0:1 - - [
  2.    25/Dec/2016:18:37:00 +0800] "GET / HTTP/1.1" 200 11418
  4. {
  5.    "path":"C:/Program Files/Apache Software Foundation/Tomcat 7.0/logs/
  6.    localhost_access_log.2016-12-25.txt",
  7.    "@timestamp":"2016-12-25T10:37:00.363Z","@version":"1","host":"Dell-PC",
  8.    "message":"0:0:0:0:0:0:0:1 - - [25/Dec/2016:18:37:00 +0800] \"GET /
  9.    HTTP/1.1\" 200 11418\r","type":"apache","tags":[]
  10. }

安装plugin

在标准的logstash中，有很多的plugin已经被安装了，但是在有些场合，我们需要手动来安装一些我们所需要的plugin，比如Exec output plugin。我们可以在bin目录先打人如下的命令：

./bin/logstash-plugin install logstash-output-exec

这样我们用如下的命令来检查上面的plugin是否已经被成功安装了：

  1. ./bin/logstash-plugin list --group output | grep exec
  3. $ ./bin/logstash-plugin list --group output | grep exec
  4. Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
  5. WARNING: An illegal reflective access operation has occurred
  6. WARNING: Illegal reflective access by org.bouncycastle.jcajce.provider.drbg.DRBG (file:/Users/liuxg/elastic/logstash-7.4.0/vendor/jruby/lib/ruby/stdlib/org/bouncycastle/bcprov-jdk15on/1.61/bcprov-jdk15on-1.61.jar) to constructor sun.security.provider.Sun()
  7. WARNING: Please consider reporting this to the maintainers of org.bouncycastle.jcajce.provider.drbg.DRBG
  8. WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
  9. WARNING: All illegal access operations will be denied in a future release
  10. logstash-output-exec

读取log文件

Logstash很容易设置来读取一个log文件。比如，我们可以通过如下的方式来读取一个Apache的log文件：

  1. input {
  2.   file {
  3.   	type => "apache"
  4.   	path => "/Users/liuxg/data/apache_logs"
  5.  	  start_position => "beginning"
  6. 	  sincedb_path => "null"
  7.   }
  8. }
  10. output {
  11. 	stdout {
  12. 		codec => rubydebug
  13. 	}
  14. }

我们甚至可以读取多个文件：

  1. # Pull in application-log data. They emit data in JSON form.
  2. input {
  3.   file {
  4.     path => [
  5.       "/var/log/app/worker_info.log",
  6.       "/var/log/app/broker_info.log",
  7.       "/var/log/app/supervisor.log"
  8.     ]
  9.     exclude => "*.gz"
  10.     type    => "applog"
  11.     codec   => "json"
  12.   }
  13. }

数据的系列化

我们可以使用已经提供的Codec来把我们的数据进行系列化，比如：

  1. input {
  2.   // Deserialize newline separated JSON
  3.   file  { path => “/some/sample.log”, codec => json }
  4. }
  6. output {
  7.   // Serialize to the msgpack format
  8.   redis { codec => msgpack }
  9.   stdout {
  10.     codec => rubydebug
  11.   }
  12. }

在我们的longstash运行起来后，我们可以通过如下的命令在一个terminal中向文件sample.json添加内容：

$ echo '{"name2", "liuxg2"}' >> ./sample.log

我们可以看到如下的输出：

  1. {
  2.       "@version" => "1",
  3.        "message" => "{\"name2\", \"liuxg2\"}",
  4.     "@timestamp" => 2019-09-12T07:37:56.639Z,
  5.           "host" => "localhost",
  6.           "tags" => [
  7.         [0] "_jsonparsefailure"
  8.     ],
  9.           "path" => "/Users/liuxg/data/sample.log"
  10. }

最常用的codec

  1. line 使用“message”中的数据将每行转换为logstash事件。 也可以将输出格式化为自定义行 。

  2. multiline: 允许您为“message”构成任意边界。 经常用于stacktraces等。也可以在filebeat中完成。

  3. json_lines: 解析换行符分隔的JSON数据

  4. json: 解析所有JSON。 仅适用于面向消息的输入/输出，如Redis / Kafka / HTTP等

还有很多其它的Codec。

解析及提取

Grok Filter

  1. filter {
  2. 	grok {
  3. 		match => [
  4. 			"message", "%{TIMESTAMP_ISO8601:timestamp_string}%{SPACE}%{GREEDYDATA:line}"
  5. 		]
  6. 	}
  7. }

上面的例子可以帮我们很方便地把如下的log信息变成一个机构化的数据：

  1. 2019-09-09T13:00:00Z Whose woods these are I think I know.

更多grok的pattern可以在地址grok pattern找到。

Date filter

  1. filter {
  2.   date {
  3.     match => ["timestamp_string", "ISO8601"]
  4.   }
  5. }

Date filter可以帮我们把一个字符串，变成一个我们想要的格式的时间，并且把这个值赋予给@timestamp字段。

Dissect filter

是一个更快，轻量级的更小的grok：

  1. filter {
  2.   dissect {
  3.     mapping => {“message” => “%{id} %{function->} %{server}”}
  4.   }
  5. }

字段和分隔符模式的格式类似于Grok。

例子：

  1.     input {
  2.       generator {
  3.         message => "<1>Oct 16 20:21:22 www1 1,2016/10/16 20:21:20,3,THREAT,SCAN,6,2016/10/16 20:21:20,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54"
  4.         count => 1
  5.       }
  6.     }
  8.     filter {
  9.       if [message] =~ "THREAT," {
  10.         dissect {
  11.           mapping => {
  12.             message => "<%{priority}>%{syslog_timestamp} %{+syslog_timestamp} %{+syslog_timestamp} %{logsource} %{pan_fut_use_01},%{pan_rec_time},%{pan_serial_number},%{pan_type},%{pan_subtype},%{pan_fut_use_02},%{pan_gen_time},%{pan_src_ip},%{pan_dst_ip},%{pan_nat_src_ip},%{pan_nat_dst_ip},%{pan_rule_name},%{pan_src_user},%{pan_dst_user},%{pan_app},%{pan_vsys},%{pan_src_zone},%{pan_dst_zone},%{pan_ingress_intf},%{pan_egress_intf},%{pan_log_fwd_profile},%{pan_fut_use_03},%{pan_session_id},%{pan_repeat_cnt},%{pan_src_port},%{pan_dst_port},%{pan_nat_src_port},%{pan_nat_dst_port},%{pan_flags},%{pan_prot},%{pan_action},%{pan_misc},%{pan_threat_id},%{pan_cat},%{pan_severity},%{pan_direction},%{pan_seq_number},%{pan_action_flags},%{pan_src_location},%{pan_dst_location},%{pan_content_type},%{pan_pcap_id},%{pan_filedigest},%{pan_cloud},%{pan_user_agent},%{pan_file_type},%{pan_xff},%{pan_referer},%{pan_sender},%{pan_subject},%{pan_recipient},%{pan_report_id},%{pan_anymore}"
  13.           }
  14.         }
  15.       }
  16.     }
  18.     output {
  19.     	stdout {
  20.     		codec => rubydebug
  21.     	}
  22.     }

运行后：

  1.     {
  2.                  "@timestamp" => 2019-09-12T09:20:46.514Z,
  3.                  "pan_dst_ip" => "9",
  4.              "pan_nat_src_ip" => "10",
  5.                    "sequence" => 0,
  6.                   "logsource" => "www1",
  7.              "pan_session_id" => "23",
  8.                    "pan_vsys" => "16",
  9.                     "pan_cat" => "34",
  10.               "pan_rule_name" => "12",
  11.                "pan_gen_time" => "2016/10/16 20:21:20",
  12.              "pan_seq_number" => "37",
  13.                 "pan_subject" => "50",
  15.                     ....
  17.                     "message" => "<1>Oct 16 20:21:22 www1 1,2016/10/16 20:21:20,3,THREAT,SCAN,6,2016/10/16 20:21:20,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54",
  18.              "pan_fut_use_02" => "6",
  19.                   "pan_flags" => "29",
  20.            "syslog_timestamp" => "Oct 16 20:21:22",
  21.                 "pan_anymore" => "53,54"
  22.     }

KV filter

解析键/值对中数据的简便方法

  1.     filter {
  2.       kv {
  3.         source => “message”
  4.         target => “parsed”
  5.         value_split => “:”
  6.       }
  7.     }

我们运行这样的conf文件：

  1.     input {
  2.       generator {
  3.         message => "pin=12345~0&d=123&e=foo@bar.com&oq=bobo&ss=12345"
  4.         count => 1
  5.       }
  6.     }
  8.     filter {
  9.     	kv {
  10.     		source => "message"
  11.     		target => "parsed"
  12.     		field_split => "&?"
  13.     	}
  14.     }
  16.     output {
  17.     	stdout {
  18.     		codec => rubydebug
  19.     	}
  20.     }

显示的结果是：

  1.     {
  2.         "@timestamp" => 2019-09-12T09:46:04.944Z,
  3.               "host" => "localhost",
  4.             "parsed" => {
  5.              "ss" => "12345",
  6.               "e" => "foo@bar.com",
  7.             "pin" => "12345~0",
  8.              "oq" => "bobo",
  9.               "d" => "123"
  10.         },
  11.            "message" => "pin=12345~0&d=123&e=foo@bar.com&oq=bobo&ss=12345",
  12.           "sequence" => 0,
  13.           "@version" => "1"
  14.     }

对于kv flter来说，我们也可以使用一个target来把信息组织到一个object里，比如：

  1.     filter {
  2.       kv {
  3.         source => “message”
  4.         target => “parsed”
  5.         value_split => “:”
  6.       }
  7.     }

核心操作Mutate filter

这个filter提供很多功能：

  • 转换字段类型（从字符串到整数等）
  • 添加/重命名/替换/复制字段
  • 大-小写转换
  • 将数组连接在一起（对于Array => String操作很有用）
  • 合并哈希
  • 将字段拆分为数组
  • 剥去空白
  1.     input {
  2.       generator {
  3.         message => "pin=12345~0&d=123&e=foo@bar.com&oq=bobo&ss=12345"
  4.         count => 1
  5.       }
  6.     }
  8.     filter {
  9.     	kv {
  10.     		source => "message"
  11.     		field_split => "&?"
  12.     	}
  14.     	if [pin] == "12345~0" {
  15.         	mutate { add_tag => [ 'metrics' ]
  16.         }
  18.         mutate {
  19.         	split => ["message", "&"]
  20.         	add_field => {"foo" => "bar-%{pin}"}
  21.       	}
  22.       }
  23.     }
  25.     output {
  26.     	stdout {
  27.     		codec => rubydebug
  28.     	}
  30.     	if "metrics" in [tags] {
  31.           stdout {
  32.              codec => line { format => "custom format: %{message}" }
  33.           }
  34.        }
  35.     }

显示的结果是：

  1.     {
  2.          "foo" => "bar-12345~0",
  3.           "e" => "foo@bar.com",
  4.           "sequence" => 0,
  5.            "message" => [
  6.             [0] "pin=12345~0",
  7.             [1] "d=123",
  8.             [2] "e=foo@bar.com",
  9.             [3] "oq=bobo",
  10.             [4] "ss=12345"
  11.         ],
  12.                "pin" => "12345~0",
  13.                  "d" => "123",
  14.               "host" => "localhost",
  15.                 "ss" => "12345",
  16.         "@timestamp" => 2019-09-14T15:03:15.141Z,
  17.                 "oq" => "bobo",
  18.           "@version" => "1",
  19.               "tags" => [
  20.             [0] "metrics"
  21.         ]
  22.     }
  23.     custom format: pin=12345~0,d=123,e=foo@bar.com,oq=bobo,ss=12345

最核心的转化filters

  • Mute - 修改/添加每个项
  • Split - 把一个事件转化为多个事件
  • Drop - 丢掉一个事件

条件逻辑

if/else

  • 可以用 =~来使用regexps（正则）
  • 可以在一个数组里检查一个会员
  1.     filter {
  2.       mutate { lowercase => “account” }
  3.       if [type] == “batch” {
  4.         split {
  5.             field => actions
  6.            target => action
  7.         }
  8.       }
  10.       if { “action” =~ /special/ } {
  11.         drop {}
  12.       }
  13.     }

GeoIP

丰富IP地址信息：

filter { geoip { fields => “my_geoip_field” }}

运行如下的配置：

  1.     input {
  2.       generator {
  3.         message => "83.149.9.216"
  4.         count => 1
  5.       }
  6.     }
  8.     filter {
  9.     	grok {
  10.         	match => {
  11.           		"message" => '%{IPORHOST:clientip}'
  12.         	}
  13.         }
  15.         geoip {
  16.         	source => "clientip"
  17.       	}
  18.     }
  20.     output {
  21.     	stdout {
  22.     		codec => rubydebug
  23.     	}
  24.     }

显示的结果如下：

  1.     {
  2.               "host" => "localhost",
  3.           "@version" => "1",
  4.           "clientip" => "83.149.9.216",
  5.            "message" => "83.149.9.216",
  6.         "@timestamp" => 2019-09-15T06:54:46.695Z,
  7.           "sequence" => 0,
  8.              "geoip" => {
  9.                   "timezone" => "Europe/Moscow",
  10.                "region_code" => "MOW",
  11.                   "latitude" => 55.7527,
  12.              "country_code3" => "RU",
  13.             "continent_code" => "EU",
  14.                  "longitude" => 37.6172,
  15.               "country_name" => "Russia",
  16.                   "location" => {
  17.                 "lat" => 55.7527,
  18.                 "lon" => 37.6172
  19.             },
  20.                         "ip" => "83.149.9.216",
  21.                "postal_code" => "102325",
  22.              "country_code2" => "RU",
  23.                "region_name" => "Moscow",
  24.                  "city_name" => "Moscow"
  25.         }
  26.     }

我们可以看到在geoip之下，有很多具体的信息。

DNS filter

用DNS信息丰富主机名的更多信息

filter { dns { fields => “my_dns_field” }}

我们定义如下的一个Logstash配置文件：

  1.     input {
  2.       generator {
  3.         message => "www.google.com"
  4.         count => 1
  5.       }
  6.     }
  8.     filter {
  9.      	mutate {
  10.         	add_field => { "hostname" => "172.217.160.110"}
  11.     	}
  13.     	dns {
  14.     		reverse => ["hostname"]
  15.     		action => "replace"
  16.     	}   
  18.     }
  20.     output {
  21.     	stdout {
  22.     		codec => rubydebug
  23.     	}
  24.     }

上面是谷歌的地址，那么它的输出结果是：

  1.     {
  2.               "host" => "localhost",
  3.           "sequence" => 0,
  4.            "message" => "www.google.com",
  5.         "@timestamp" => 2019-09-15T11:35:43.791Z,
  6.           "hostname" => "tsa03s06-in-f14.1e100.net",
  7.           "@version" => "1"
  8.     }

在这里我们可以看到hostname的值。

Useragent filer

让浏览器的useragent信息更加丰富。我们使用如下的Logstash配置：

  1.     input {
  2.       generator {
  3.         message => '83.149.9.216 - - [17/May/2015:10:05:50 +0000] "GET /presentations/logstash-monitorama-2013/images/kibana-dashboard.png HTTP/1.1" 200 321631 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"'
  4.         count => 1
  5.       }
  6.     }
  8.     filter {
  9.     	grok {
  10.     	    match => {
  11.     	      "message" => '%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int}) %{QS:referrer} %{QS:agent}'
  12.     	    }
  13.     	  }
  15.     	useragent {
  16.         	source => "agent"
  17.         	target => "useragent"
  18.       	}
  19.     }
  21.     output {
  22.     	stdout {
  23.     		codec => rubydebug
  24.     	}
  25.     }

运行出来的结果是：

  1.     {
  2.             "request" => "/presentations/logstash-monitorama-2013/images/kibana-dashboard.png",
  3.           "useragent" => {
  4.                 "name" => "Chrome",
  5.                "build" => "",
  6.               "device" => "Other",
  7.             "os_major" => "10",
  8.                   "os" => "Mac OS X",
  9.                "minor" => "0",
  10.                "major" => "32",
  11.              "os_name" => "Mac OS X",
  12.                "patch" => "1700",
  13.             "os_minor" => "9"
  14.         },
  15.            "sequence" => 0,
  16.             "message" => "83.149.9.216 - - [17/May/2015:10:05:50 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-dashboard.png HTTP/1.1\" 200 321631 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
  17.           "timestamp" => "17/May/2015:10:05:50 +0000",
  18.            "referrer" => "\"http://semicomplete.com/presentations/logstash-monitorama-2013/\"",
  19.            "clientip" => "83.149.9.216",
  20.               "ident" => "-",
  21.                "auth" => "-",
  22.            "response" => 200,
  23.            "@version" => "1",
  24.                "verb" => "GET",
  25.                "host" => "localhost",
  26.          "@timestamp" => 2019-09-15T12:03:34.650Z,
  27.         "httpversion" => "1.1",
  28.               "bytes" => 321631,
  29.               "agent" => "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\""
  30.     }

我们在useragent里可以看到更加详细的信息啊。

Translate Filter

使用本地的数据来使得数据更加丰富。我们使用如下的Logstash配置文件：

  1.     input {
  2.       generator {
  3.         message => '83.149.9.216 - - [17/May/2015:10:05:50 +0000] "GET /presentations/logstash-monitorama-2013/images/kibana-dashboard.png HTTP/1.1" 200 321631 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"'
  4.         count => 1
  5.       }
  6.     }
  8.     filter {
  9.     	grok {
  10.     	    match => {
  11.     	      "message" => '%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int}) %{QS:referrer} %{QS:agent}'
  12.     	    }
  13.     	 }
  15.     	translate {
  16.     		field => "[response]"
  17.     		destination => "[http_status_description]"
  18.     		dictionary => {
  19.              	"100" => "Continue"
  20.               	"101" => "Switching Protocols"
  21.               	"200" => "OK"
  22.               	"500" => "Server Error"
  23.     		}
  25.     		fallback => "I'm a teapot"
  26.     	}
  28.     }
  30.     output {
  31.     	stdout {
  32.     		codec => rubydebug
  33.     	}
  34.     }

运行显示的结果是：

  1.     {
  2.                            "auth" => "-",
  3.                            "host" => "localhost",
  4.                       "timestamp" => "17/May/2015:10:05:50 +0000",
  5.                         "message" => "83.149.9.216 - - [17/May/2015:10:05:50 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-dashboard.png HTTP/1.1\" 200 321631 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
  6.                     "httpversion" => "1.1",
  7.                        "@version" => "1",
  8.                        "response" => 200,
  9.                        "clientip" => "83.149.9.216",
  10.                            "verb" => "GET",
  11.                        "sequence" => 0,
  12.                        "referrer" => "\"http://semicomplete.com/presentations/logstash-monitorama-2013/\"",
  13.                           "agent" => "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
  14.         "http_status_description" => "OK",
  15.                           "ident" => "-",
  16.                      "@timestamp" => 2019-09-15T12:30:09.575Z,
  17.                           "bytes" => 321631,
  18.                         "request" => "/presentations/logstash-monitorama-2013/images/kibana-dashboard.png"
  19.     }

我们可以看到一项http_status_description，它的值变为“OK”。

Elasticsearch Filter

从Elasticsearch中的index得到数据，并丰富事件。为了做这个测试，我们先建立一个叫做elasticsearch_filter的index:

  1.     PUT ç/_doc/1
  2.     {
  3.       "name":"liuxg",
  4.       "age": 20,
  5.       "@timestamp": "2019-09-15"
  6.     }

在这里，我必须指出来的是：我们必须有一个叫做@timestamp的项，否则会有错误。这个是用来做sort用的。

我们采用如下的Logstash配置：

  1.     input {
  2.       generator {
  3.         message => "liuxg"
  4.         count => 1
  5.       }
  6.     }
  8.     filter {
  9.     	elasticsearch {
  10.     		hosts => ["http://localhost:9200"]
  11.     		index => ["elasticsearch_filter"]
  12.     		query => "name.keyword:%{[message]}"
  13.     		result_size => 1
  14.     		fields => {"age" => "user_age"}
  15.     	}
  16.     }
  18.     output {
  19.     	stdout {
  20.     		codec => rubydebug
  21.     	}
  22.     }

运行上面的例子显示的结果是：

  1.     {
  2.           "user_age" => 20,
  3.               "host" => "localhost",
  4.            "message" => "liuxg",
  5.           "@version" => "1",
  6.         "@timestamp" => 2019-09-15T13:21:29.742Z,
  7.           "sequence" => 0
  8.     }

我们可以看到user_age是20。这个是通过搜索name:liuxg来得到的。

参考：https://opensource.com/article/17/10/logstash-fundamentals

Logstash：Data转换，分析，提取，丰富及核心操作的更多相关文章

  1. Linux网络地址转换分析

    Linux网络地址转换分析 地址转换用来改变源/目的端口,是netfilter的一部分,也是通过hook点上注册相应的结构来工作. Nat注册的hook点和conntrack相同,只是优先级不同,数据 ...

  2. Java线程Thread的状态解析以及状态转换分析 多线程中篇（七）

    线程与操作系统中线程(进程)的概念同根同源,尽管千差万别. 操作系统中有状态以及状态的切换,Java线程中照样也有. State 在Thread类中有内部类 枚举State,用于抽象描述Java线程的 ...

  3. Zepto源码分析（一）核心代码分析

    本文只分析核心的部分代码,并且在这部分代码有删减,但是不影响代码的正常运行. 目录 * 用闭包封装Zepto * 开始处理细节 * 正式处理数据(获取选择器选择的DOM) * 正式处理数据(添加DOM ...

  4. kube-scheduler源码分析（2）-核心处理逻辑分析

    kube-scheduler源码分析(2)-核心处理逻辑分析 kube-scheduler简介 kube-scheduler组件是kubernetes中的核心组件之一,主要负责pod资源对象的调度工作 ...

  5. [OpenCV-Python] OpenCV 核心操作 部分 III

    部分 III核心操作 OpenCV-Python 中文教程(搬运)目录 9 图像的基础操作 目标 • 获取像素值并修改 • 获取图像的属性(信息) • 图像的 ROI() • 图像通道的拆分及合并几乎 ...

  6. Python股票分析系列——基础股票数据操作（一）.p3

    该系列视频已经搬运至bilibili: 点击查看 欢迎来到Python for Finance教程系列的第3部分.在本教程中,我们将使用我们的股票数据进一步分解一些基本的数据操作和可视化.我们将要使用 ...

  7. 关于activiti流程通过、驳回、会签、转办、中止、挂起等核心操作功能的封装

    http://blog.csdn.net/aochuanguying/article/details/7594197 package com.famousPro.process.service.imp ...

  8. Activiti之流程通过、驳回、会签、转办、中止、挂起等核心操作封装(Activiti5.9)

    http://blog.csdn.net/rosten/article/details/38300267 package com.famousPro.process.service.impl; imp ...

  9. Docker搭建ElasticSearch+Redis+Logstash+Filebeat日志分析系统

    一.系统的基本架构 在以前的博客中有介绍过在物理机上搭建ELK日志分析系统,有兴趣的朋友可以看一看-------------->>链接戳我<<.这篇博客将介绍如何使用Docke ...

随机推荐

  1. springCloud实战篇——纯文本

    什么是微服务架构? 微服务是系统架构的一种设计风格,主旨是将原本独立的系统产分为多个小型的服务,这些服务都在各自的进程中运行.服务与服务之间基于HTTP的RESTful API进行通信协作. 构造背景 ...

  2. 环信即时通讯在工程中的安装——Nusen_Liu

    即时通讯-环信 准备 1.下载SDK http://www.easemob.com/download 2.证书下载上传 后期发送消息 需要推送发送的内容 http://docs.easemob.com ...

  3. iOS 多线程知识梳理

    #iOS多线程知识梳理 ##线程进程基础概念 ###进程 进程是指在系统中正在运行的一个应用程序每个进程之间是独立的,每个进程均运行在其专用且受保护的内存空间内 ###线程 1个进程要想执行任务,必须 ...

  4. 使用 gitlab 进行代码管理

    这里使用 gitlab 做服务器, 客户端主要使用 git extensions. ============================= gitlab 项目成员类型: ============= ...

  5. 『006』Shell脚本

    『003』索引-Linux Shell Script Shel脚本-初步入门 [001]- 点我快速打开文章[<01 什么是 Shell>] [002]- 点我快速打开文章[<02 ...

  6. 第16讲：ODBC&JDBC简介

    一.ODBC简介 1. ODBC的概念 ①ODBC:Open DataBase Connection,即开放数据库连接 ②ODBC是一种标准,它规定了不同语言的应用程序与不同数据库服务器之间通讯的方式 ...

  7. fetchone函数和fetchall函数返回值的区别

     fetchone函数和fetchall函数返回值的区别 1.fetchone() 返回单个的元组,也就是一条记录(row),如果没有结果,则python返回 None 有结果时,如图: 没结果时,如 ...

  8. (二)Amazon Lightsail 部署LAMP应用程序之部署单片LAMP应用程序

    部署单片LAMP应用程序 简介:通过复制应用程序代码并提供链接PHP前端和本地MySQL数据库的参数,将LAMP对战应用程序部署到先前启动的Lightsail实例中.完成后,Apache/PHP前端和 ...

  9. c# WF 第9节 button控件

    本节内容: 1:实现实例 1:实现实例 每当点击一个确定就出现一个窗口,当点击最后的确定时,关闭所有的窗口. 实现: 步骤1:对Form 1 -Form3 依次进行如下设置: 步骤2 : 当每点击一个 ...

  10. python3.5.3rc1学习九：正则表达式

    # 正则表达式 ''''' 正则表达式是有一些特殊字符组成,能够帮你找到一些符合一定规则的字符串 先来了解几个符号所代表的意思 \d 匹配所有的数字 \D 匹配所有,但是数字除外 \s 空格 \S 匹 ...