ELK日志处理

ELK的工作原理：

使用多播进行机器发现同一个集群内的节点，并汇总各个节点的返回组成一个集群，主节点要读取各个节点的状态，在关键时候进行数据的恢复，主节点会坚持各个节点的状态，并决定每个分片的位置，通过ping的request检测各失效的节点.

ELK架构：

ElasticSearch:用于存储、索引日志.

Logstash:用于收集、处理和转发事件或日志信息的工具.

Kibana:搜索和可视化的日志的WEB界面.

ELK优点：

a.处理方式灵活：ElasticSearch是实时全文索引.

b.配置简单易上手.

c.检索性能高效:虽然每次计算都是实时计算的，但是优秀的设计基本可以达到全天数据查询的秒级响应.

d.集群线性扩展：ElasticSearch和Logstash集群都是可以线性扩展的.

e.前端操作绚丽：Kibana界面上，只需要点击鼠标，就可以完成搜索、聚合功能，生成绚丽的仪表板.

0.安装前准备：

ElasticSearch和Logstash需要java环境，需要安装JDK1.7以上的版本.

a.下载JDK的rpm包

b.安装

c.Java -version :检测安装的JDK

Elasticsearch：

概念：

1.索引：数据会放在多个索引中，索引可以理解为database,索引里面存放的基本单位是文档，elasticsearch会把索引分片，便于横向扩展，分别可以做备份，多个分片读比较快，备份分片在主的挂掉之后可以自动将自己提升为主分片（实现横向扩展和冗余）

2.文档类型：和redis一样，key是有类型的

3.节点：一个elasticsearch的实例是一个节点

4.集群：多节点的集合组成集群，类似于zookeeper会选举出主节点，客户端不需要关注主节点，连接任何一个都可以，数据会自动同步.

安装Elasticsearch

a.wget https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/rpm/elasticsearch/2.3.5/elasticsearch-2.3.5.rpm

b.rpm -ivh elasticsearch-2.3.5.rpm

c.mkdir /opt/develop/elasticsearch/data -p

mkdir /opt/develop/elasticsearch/log -p

d.# vi /usr/share/elasticsearch/config/elasticsearch.yml

Cluster.name:my-application --集群的名称，名称相同就是一个集群

Node.name:node-1 --集群情况下，当前node的名字，每个node应该不一样

Path.data=/opt/develop/elasticsearch/data

Path.log=/opt/develop/elasticsearch/log

Network.host=xxx.xxx.xx.xx

http.port:9200 --客户端访问端口

node.max_local_storage_nodes: 1

e.ElasticSearch需要使用非root用户启动服务

Groupadd ela

Useradd ela -g ela -p xxx

Su – ela

执行安装路径下的/elasticsearch启动服务

f.curl -X GET http://localhost:9200/ 查看ElasticSearch的安装信息----启动成功

g.chkconfig –add elasticsearch

Elasticsearch集群：

1.基于http的restful API：以jsop返回查询结果：

$curl -XGET http://10.26.44.42:9200/_count?pretty -d '

{

"query":{

"match_all":{}

}

}

'

{

"count" : 308590265,

"_shards" : {

"total" : 4180,

"successful" : 4180,

"failed" : 0

}

}

安装Logstash

a.wget https://download.elastic.co/logstash/logstash/packages/centos/logstash-2.3.4-1.noarch.rpm

b.安装

c.启动服务

d.测试：cd /opt/logstash/bin

./logstash -e ‘input { stdin {} } output { stdout {} }’

e.使用ruby进行更详细的输出：

./logstash -e 'input { stdin {}} output { stdout{codec => rubydebug}}'

Settings: Default pipeline workers: 8

Pipeline main started

asd

{

"message" => "asd",

"@version" => "1",

"@timestamp" => "2017-02-13T08:39:56.079Z",

"host" => "ali-hk-ops-elk1"

}

f.通过logstash将输出交给elasticsearch:

./logstash -e ‘input { stdin{} } output { elasticsearch { host => “ali-hk-ops-elk1:9200”protocol => “http”} }’

g.配置文件格式：

input {

file {

path => “/var/log/messages”

type => “syslog”

}

file {

path => “/var/log/apache/access.log”

type => “apache”

}

}

Logstash的input使用语法：

1.input,默认不支持目录的递归，即目录中还有文件是不支持直接读取的，但是可以使用/进行匹配

2.Exclude---->排除文件

Exclude => “*.gz”

3.sincedb_path,记录读取的时候位置，默认是一个隐藏文件

4.Sincedb_write_interval,记录sincedb_path文件的写间隔，默认是15秒

5.Start_position,从这个文件的什么位置开始读，默认是end，可以改成beginning

6.start_interval,多久检测一次此文件的更新状态

logstash的output使用及插件:

1.可以输出到文件、redis等

2.gzip，是否压缩，默认为false，压缩是安装数据流一点点增量压缩的

3.Message_format,消息的格式

Logstash-->file-->elasticsearch:

通过logstash输出到文件再输出到elasticsearch；

1.启动脚本：

Vim /etc/init.d/logstash

-#!/bin/sh

-# Init script for logstash

-# Maintained by Elasticsearch

-# Generated by pleaserun.

-# Implemented based on LSB Core 3.1:

-# * Sections: 20.2, 20.3

-#

-### BEGIN INIT INFO

-# Provides: logstash

-# Required-Start: $remote_fs $syslog

-# Required-Stop: $remote_fs $syslog

-# Default-Start: 2 3 4 5

-# Default-Stop: 0 1 6

-# Short-Description:

-# Description: Starts Logstash as a daemon.

-### END INIT INFO

PATH=/sbin:/usr/sbin:/bin:/usr/bin

export PATH

if [ id -u -ne 0 ]; then

echo "You need root privileges to run this script"

exit 1

fi

name=logstash

pidfile="/var/run/$name.pid"

LS_USER=logstash

LS_GROUP=logstash

LS_HOME=/var/lib/logstash

LS_HEAP_SIZE="4g"

LS_LOG_DIR=/var/log/logstash

LS_LOG_FILE="${LS_LOG_DIR}/$name.log"

LS_CONF_DIR=/etc/logstash/conf.d

LS_OPEN_FILES=16384

LS_NICE=19

KILL_ON_STOP_TIMEOUT=${KILL_ON_STOP_TIMEOUT-0} #default value is zero to this variable but could be updated by user request

LS_OPTS=""

[ -r /etc/default/$name ] && . /etc/default/$name

[ -r /etc/sysconfig/$name ] && . /etc/sysconfig/$name

program=/opt/logstash/bin/logstash

args="agent -f ${LS_CONF_DIR} -l ${LS_LOG_FILE} ${LS_OPTS}"

quiet() {

"$@" > /dev/null 2>&1

return $?

}

start() {

LS_JAVA_OPTS="${LS_JAVA_OPTS} -Djava.io.tmpdir=${LS_HOME}"

HOME=${LS_HOME}

export PATH HOME LS_HEAP_SIZE LS_JAVA_OPTS LS_USE_GC_LOGGING LS_GC_LOG_FILE

-# chown doesn't grab the suplimental groups when setting the user:group - so we have to do it for it.

-# Boy, I hope we're root here.

SGROUPS=$(id -Gn "$LS_USER" | tr " " "," | sed 's/,$//'; echo '')

if [ ! -z $SGROUPS ]

then

EXTRA_GROUPS="--groups $SGROUPS"

fi

-# set ulimit as (root, presumably) first, before we drop privileges

ulimit -n ${LS_OPEN_FILES}

-# Run the program!

nice -n ${LS_NICE} chroot --userspec $LS_USER:$LS_GROUP $EXTRA_GROUPS / sh -c "

cd $LS_HOME

ulimit -n ${LS_OPEN_FILES}

exec "$program" $args

" > "${LS_LOG_DIR}/$name.stdout" 2> "${LS_LOG_DIR}/$name.err" &

-# Generate the pidfile from here. If we instead made the forked process

-# generate it there will be a race condition between the pidfile writing

-# and a process possibly asking for status.

echo $! > $pidfile

echo "$name started."

return 0

}

stop() {

-# Try a few times to kill TERM the program

if status ; then

pid=cat "$pidfile"

echo "Killing $name (pid $pid) with SIGTERM"

kill -TERM $pid

-# Wait for it to exit.

for i in 1 2 3 4 5 6 7 8 9 ; do

echo "Waiting $name (pid $pid) to die..."

status || break

sleep 1

done

if status ; then

if [ $KILL_ON_STOP_TIMEOUT -eq 1 ] ; then

echo "Timeout reached. Killing $name (pid $pid) with SIGKILL. This may result in data loss."

kill -KILL $pid

echo "$name killed with SIGKILL."

else

echo "$name stop failed; still running."

return 1 # stop timed out and not forced

fi

else

echo "$name stopped."

fi

fi

}

status() {

if [ -f "$pidfile" ] ; then

pid=cat "$pidfile"

if kill -0 $pid > /dev/null 2> /dev/null ; then

-# process by this pid is running.

-# It may not be our pid, but that's what you get with just pidfiles.

-# TODO(sissel): Check if this process seems to be the same as the one we

-# expect. It'd be nice to use flock here, but flock uses fork, not exec,

-# so it makes it quite awkward to use in this case.

return 0

else

return 2 # program is dead but pid file exists

fi

else

return 3 # program is not running

fi

}

reload() {

if status ; then

kill -HUP cat "$pidfile"

fi

}

force_stop() {

if status ; then

stop

status && kill -KILL cat "$pidfile"

fi

}

configtest() {

-# Check if a config file exists

if [ ! "$(ls -A ${LS_CONF_DIR}/* 2> /dev/null)" ]; then

echo "There aren't any configuration files in ${LS_CONF_DIR}"

return 1

fi

HOME=${LS_HOME}

export PATH HOME

test_args="--configtest -f ${LS_CONF_DIR} ${LS_OPTS}"

$program ${test_args}

[ $? -eq 0 ] && return 0

-# Program not configured

return 6

}

case "$1" in

start)

status

code=$?

if [ $code -eq 0 ]; then

echo "$name is already running"

else

start

code=$?

fi

exit $code

;;

stop) stop ;;

force-stop) force_stop ;;

status)

status

code=$?

if [ $code -eq 0 ] ; then

echo "$name is running"

else

echo "$name is not running"

fi

exit $code

;;

reload) reload ;;

restart)

quiet configtest
RET=$?
if [ ${RET} -ne 0 ]; then
  echo "Configuration error. Not restarting. Re-run with configtest parameter for details"
  exit ${RET}
fi
stop && start
;;

configtest)

configtest

exit $?

;;

*)

echo "Usage: $SCRIPTNAME {start|stop|force-stop|status|reload|restart|configtest}" >&2

exit 3

;;

esac

exit $?

分析的日志类型：

1.系统日志：/var/log下的所有的内容，google每一个文件的内容

2.通过elasticsearch分析某一个访问记录

3.错误日志，收集后反馈给开发

4.系统运行日志

5.其他类型的日志

日志的字段划分：

1.gork模块：通过正则表达式，比较复杂，而且当数据大的时候会占用CPU

2.Json,简单易用

3.将nginx的日志设置为json模式

安装kibana

a.wget https://download.elastic.co/kibana/kibana/kibana-4.5.4-1.x86_64.rpm

b.安装

c.vi /opt/kibana/config/kibana.yml

server.port:5601

server.host:’0.0.0.0’

elasticsearch.url:’http://xxx.xxx.xx.xx:9200’

d.service kibana start

e.chkconfig –add kibana

f.访问网页：http://localhost:5601

常用模块：

1.系统日志收集--->syslog：配置syslog结果写入到elasticsearch,指定端口514，主机就是要收集日志的服务器IP地址

2.访问日志：nginx转换成json格式

3.错误日志：使用codec插件：

Input {

Stdin {

Codec =>multiline {

Pattern => “^\s”

Negate => “false”

What => “previous”

}

}

}

Pattern:使用正则表达式匹配文件.

Negate的默认值为false,当设置为true的时候，不匹配pattern的信息会继续执行what的内容.

What：值为previous或next：将匹配到的信息合并到前一行还是下一行.

4.运行日志codec =>json,如果不是json要使用gork进行匹配

在地图显示IP的访问次数统计：

1.在easticsearch服务器用户家目录下载一个filebeat：

2.加载模板：

$curl -XPUT 'http://10.26.44.42:9200/_template/filebeat?pretty' -d@/etc/filebeat/filebeat.template.json

$curl -XPUT 'http://10.26.44.42:9200/_template/filebeat?pretty' -d@/etc/filebeat/filebeat.template-es2x.json

$curl -XPUT 'http://10.26.44.42:9200/_template/filebeat?pretty' -d@/root/filebeat.template.json

3.下载GeoIP数据库文件：

$cd /opt/logstash

$curl -o “http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz”

$gunzip GeoLiteCity.dat.gz

4.配置logstash使用GeoIP:

input {

redis {

data_type => "list"

key => "mobile-tomcat-access-log"

host => "192.168.0.251"

port => "6379"

db => "0"

codec => "json"

}

}

--#input部分为从redis读取客户端logstash分析提交后的访问日志

filter {

if [type] == "mobile-tomcat" {

geoip {

source => "client" --client 是客户端logstash收集日志时定义的公网IP的key名称，一定要和实际名称一致，因为要通过此名称获取到其对于的ip地址

target => "geoip"

database => "/etc/logstash/GeoLiteCity.dat"

add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]

add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]

}

mutate {

convert => [ "[geoip][coordinates]", "float"]

}

}

}

output {

if [type] == "mobile-tomcat" {

elasticsearch {

hosts => ["192.168.0.251"]

manage_template => true

index => "logstash-mobile-tomcat-access-log-%{+YYYY.MM.dd}" --index的名称一定要是logstash开头的，否则会在使用地图的时候出现geoIP type无法找找到的类似错误

flush_size => 2000

idle_flush_time => 10

}

}

}

5.在kibana界面添加新的索引：

visualize---->Tile map---->From a new search---->Select a index patterm--->选择之前的index---->Geo coordinates

【参考文档：】

1.https://www.elastic.co/guide/index.html

2.http://www.ttlsa.com/elk/howto-install-elasticsearch-logstash-and-kibana-elk-stack/

3.https://www.elastic.co/guide/en/logstash/current/plugins-inputs-log4j.html

4.http://blog.chinaunix.net/xmlrpc.php?r=blog/article&uid=21142030&id=5671032

1. http://blog.csdn.net/super_scan/article/details/45694289

6.http://517sou.net/archives/centos下使用elk套件搭建日志分析和监控平台/

问题：

1.重新启动elasticsearch后，报错：Elasticsearch is still initializing the kibana index.

解决：curl -XDELETE http://localhost:9200/.kibana

---上述方法会丢失所有的kibana配置，索引、图、仪表板，如果只是区分索引，可使用以下方法：

curl -s http://localhost:9200/.kibana/_recovery?pretty

curl -XPUT 'localhost:9200/.kibana/_settings' -d '

{

"index" : {

"number_of_replicas" : 0

}

}'

修改后还有报错的话，重启kibana.

哈哈！忘记重启elasticsearch,导致页面索引丢失，没有数据.

添加索引模板：

$curl -XPUT 'http://10.26.44.42:9200/_template/filebeat?pretty' -d@/root/filebeat.template.json

模板文件：

Vim /root/fillebeat.template.json

{

"mappings": {

"default": {

"_all": {

"enabled": true,

"norms": {

"enabled": false

}

},

"dynamic_templates": [

{

"template1": {

"mapping": {

"doc_values": true,

"ignore_above": 1024,

"index": "not_analyzed",

"type": "{dynamic_type}"

},

"match": ""

}

}

],

"properties": {

"geoip": {

"properties" : {

"location": {

"type": "geo_point"

},

"ip": { "type": "ip" },

"coordinates": { "type": "geo_point" }

}},

"@timestamp": {

"type": "date"

},

"message": {

"type": "string",

"index": "analyzed"

},

"offset": {

"type": "long",

"doc_values": "true"

}

}

}

},

"settings": {

"index.refresh_interval": "5s"

},

"template": "filebeat-"

}

查看集群的状态：

$ curl -XGET 'http://10.26.44.42:9200/_cluster/health?pretty=true'

{

"cluster_name" : "elks",

"status" : "red",

"timed_out" : false,

"number_of_nodes" : 3,

"number_of_data_nodes" : 3,

"active_primary_shards" : 5269,

"active_shards" : 6812,

"relocating_shards" : 0,

"initializing_shards" : 6,

"unassigned_shards" : 4151,

"delayed_unassigned_shards" : 0,

"number_of_pending_tasks" : 5136,

"number_of_in_flight_fetch" : 0,

"task_max_waiting_in_queue_millis" : 4711822,

"active_shards_percent_as_number" : 62.10228826693409

}

查看unassigned_shards:

$curl -s 'http://10.26.44.42:9200/_cat/shards' | grep UNASSIGNED | awk '{print $1}' | sort | uniq

elk集群存在问题：单节点删除过索引

将unassigned_shards删除后，重启elasticsearch，服务状态正常.

将unassigned_shards清除：

curl -XPUT 'localhost:9200/_all/_settings?pretty' -H 'Content-Type: application/json' -d'

{

"settings": {

"index.unassigned.node_left.delayed_timeout": "0"

}

}

'