ELK之logstash收集日志写入redis及读取redis

logstash->redis->logstash->elasticsearch

1.安装部署redis

cd /usr/local/src
wget http://download.redis.io/releases/redis-3.2.8.tar.gz
tar xf redis-3.2.8.tar.gz
cd redis-3.2.8/
make
ln -s /usr/local/src/redis-3.2.8 /usr/local/redis
cd /usr/local/redis/

vim redis.conf
bind 10.0.0.22
daemonize yes
save ""
#save 900 1
#save 300 10
#save 60 10000
requirepass root123

cp src/redis-server /usr/bin/
cp src/redis-cli /usr/bin/
redis-server /usr/local/redis/redis.conf

登录redis需要认证

配置logstash的systemlog_to_redis.conf

vim systemlog_to_redis.conf
input {
  file {
    path => "/var/log/messages"
    type => "systemlog"
    start_position => "beginning"
    stat_interval => "2"
  }
}

output {
  if [type] == "systemlog" {
    redis {
      data_type => "list"
      host => "10.0.0.22"
      db => "1"
      port => "6379"
      password => "root123"
      key => "systemlog"
    }
  }
}
systemctl restart logstash
# 手动写入messages日志
cat /etc/hosts >> /var/log/messages
echo "helloword" >> /var/log/messages

登陆redis查看

2.配置logstash从reids中取出数据到elasticsearch

# 使用linux-elk2(10.0.0.33)上的logstash从redis取数据
vim redis-es.conf
input {
  redis {
    data_type => "list"
    host => "10.0.0.22"
    db => "1"
    port => "6379"
    key => "systemlog"
    password => "root123"
  }
}

output {
  elasticsearch {
    hosts => ["10.0.0.33:9200"]
    index => "redis-systemlog-%{+YYYY.MM.dd}"
  }
}
systemctl restart logstash

logstash统计日志,有两个以上的key时,就必须加判断

收集日志写入redis及读取redis:http://blog.51cto.com/jinlong/2056563