ELK-全过程搭建
环境说明:
软件包我都 给你们放/usr/local/src/elk目录下
安装目录都放在/usr/local/下
数据都放在/data0/elk/目录下
日志都放在/data0/logs/elk目录下
系统 服务器地址 服务分布
CentOS7 192.168.9.176 Filebeat+Logstash+Elasticsearch(含head插
件)+kibana
CentOS7 192.168.9.232 Filebeat+Logstash+Elasticsearch
CentOS7 192.168.9.234 Filebeat+Logstash+Elasticsearch
一,安装Elasticsearch
1,安装JAVA环境
(1)下载jdk:
http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
(2)查看系统是否安装低版本
java -version
#如果低于8请卸载
yum remove -y java
yum groupremove -y java
(3)rpm安装jdk
rpm -ivh jdk-8u161-linux-x64.rpm
echo "JAVA_HOME=/usr/java/jdk1.8.0_161
JRE_HOME=/usr/java/jdkjdk1.8.0_161/jre
PATH=${PATH}:${JAVA_HOME}/bin:${JRE_HOME}/bin
CLASSPATH=:${JAVA_HOME}/lib/dt.jar:${JAVA_HOME}/lib/tools.jar:${JRE_HOME}/lib
export JAVA_HOME JRE_HOME PATH CLASSPATH" >> /etc/profile.d/java.sh
source /etc/profile.d/java.sh
2,安装ELasticsearch
(1)安装elasticsearch:
cd /usr/local/src
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.3.tar.gz
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.3.tar.gz.sha512
#shasum -a 512 -c elasticsearch-6.2.3.tar.gz.sha512
sha512sum -c elasticsearch-6.2.3.tar.gz.sha512
tar -xzf elasticsearch-6.2.3.tar.gz
mv elasticsearch-6.2.3 /usr/local/
注意:Centos6不支持SecComp,而ES6默认bootstrap.system_call_filter为true,需要禁用。
禁用方法:在elasticsearch.yml中配置bootstrap.system_call_filter为false,注意要在Memory下面:
取消bootstrap.memory_lock的注释,添加bootstrap.system_call_filter 配置
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
(2)配置优化elasticsrach
1)#jvm启动参数优化:/usr/local/elasticsearch-6.2.3/config/jvm.options
-Xms1g
-Xmx1g
#初始和最大最好一致,避免GC
#Xms代表总的堆空间的初始大小
#Xmx表示总的堆空间的最大大小
2)elasticsearch配置文件/usr/local/elasticsearch-6.2.3/config/elasticsearch.yml
#集群名称
cluster.name: sinashow-elk
#节点名称
node.name: sinashow-elk-192-168-9-176
#锁定内存
bootstrap.memory_lock: true
#通信绑定地址和http端口
network.host: 192.168.9.176
http.port: 9200
#数据和日志存放目录
logs: /data0/es
data: /data0/es
discovery.zen.ping.unicast.hosts: ["192.168.9.176"]
#增加参数,使head插件可以访问es
http.cors.enabled: true
http.cors.allow-origin: "*"
3)系统设置更改:
echo "es soft nofile 65536
es hard nofile 131072
es soft memlock unlimited
es hard memlock unlimited
es soft nproc 4096
es hard nproc 4096" >> /etc/security/limits.conf
sysctl -w vm.max_map_count=262144
echo "vm.max_map_count=262144" >> /etc/sysctl.conf
#查看GET _nodes/stats/process?filter_path=**.max_file_descriptors
禁用swap分区:
swapoff -a
更改/etc/fstab
#查看GET _nodes?filter_path=**.mlockall
(3)安装head插件
1)安装node
cd /usr/local/src/elk
wget https://npm.taobao.org/mirrors/node/latest-v4.x/node-v4.4.7-linux-x64.tar.gz
tar -zxvf node-v4.4.7-linux-x64.tar.gz
#配置环境变量
# vim /etc/profile.d/node.sh
export NODE_HOME=/usr/local/src/elk/node-v4.4.7-linux-x64
export PATH=${PATH}:${NODE_HOME}/bin
export NODE_PATH=${NODE_HOME}/lib/node_modules
#加载环境变量
source /etc/profile.d/node.sh
2)安装grunt
npm install -g cnpm --registry=https://registry.npm.taobao.org
npm install -g grunt
npm install -g grunt-cli --registry=https://registry.npm.taobao.org --no-proxy
node -v
npm -v
grunt -version
3)下载head插件
wget https://github.com/mobz/elasticsearch-head/archive/master.zip
unzip master.zip
npm install
#npm install -g cnpm --registry=https://registry.npm.taobao.org
#cnpm install
4)修改Gruntfile.js
connect: {
server: {
options: {
port: 9100,
base: '.',
keepalive: true,
hostname: '*'
5)修改连接地址_site/app.js
this.base_uri = this.config.base_uri || this.prefs.get("app-base_uri") || "http://192.168.9.176:9200";
(4)启Elasticsearch并测试
groupadd es
useradd es -g es
passwd es
chown -R es.es /usr/lcoal/elasticsearch-6.2.3
mkdir -p /data0/es
chown -R es.es /usr/local/elasticsearch-6.2.3/
chown -R es.es /data0/es/
su es
cd /usr/lcoal/elasticsearch-6.2.3
./bin/elasticsearch
# curl -i "http://localhost:9200/"
HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-length: 435
{
"name" : "fS0IUJs",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "NmnxM5hkSVKFDId9-pC-CA",
"version" : {
"number" : "6.2.2",
"build_hash" : "10b1edd",
"build_date" : "2018-02-16T19:01:30.685723Z",
"build_snapshot" : false,
"lucene_version" : "7.2.1",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}
(5)启动head插件
/usr/local/src/elk/elasticsearch-head-master
nohup grunt server >/dev/null 2>&1 &
二,安装kibana环境
1,下载安装kibana
wget https://artifacts.elastic.co/downloads/kibana/kibana-6.2.3-linux-x86_64.tar.gz
sha1sum kibana-6.2.3-linux-x86_64.tar.gz
tar -xzf kibana-6.2.3-linux-x86_64.tar.gz
cd kibana-6.2.3-linux-x86_64/
2,配置kibana /usr/local/kibana-6.2.3-linux-x86_64/config/kibana.yml
#监听端口
server.port: 5601
#监听地址
server.host: "192.168.9.176"
#服务名称
server.name: "sinashow-kibana"
#连接Elasticsearch地址
elasticsearch.url: "http://192.168.9.176:9200"
#超时时间
elasticsearch.pingTimeout: 5000
elasticsearch.requestTimeout: 60000
3,启动
cd /usr/local/kibana-6.2.3-linux-x86_64
nohup ./bin/kibana >dev/null 2>&1 &
4,测试
curl -I "http://192.168.9.176:5601"
三,kibana和head插件配置nginx代理访问控制
1,nginx环境安装
(1)安装pcre-devel,Nginx支持HTTP Rewrite
yum -y install gcc gcc-c++ autoconf automake zlib zlib-devel \
pcre pcre-devel openssl bzip2-devel libxml2 libxml2-devel curl-devel \
libjpeg libjpeg-devel libpng libpng-devel openssl-devel libevent libevent-devel
(2)安装Nginx
# 首先添加用户nginx,实现以之运行nginx服务进程:
groupadd -r nginx
useradd -r -g nginx -s /sbin/nologin nginx
# 接着开始编译和安装:
wget http://nginx.org/download/nginx-1.12.2.tar.gz
tar xf nginx-1.12.2.tar.gz
cd nginx-1.12.2
./configure \
--prefix=/usr/local/nginx \
--user=nginx \
--group=nginx \
--with-http_ssl_module \
--with-http_flv_module \
--with-file-aio \
--with-http_stub_status_module \
--with-http_gzip_static_module \
--with-pcre \
--with-stream \
--with-http_mp4_module
make && make install
(3)Nginx日志切割
0 0 * * * cd /data0/logs/ && /bin/sh cut_nginx_log.sh >/dev/null 2>&1
# mkdir -p /data0/logs/nginx
# cat /data0/logs/cut_nginx_log.sh
#!/bin/bash
logs_path="/data0/logs/nginx"
YDATE=`date +"%Y"`
MDATE=`date +"%m"`
DDATE=`date -d"1 day ago" +"%d"`
HDATE=`date +"%H"`
mkdir -p ${logs_path}/${YDATE}/${MDATE}/${DDATE}/
yesterday=`date -d "yesterday" +"%Y%m%d"`
for logname in `find ${logs_path} -maxdepth 1 -name "*.log"`
do
domain=`echo $logname | sed "s#/data0/logs/nginx/##"`
mv ${logs_path}/$domain
${logs_path}/${YDATE}/${MDATE}/${DDATE}/$domain\_${yesterday}.log
done
/usr/local/nginx/sbin/nginx -s reload
find ${logs_path} -name "*.log" -mtime +15 -exec rm -fr {} \;
2,Kibana配置nginx代理访问控制:
(1)nginx主配置文件
# cat /usr/local/nginx/conf/nginx.conf
user nginx;
worker_processes 8;
worker_rlimit_nofile 102400;
error_log /data0/logs/nginx/error.log notice;
pid /data0/logs/nginx/nginx.pid;
events {
use epoll;
worker_connections 65535;
}
http {
include mime.types;
default_type application/octet-stream;
log_format nginx_log '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
client_max_body_size 50m;
client_body_buffer_size 256k;
client_header_timeout 120s;
client_body_timeout 120s;
send_timeout 1m;
sendfile on;
keepalive_timeout 120;
proxy_ignore_client_abort on;
proxy_connect_timeout 600;
proxy_read_timeout 600;
proxy_send_timeout 600;
proxy_buffer_size 32k;
proxy_buffers 8 128k;
proxy_busy_buffers_size 128k;
proxy_temp_file_write_size 512k;
gzip on;
server {
listen 80 default_server;
server_name _;
return 404;
}
include vhosts/*.conf;
}
(2)/usr/local/nginx/conf/vhosts/es-head.sinashow.com.conf
upstream es-head.sinashow.com {
#server 127.0.0.1:9200 max_fails=3 fail_timeout=1000s;
server 192.168.9.176:9100 max_fails=3 fail_timeout=1000s;
ip_hash;
}
server {
listen 80;
server_name es-head.sinashow.com;
location / {
root html;
index index.html index.htm;
proxy_pass http://es-head.sinashow.com;
allow 111.198.228.124;
deny all;
auth_basic "closed site";
auth_basic_user_file /usr/local/nginx/conf/.htpasswd;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
client_max_body_size 512k;
access_log /data0/logs/nginx/es-head.sinashow.com.log nginx_log;
}
}
(3)/usr/local/nginx/conf/vhosts/es-kibana.sinashow.com.conf
upstream es-kibana.sinashow.com {
#server 127.0.0.1:9200 max_fails=3 fail_timeout=1000s;
server 192.168.9.176:5601 max_fails=3 fail_timeout=1000s;
ip_hash;
}
server {
listen 80;
server_name es-kibana.sinashow.com;
location / {
root html;
index index.html index.htm;
proxy_pass http://es-kibana.sinashow.com;
allow 111.198.228.124;
deny all;
auth_basic "closed site";
auth_basic_user_file /usr/local/nginx/conf/.htpasswd;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
client_max_body_size 512k;
access_log /data0/logs/nginx/es-kibana.sinashow.com.log nginx_log;
}
}
(4)创建nginx访问控制账号
yum install -y httpd
htpasswd -c /usr/local/nginx/conf/.htpasswd esuser
3,启动nginx并测试
#检测
/usr/local/nginx/sbin/nginx -t
#启动
/usr/local/nginx/sbin/nginx
#停止
/usr/local/nginx/sbin/nginx -s stop
#重载
/usr/local/nginx/sbin/nginx -s reload
测试:
http://es-kibana.sinashow.com
http://es-head.sinashow.com
四,安装Logstash环境
1,安装logstash
wget https://artifacts.elastic.co/downloads/logstash/logstash-6.2.3.tar.gz
tar xf logstash-6.2.3.tar.gz
cd logstash-6.2.3
./bin/logstash -e 'input { stdin { } } output { stdout {} }'
2,配置logstash /usr/local/logstash-6.2.3/config/logstash.yml
input {
beats {
port => 5044
}
}
output {
elasticsearch {
hosts => "192.168.9.176:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
3,启动logstash
cd /usr/local/logstash-6.2.3/
五,安装Filebeat环境
1,rpm包安装
curl -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.2.3-x86_64.rpm
rpm -vi filebeat-6.2.3-x86_64.rpm
2,配置filebeat /etc/filebeat/filebeat.yml
filebeat.prospectors:
- type: log
enabled: true
paths:
- /var/log/*.log
output.logstash:
hosts: ["192.168.9.176:5044"]
3,启动filebeat
systemctl start filebeat.service
ELK-全过程搭建的更多相关文章
- ELK+redis搭建nginx日志分析平台
ELK+redis搭建nginx日志分析平台发表于 2015-08-19 | 分类于 Linux/Unix | ELK简介ELKStack即Elasticsearch + Logstas ...
- 使用elk+redis搭建nginx日志分析平台
elk+redis 搭建nginx日志分析平台 logstash,elasticsearch,kibana 怎么进行nginx的日志分析呢?首先,架构方面,nginx是有日志文件的,它的每个请求的状态 ...
- linux下利用elk+redis 搭建日志分析平台教程
linux下利用elk+redis 搭建日志分析平台教程 http://www.alliedjeep.com/18084.htm elk 日志分析+redis数据库可以创建一个不错的日志分析平台了 ...
- ELK初学搭建(kibana)
ELK初学搭建(kibana) elasticsearch logstash kibana ELK初学搭建 kibana 1.环境准备 centos6.8_64 mini IP:192.168.10. ...
- ELK初学搭建(elasticsearch)
ELK初学搭建(elasticsearch) elasticsearch logstash kibana ELK初学搭建 elasticsearch 1.环境准备 centos6.8_64 mini ...
- ELK初学搭建(logstash)
ELK初学搭建(logstash) elasticsearch logstash kibana ELK初学搭建 logstash 1.环境准备 centos6.8_64 mini IP:192.168 ...
- ELK平台搭建(上)
一.目的 为指导在Centos6.8系统下搭建标准ELK平台的工作. 二.定义 Elasticsearch Logstash Kibana结合Redis协同工作. 三.适用范围 适用于运营维护组运维工 ...
- ELK 环境搭建4-Kafka + zookeeper
一.安装前准备 1.节点 192.168.30.41 192.168.30.42 192.168.30.43 2.操作系统: Centos7.5 3.安装包 a.java8: jdk-8u181-li ...
- ELK 环境搭建3-Logstash
一.Logstash是一款轻量级的日志搜集处理框架,可以方便的把分散的.多样化的日志搜集起来,并进行自定义的处理,然后传输到指定的位置,比如某个服务器或者文件或者中间件. 二.搭建 1.因为要涉及到收 ...
- ELK 环境搭建2-Kibana
一.安装前准备 1.节点 192.168.30.41 2.操作系统: Centos7.5 3.安装包 a.java8: jdk-8u181-linux-x64.tar.gz b.Kibana kiba ...
随机推荐
- 创建express项目(nodejs)
1.下载nodejs安装包 nodejs官网下载最新版本就行,网址:http://nodejs.cn/download/,点击自己适用的系统,自动下载跟电脑操作系统位数符合的安装包, 2.配置环境 最 ...
- LVS之NAT模式
目录: 网络环境 LVS服务器配置 LVS服务器添加ipvs规则 RS服务器配置 访问验证 抓包分析 注意事项 [网络环境] 服务器类型 IP 说明 lvs_vip 192.168.2.130 vip ...
- ElasticSearch1:基本概念
ElasticSearch的基本概念 es基本概念: Elasticsearch是面向文档型数据库,一条数据在这里就是一个文档,用JSON作为文档序列化的格式 NRT:Nearly Real Time ...
- legend3---9、项目的日志以及调试信息数据量非常大
legend3---9.项目的日志以及调试信息数据量非常大 一.总结 一句话总结: legend2我开发调试,最近竟然发现日志等的信息有1.5G,数据量实在太大 1.juqery如何找后代? chil ...
- Java访问Oracle服务器
Java访问Oracle服务器--orcl数据库---emp表 private static String driver = "oracle.jdbc.driver.Orac ...
- Nginx知识
OpenResty最佳实践->location匹配规则传说中图片防盗链的爱恨情仇
- JAVA反射机制,总结下
1.总体概述 Java反射机制指的是在Java程序运行状态中,对于任何一个类,都可以获得这个类的所有属性和方法:对于给定的一个对象,都能够调用它的任意一个属性和方法.这种动态获取类的内容以及动态调用对 ...
- 转 MySQL: Starting MySQL….. ERROR! The server quit without updating PID file解决办法
http://blog.sina.com.cn/s/blog_637e04c9010117ri.html 1 问题 [root@localhost mysql]# /etc/rc.d/init.d/m ...
- MongoDB Input
Configure Connection Tab Host name(s) or IP address(es):网络名称或者地址.可以输入多个主机名或IP地址,用逗号分隔.还可以通过将主机名和端口号与 ...
- DELPHI ClientData使用详解
在三层结构中,TClientDataSet的地位是不可估量的,她的使用正确与否,是十分关键的,本文从以下几个方面阐述她的使用,希望对你有所帮助. 1.动态索引procedure TForm1.DBGr ...