kubernetes 1.14安装部署EFK日志收集系统

简单介绍：

EFK 组合插件是k8s项目的一个日志解决方案，它包括三个组件：Elasticsearch, Fluentd, Kibana。相对于ELK这样的架构，k8s官方推行了EFK，可能Fluentd相对于Logstash更加轻量级吧。

Elasticsearch 是日志存储和日志搜索引擎，Fluentd 负责把k8s集群的日志发送给 Elasticsearch, Kibana 则是可视化界面查看和检索存储在 Elasticsearch 的数据。

GitHub官网：https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch

特别补充：

Kubernetes官方推荐的EFK组合，其实对于我们大多数公司来说并不合适，其中一点原因就是这个组合需要应用日志打印到标准输出。

推荐做法：还是使用传统的ELK组合，ELK可以部署在K8S集群之外，日志写入指定存储目录文件中，并采用filebeat方式采集。

测试环境：

System	Hostname	IP
CentOS 7.6	k8s-master	138.138.82.14
CentOS 7.6	k8s-node1	138.138.82.15
CentOS 7.6	k8s-node2	138.138.82.16

注：以下是简单的安装部署操作，实际用于生产环境的话还要根据不同的后端存储进而修改yaml中的相应存储路径。

操作步骤：

1. 获取yaml文件

从github官网下载：https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch

或者下载整个tar包：https://storage.googleapis.com/kubernetes-release/release/v1.14.1/kubernetes-server-linux-amd64.tar.gz

解压后yanl文件位置： kubernetes/cluster/addons/fluentd-elasticsearch/

复制yaml文件到另一处进行修改操作：

[root@k8s-master ~]# mkdir efk

[root@k8s-master ~]# cp kubernetes/cluster/addons/fluentd-elasticsearch/*.yaml  efk/

[root@k8s-master ~]# cd efk/

[root@k8s-master efk]# ll

总用量 36

-rw-r--r-- 1 root root   382 5月   5 13:55 es-service.yaml

-rw-r--r-- 1 root root  2906 5月   5 14:44 es-statefulset.yaml

-rw-r--r-- 1 root root 16124 5月   5 13:55 fluentd-es-configmap.yaml

-rw-r--r-- 1 root root  2702 5月   5 13:55 fluentd-es-ds.yaml

-rw-r--r-- 1 root root  1052 5月   5 13:55 kibana-deployment.yaml

-rw-r--r-- 1 root root   354 5月   5 13:55 kibana-service.yaml

2. 准备镜像（在所有主机上）

注：因为被墙的缘故，加上镜像文件比较大，启动会费时，可以提前单独下载好所需镜像。

①修改 es-statefulset.yaml （https://www.docker.elastic.co/#）

②修改 fluentd-es-ds.yaml

③无需修改 kibana-deployment.yaml

说明：关于kibana镜像以-oss结尾的版本好奇（default release that contains the commercially licensed code. The release that contains only open source will have the -oss appended to it.）：

https://www.elastic.co/guide/en/kibana/6.0/docker.html

https://discuss.elastic.co/t/what-are-the-differences-between-the-kibana-oss-and-non-oss-build/152364/3

附上镜像拉取文本：

~]# docker pull docker.elastic.co/elasticsearch/elasticsearch:6.6.

~]# docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/fluentd-elasticsearch:v2.4.0

~]# docker pull docker.elastic.co/kibana/kibana-oss:6.6.

3. 发布yaml文件到k8s中

[root@k8s-master efk]# kubectl apply -f .

4. 查看各pod状态

[root@k8s-master efk]# kubectl get po -n kube-system -o wide

NAME                                    READY   STATUS    RESTARTS   AGE     IP              NODE         NOMINATED NODE   READINESS GATES

calico-node-b78m4                       /     Running             7d1h    138.138.82.16   k8s-node2    <none>           <none>

calico-node-r5mlj                       /     Running             7d1h    138.138.82.14   k8s-master   <none>           <none>

calico-node-z5tdh                       /     Running             7d1h    138.138.82.15   k8s-node1    <none>           <none>

coredns-fb8b8dccf-6mgks                 /     Running             7d1h    192.168.0.7     k8s-master   <none>           <none>

coredns-fb8b8dccf-cbtlx                 /     Running             7d1h    192.168.0.6     k8s-master   <none>           <none>

elasticsearch-logging-0                 /     Running             10m     192.168.2.11    k8s-node2    <none>           <none>

elasticsearch-logging-1                 /     Running             10m     192.168.1.17    k8s-node1    <none>           <none>

etcd-k8s-master                         /     Running             7d1h    138.138.82.14   k8s-master   <none>           <none>

fluentd-es-v2.4.0-fm9n2                 /     Running             10m     192.168.1.16    k8s-node1    <none>           <none>

fluentd-es-v2.4.0-xjx6h                 /     Running             10m     192.168.2.10    k8s-node2    <none>           <none>

kibana-logging-f4d99b69f-hc4h9          /     Running             10m     192.168.1.15    k8s-node1    <none>           <none>

kube-apiserver-k8s-master               /     Running             7d1h    138.138.82.14   k8s-master   <none>           <none>

kube-controller-manager-k8s-master      /     Running             7d1h    138.138.82.14   k8s-master   <none>           <none>

kube-proxy-c9xd2                        /     Running             7d1h    138.138.82.14   k8s-master   <none>           <none>

kube-proxy-fp2r2                        /     Running             7d1h    138.138.82.15   k8s-node1    <none>           <none>

kube-proxy-lrsw7                        /     Running             7d1h    138.138.82.16   k8s-node2    <none>           <none>

kube-scheduler-k8s-master               /     Running             7d1h    138.138.82.14   k8s-master   <none>           <none>

kubernetes-dashboard-5f7b999d65-45vlk   /     Running             6d7h    192.168.1.14    k8s-node1    <none>           <none>

metrics-server-7579f696d8-pgcc4         /     Running             6d22h   192.168.1.12    k8s-node1    <none>           <none>

顺利的话，稍等一小会就成功running状态了。如果有问题，就查看容器日志，基本上可以定位到所有错误原因。

5. 查看kibana

留意kibana的日志输出，这里给出我的实际一段：

[root@k8s-master efk]# kubectl logs -n kube-system kibana-logging-f4d99b69f-hc4h9 -f

{"type":"log","@timestamp":"2019-05-05T08:02:58Z","tags":["warning","config","deprecation"],"pid":,"message":"You should set server.basePath along with server.rewriteBasePath. Starting in 7.0, Kibana will expect that all requests start with server.basePath rather than expecting you to rewrite the requests in your reverse proxy. Set server.rewriteBasePath to false to preserve the current behavior and silence this warning."}

{"type":"log","@timestamp":"2019-05-05T08:02:58Z","tags":["plugin","warning"],"pid":,"path":"/usr/share/kibana/src/legacy/core_plugins/ems_util","message":"Skipping non-plugin directory at /usr/share/kibana/src/legacy/core_plugins/ems_util"}

{"type":"log","@timestamp":"2019-05-05T08:02:59Z","tags":["warning","elasticsearch","config","deprecation"],"pid":,"message":"Config key \"url\" is deprecated. It has been replaced with \"hosts\""}

{"type":"log","@timestamp":"2019-05-05T08:03:00Z","tags":["status","plugin:kibana@6.6.1","info"],"pid":,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}

{"type":"log","@timestamp":"2019-05-05T08:03:00Z","tags":["status","plugin:elasticsearch@6.6.1","info"],"pid":,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}

{"type":"log","@timestamp":"2019-05-05T08:03:00Z","tags":["status","plugin:console@6.6.1","info"],"pid":,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}

{"type":"log","@timestamp":"2019-05-05T08:03:00Z","tags":["error","elasticsearch","admin"],"pid":,"message":"Request error, retrying\nHEAD http://elasticsearch-logging:9200/ => connect ECONNREFUSED 10.100.54.183:9200"}

{"type":"log","@timestamp":"2019-05-05T08:03:00Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:00Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:00Z","tags":["status","plugin:elasticsearch@6.6.1","error"],"pid":,"state":"red","message":"Status changed from yellow to red - Unable to connect to Elasticsearch.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}

{"type":"log","@timestamp":"2019-05-05T08:03:01Z","tags":["status","plugin:interpreter@6.6.1","info"],"pid":,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}

{"type":"log","@timestamp":"2019-05-05T08:03:01Z","tags":["status","plugin:metrics@6.6.1","info"],"pid":,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}

{"type":"log","@timestamp":"2019-05-05T08:03:01Z","tags":["status","plugin:timelion@6.6.1","info"],"pid":,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}

{"type":"log","@timestamp":"2019-05-05T08:03:03Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:03Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:06Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:06Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:08Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:08Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:11Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:11Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:13Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:13Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:16Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:16Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:18Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:18Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:21Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:21Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:23Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:23Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:26Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:26Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:28Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:28Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:31Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:31Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:33Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:33Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:36Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:36Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:38Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:38Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:41Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:41Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:44Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:44Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:46Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:46Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:49Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:49Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:51Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:51Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:54Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:54Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:56Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:56Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:59Z","tags":["error","elasticsearch","admin"],"pid":,"message":"Request error, retrying\nHEAD http://elasticsearch-logging:9200/ => connect ECONNREFUSED 10.100.54.183:9200"}

{"type":"log","@timestamp":"2019-05-05T08:03:59Z","tags":["warning"],"pid":,"kibanaVersion":"6.6.1","nodes":[{"version":"6.6.2","http":{"publish_address":"192.168.2.11:9200"},"ip":"192.168.2.11"}],"message":"You're running Kibana 6.6.1 with some different versions of Elasticsearch. Update Kibana or Elasticsearch to the same version to prevent compatibility issues: v6.6.2 @ 192.168.2.11:9200 (192.168.2.11)"}

{"type":"log","@timestamp":"2019-05-05T08:03:59Z","tags":["status","plugin:elasticsearch@6.6.1","info"],"pid":,"state":"green","message":"Status changed from red to green - Ready","prevState":"red","prevMsg":"Unable to connect to Elasticsearch."}

{"type":"log","@timestamp":"2019-05-05T08:03:59Z","tags":["info","migrations"],"pid":,"message":"Creating index .kibana_1."}

{"type":"log","@timestamp":"2019-05-05T08:04:00Z","tags":["info","migrations"],"pid":,"message":"Pointing alias .kibana to .kibana_1."}

{"type":"log","@timestamp":"2019-05-05T08:04:00Z","tags":["info","migrations"],"pid":,"message":"Finished in 994ms."}

{"type":"log","@timestamp":"2019-05-05T08:04:00Z","tags":["listening","info"],"pid":,"message":"Server running at http://0:5601"}

以上显示了启动成功的一段过程。

查看kibana地址

[root@k8s-master ~]# kubectl cluster-info |grep kibana

Kibana is running at https://138.138.82.14:6443/api/v1/namespaces/kube-system/services/kibana-logging/proxy

在浏览器中输入以上链接即可，效果如下：

EFK插件简单部署到此结束，实际生产应用还需修改yaml文件相关参数。

结束.

kubernetes 1.14安装部署EFK日志收集系统的更多相关文章

Kubernetes 系列（八）：搭建EFK日志收集系统
Kubernetes 中比较流行的日志收集解决方案是 Elasticsearch.Fluentd 和 Kibana(EFK)技术栈,也是官方现在比较推荐的一种方案. Elasticsearch 是一个 ...
十九，基于helm搭建EFK日志收集系统
目录 EFK日志系统一,EFK日志系统简介: 二,EFK系统部署 1,EFK系统部署方式 2,基于Helm方式部署EFK EFK日志系统一,EFK日志系统简介: 关于系统日志收集处理方案,其实有很 ...
Docker搭建EFK日志收集系统，并自定义es索引名
EFK架构图一.EFK简介 EFK不是一个软件,而是一套解决方案,并且都是开源软件,之间互相配合使用,完美衔接,高效的满足了很多场合的应用,是目前主流的一种日志系统. EFK是三个开源软件的缩写,分 ...
kubernetes 1.14安装部署dashboard
简单介绍: Dashboard是一个基于web的Kubernetes用户界面.您可以使用Dashboard将容器化应用程序部署到Kubernetes集群,对容器化应用程序进行故障诊断,并管理集群资源. ...
kubernetes 1.14安装部署metrics-server插件
简单介绍: 如果使用kubernetes的自动扩容功能的话,那首先得有一个插件,然后该插件将收集到的信息(cpu.memory..)与自动扩容的设置的值进行比对,自动调整pod数量.关于该插件,在ku ...
kubernetes 1.14安装部署helm插件
简单介绍: Helm其实就是一个基于Kubernetes的程序包(资源包)管理器,它将一个应用的相关资源组织成为Charts,并通过Charts管理程序包.再简单点说,可以当做RHEL/CentOS系 ...
kubernetes 1.14安装部署ingress
简单介绍: Ingress是Kubernetes API的标准资源类型之一,它其实就是一组基于DNS名称或URL路径把请求转发至指定的Service资源的规则,用来将集群外部的请求流量转发至集群内部. ...
用fabric部署维护kle日志收集系统
最近搞了一个logstash kafka elasticsearch kibana 整合部署的日志收集系统.部署参考lagstash + elasticsearch + kibana 3 + kafk ...
容器云平台No.9~kubernetes日志收集系统EFK
EFK介绍 EFK,全称Elasticsearch Fluentd Kibana ,是kubernetes中比较常用的日志收集方案,也是官方比较推荐的方案. 通过EFK,可以把集群的所有日志收集到El ...

随机推荐

《Linux》跟老男孩学Linux核心系统命令
一.命令行简介 1.1 Linux 命令行提示符介绍 [root@root_pc ~]# #<==这是超级管理员root用户对应的命令行 [oldboy@oldboy_pc ~]$ #<= ...
【BZOJ 2351】Matrix（Hash）
题目链接二维$Hash$类似二维前缀和,每一行看成一个$h$进制数,每一个以(1,1)为左上角的矩阵看成一个由每一行的$Hash$值组成的$l$进制数. 然后自己推推柿子就行. #i ...
react之高阶组件（二）
高阶组件的使用接上文———— 一.像函数一样直接调用 import React, { Component } from 'react' import A from './A' class C ext ...
go语言实现限流器
本文:https://chai2010.cn/advanced-go-programming-book/ch5-web/ch5-06-ratelimit.html Ratelimit 服务流量限制计 ...
jenkens + github
一获取github accessToken 依次点击 settings----> Developer settings --->Personal access tokens 到这里如 ...
ansible之基础篇(一)
ansible简介 ansible是新出现的自动化运维工具,基于Python开发,集合了众多运维工具(puppet.cfengine.chef.func.fabric)的优点,实现了批量系统配置.批量 ...
数据库事务和锁（三）——INNODB_LOCKS, INNODB_LOCK_WAITS, INNODB_TRX表的简单介绍
INNODB_LOCKS, INNODB_LOCK_WAITS, INNODB_TRX是MYSQL中事务和锁相关的表.通常我们遇到事务超时或锁相关问题时,直接运行下面SQL语句即可进行简单检查: -- ...
Pandas进阶笔记（0）为什么写这个系列
使用Pandas数年之久了,从最早的0.17版本开始接触Pandas,到现在0.25版本,踩过不少坑,面对各种稀奇古怪的bug抓耳挠腮.每每想要解决bug,或者想要实现一个特定的数据操作需求,首先想到 ...
小程序框架之视图层 View~基础组件
框架为开发者提供了一系列基础组件,开发者可以通过组合这些基础组件进行快速开发.详细介绍请参考组件文档. 什么是组件: 组件是视图层的基本组成单元. 组件自带一些功能与微信风格一致的样式. 一个组件通常 ...
P1600 天天爱跑步[桶+LCA+树上差分]
题目描述小c同学认为跑步非常有趣,于是决定制作一款叫做<天天爱跑步>的游戏.<天天爱跑步>是一个养成类游戏,需要玩家每天按时上线,完成打卡任务. 这个游戏的地图可以看作一一棵 ...

kubernetes 1.14安装部署EFK日志收集系统

特别补充：

kubernetes 1.14安装部署EFK日志收集系统的更多相关文章

随机推荐

热门专题