简单介绍：

EFK 组合插件是k8s项目的一个日志解决方案，它包括三个组件：Elasticsearch, Fluentd, Kibana。相对于ELK这样的架构，k8s官方推行了EFK，可能Fluentd相对于Logstash更加轻量级吧。

Elasticsearch 是日志存储和日志搜索引擎，Fluentd 负责把k8s集群的日志发送给 Elasticsearch, Kibana 则是可视化界面查看和检索存储在 Elasticsearch 的数据。

GitHub官网：https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch

特别补充：

Kubernetes官方推荐的EFK组合，其实对于我们大多数公司来说并不合适，其中一点原因就是这个组合需要应用日志打印到标准输出。

推荐做法：还是使用传统的ELK组合，ELK可以部署在K8S集群之外，日志写入指定存储目录文件中，并采用filebeat方式采集。

测试环境：

System Hostname IP
CentOS 7.6 k8s-master 138.138.82.14
CentOS 7.6 k8s-node1 138.138.82.15
CentOS 7.6 k8s-node2 138.138.82.16

注：以下是简单的安装部署操作，实际用于生产环境的话还要根据不同的后端存储进而修改yaml中的相应存储路径。

操作步骤：

1. 获取yaml文件

从github官网下载：https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch

或者下载整个tar包：https://storage.googleapis.com/kubernetes-release/release/v1.14.1/kubernetes-server-linux-amd64.tar.gz

解压后yanl文件位置： kubernetes/cluster/addons/fluentd-elasticsearch/

复制yaml文件到另一处进行修改操作：

[root@k8s-master ~]# mkdir efk

[root@k8s-master ~]# cp kubernetes/cluster/addons/fluentd-elasticsearch/*.yaml  efk/

[root@k8s-master ~]# cd efk/

[root@k8s-master efk]# ll

总用量 36

-rw-r--r-- 1 root root   382 5月   5 13:55 es-service.yaml

-rw-r--r-- 1 root root  2906 5月   5 14:44 es-statefulset.yaml

-rw-r--r-- 1 root root 16124 5月   5 13:55 fluentd-es-configmap.yaml

-rw-r--r-- 1 root root  2702 5月   5 13:55 fluentd-es-ds.yaml

-rw-r--r-- 1 root root  1052 5月   5 13:55 kibana-deployment.yaml

-rw-r--r-- 1 root root   354 5月   5 13:55 kibana-service.yaml

2. 准备镜像（在所有主机上）

注：因为被墙的缘故，加上镜像文件比较大，启动会费时，可以提前单独下载好所需镜像。

①修改 es-statefulset.yaml （https://www.docker.elastic.co/#）

②修改 fluentd-es-ds.yaml

③无需修改 kibana-deployment.yaml

说明：关于kibana镜像以-oss结尾的版本好奇（default release that contains the commercially licensed code. The release that contains only open source will have the -oss appended to it.）：

https://www.elastic.co/guide/en/kibana/6.0/docker.html

https://discuss.elastic.co/t/what-are-the-differences-between-the-kibana-oss-and-non-oss-build/152364/3

附上镜像拉取文本：

~]# docker pull docker.elastic.co/elasticsearch/elasticsearch:6.6.

~]# docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/fluentd-elasticsearch:v2.4.0

~]# docker pull docker.elastic.co/kibana/kibana-oss:6.6.

3. 发布yaml文件到k8s中

[root@k8s-master efk]# kubectl apply -f .

4. 查看各pod状态

[root@k8s-master efk]# kubectl get po -n kube-system -o wide

NAME                                    READY   STATUS    RESTARTS   AGE     IP              NODE         NOMINATED NODE   READINESS GATES

calico-node-b78m4                       /     Running             7d1h    138.138.82.16   k8s-node2    <none>           <none>

calico-node-r5mlj                       /     Running             7d1h    138.138.82.14   k8s-master   <none>           <none>

calico-node-z5tdh                       /     Running             7d1h    138.138.82.15   k8s-node1    <none>           <none>

coredns-fb8b8dccf-6mgks                 /     Running             7d1h    192.168.0.7     k8s-master   <none>           <none>

coredns-fb8b8dccf-cbtlx                 /     Running             7d1h    192.168.0.6     k8s-master   <none>           <none>

elasticsearch-logging-0                 /     Running             10m     192.168.2.11    k8s-node2    <none>           <none>

elasticsearch-logging-1                 /     Running             10m     192.168.1.17    k8s-node1    <none>           <none>

etcd-k8s-master                         /     Running             7d1h    138.138.82.14   k8s-master   <none>           <none>

fluentd-es-v2.4.0-fm9n2                 /     Running             10m     192.168.1.16    k8s-node1    <none>           <none>

fluentd-es-v2.4.0-xjx6h                 /     Running             10m     192.168.2.10    k8s-node2    <none>           <none>

kibana-logging-f4d99b69f-hc4h9          /     Running             10m     192.168.1.15    k8s-node1    <none>           <none>

kube-apiserver-k8s-master               /     Running             7d1h    138.138.82.14   k8s-master   <none>           <none>

kube-controller-manager-k8s-master      /     Running             7d1h    138.138.82.14   k8s-master   <none>           <none>

kube-proxy-c9xd2                        /     Running             7d1h    138.138.82.14   k8s-master   <none>           <none>

kube-proxy-fp2r2                        /     Running             7d1h    138.138.82.15   k8s-node1    <none>           <none>

kube-proxy-lrsw7                        /     Running             7d1h    138.138.82.16   k8s-node2    <none>           <none>

kube-scheduler-k8s-master               /     Running             7d1h    138.138.82.14   k8s-master   <none>           <none>

kubernetes-dashboard-5f7b999d65-45vlk   /     Running             6d7h    192.168.1.14    k8s-node1    <none>           <none>

metrics-server-7579f696d8-pgcc4         /     Running             6d22h   192.168.1.12    k8s-node1    <none>           <none>

顺利的话，稍等一小会就成功running状态了。如果有问题，就查看容器日志，基本上可以定位到所有错误原因。

5. 查看kibana

留意kibana的日志输出，这里给出我的实际一段：

[root@k8s-master efk]# kubectl logs -n kube-system kibana-logging-f4d99b69f-hc4h9 -f

{"type":"log","@timestamp":"2019-05-05T08:02:58Z","tags":["warning","config","deprecation"],"pid":,"message":"You should set server.basePath along with server.rewriteBasePath. Starting in 7.0, Kibana will expect that all requests start with server.basePath rather than expecting you to rewrite the requests in your reverse proxy. Set server.rewriteBasePath to false to preserve the current behavior and silence this warning."}

{"type":"log","@timestamp":"2019-05-05T08:02:58Z","tags":["plugin","warning"],"pid":,"path":"/usr/share/kibana/src/legacy/core_plugins/ems_util","message":"Skipping non-plugin directory at /usr/share/kibana/src/legacy/core_plugins/ems_util"}

{"type":"log","@timestamp":"2019-05-05T08:02:59Z","tags":["warning","elasticsearch","config","deprecation"],"pid":,"message":"Config key \"url\" is deprecated. It has been replaced with \"hosts\""}

{"type":"log","@timestamp":"2019-05-05T08:03:00Z","tags":["status","plugin:kibana@6.6.1","info"],"pid":,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}

{"type":"log","@timestamp":"2019-05-05T08:03:00Z","tags":["status","plugin:elasticsearch@6.6.1","info"],"pid":,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}

{"type":"log","@timestamp":"2019-05-05T08:03:00Z","tags":["status","plugin:console@6.6.1","info"],"pid":,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}

{"type":"log","@timestamp":"2019-05-05T08:03:00Z","tags":["error","elasticsearch","admin"],"pid":,"message":"Request error, retrying\nHEAD http://elasticsearch-logging:9200/ => connect ECONNREFUSED 10.100.54.183:9200"}

{"type":"log","@timestamp":"2019-05-05T08:03:00Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:00Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:00Z","tags":["status","plugin:elasticsearch@6.6.1","error"],"pid":,"state":"red","message":"Status changed from yellow to red - Unable to connect to Elasticsearch.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}

{"type":"log","@timestamp":"2019-05-05T08:03:01Z","tags":["status","plugin:interpreter@6.6.1","info"],"pid":,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}

{"type":"log","@timestamp":"2019-05-05T08:03:01Z","tags":["status","plugin:metrics@6.6.1","info"],"pid":,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}

{"type":"log","@timestamp":"2019-05-05T08:03:01Z","tags":["status","plugin:timelion@6.6.1","info"],"pid":,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}

{"type":"log","@timestamp":"2019-05-05T08:03:03Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:03Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:06Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:06Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:08Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:08Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:11Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:11Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:13Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:13Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:16Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:16Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:18Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:18Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:21Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:21Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:23Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:23Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:26Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:26Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:28Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:28Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:31Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:31Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:33Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:33Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:36Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:36Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:38Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:38Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:41Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:41Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:44Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:44Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:46Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:46Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:49Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:49Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:51Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:51Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:54Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:54Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:56Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:56Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:59Z","tags":["error","elasticsearch","admin"],"pid":,"message":"Request error, retrying\nHEAD http://elasticsearch-logging:9200/ => connect ECONNREFUSED 10.100.54.183:9200"}

{"type":"log","@timestamp":"2019-05-05T08:03:59Z","tags":["warning"],"pid":,"kibanaVersion":"6.6.1","nodes":[{"version":"6.6.2","http":{"publish_address":"192.168.2.11:9200"},"ip":"192.168.2.11"}],"message":"You're running Kibana 6.6.1 with some different versions of Elasticsearch. Update Kibana or Elasticsearch to the same version to prevent compatibility issues: v6.6.2 @ 192.168.2.11:9200 (192.168.2.11)"}

{"type":"log","@timestamp":"2019-05-05T08:03:59Z","tags":["status","plugin:elasticsearch@6.6.1","info"],"pid":,"state":"green","message":"Status changed from red to green - Ready","prevState":"red","prevMsg":"Unable to connect to Elasticsearch."}

{"type":"log","@timestamp":"2019-05-05T08:03:59Z","tags":["info","migrations"],"pid":,"message":"Creating index .kibana_1."}

{"type":"log","@timestamp":"2019-05-05T08:04:00Z","tags":["info","migrations"],"pid":,"message":"Pointing alias .kibana to .kibana_1."}

{"type":"log","@timestamp":"2019-05-05T08:04:00Z","tags":["info","migrations"],"pid":,"message":"Finished in 994ms."}

{"type":"log","@timestamp":"2019-05-05T08:04:00Z","tags":["listening","info"],"pid":,"message":"Server running at http://0:5601"}

以上显示了启动成功的一段过程。

查看kibana地址

[root@k8s-master ~]# kubectl cluster-info |grep kibana

Kibana is running at https://138.138.82.14:6443/api/v1/namespaces/kube-system/services/kibana-logging/proxy

在浏览器中输入以上链接即可，效果如下：

EFK插件简单部署到此结束，实际生产应用还需修改yaml文件相关参数。

结束.

kubernetes 1.14安装部署EFK日志收集系统的更多相关文章

  1. Kubernetes 系列（八）：搭建EFK日志收集系统

    Kubernetes 中比较流行的日志收集解决方案是 Elasticsearch.Fluentd 和 Kibana(EFK)技术栈,也是官方现在比较推荐的一种方案. Elasticsearch 是一个 ...

  2. 十九，基于helm搭建EFK日志收集系统

    目录 EFK日志系统 一,EFK日志系统简介: 二,EFK系统部署 1,EFK系统部署方式 2,基于Helm方式部署EFK EFK日志系统 一,EFK日志系统简介: 关于系统日志收集处理方案,其实有很 ...

  3. Docker搭建EFK日志收集系统，并自定义es索引名

    EFK架构图 一.EFK简介 EFK不是一个软件,而是一套解决方案,并且都是开源软件,之间互相配合使用,完美衔接,高效的满足了很多场合的应用,是目前主流的一种日志系统. EFK是三个开源软件的缩写,分 ...

  4. kubernetes 1.14安装部署dashboard

    简单介绍: Dashboard是一个基于web的Kubernetes用户界面.您可以使用Dashboard将容器化应用程序部署到Kubernetes集群,对容器化应用程序进行故障诊断,并管理集群资源. ...

  5. kubernetes 1.14安装部署metrics-server插件

    简单介绍: 如果使用kubernetes的自动扩容功能的话,那首先得有一个插件,然后该插件将收集到的信息(cpu.memory..)与自动扩容的设置的值进行比对,自动调整pod数量.关于该插件,在ku ...

  6. kubernetes 1.14安装部署helm插件

    简单介绍: Helm其实就是一个基于Kubernetes的程序包(资源包)管理器,它将一个应用的相关资源组织成为Charts,并通过Charts管理程序包.再简单点说,可以当做RHEL/CentOS系 ...

  7. kubernetes 1.14安装部署ingress

    简单介绍: Ingress是Kubernetes API的标准资源类型之一,它其实就是一组基于DNS名称或URL路径把请求转发至指定的Service资源的规则,用来将集群外部的请求流量转发至集群内部. ...

  8. 用fabric部署维护kle日志收集系统

    最近搞了一个logstash kafka elasticsearch kibana 整合部署的日志收集系统.部署参考lagstash + elasticsearch + kibana 3 + kafk ...

  9. 容器云平台No.9~kubernetes日志收集系统EFK

    EFK介绍 EFK,全称Elasticsearch Fluentd Kibana ,是kubernetes中比较常用的日志收集方案,也是官方比较推荐的方案. 通过EFK,可以把集群的所有日志收集到El ...

随机推荐

  1. Mongodb 学习笔记(二) ：索引

    Mongodb 是基于集合建立索引 (Index),索引的作用类似于传统关系型数据库,目的是为了提高查询速度 . 如果没有建立索引, Mongodb  在读取数据时必须扫描集合中的 所有文档记录. 这 ...

  2. a标签中target属性为“_blank”时存在安全问题

    今天看到一个比较有意思的洞,虽然不够严重,但是却普遍存在各大src中 熟悉js的朋友都应该知道当我们在调用window下的open方法创建一个新窗口的同时,我们可以获得一个创建窗口的opener句柄, ...

  3. 简单了解Django应用app及分布式路由

    前言 应用在Django的项目中是一个独立的业务模块,可以包含自己的路由,视图,模板,模型. 一 创建应用程序 创建步骤 用manage.py中的子命令startapp创建应用文件夹 在setting ...

  4. orangepi自启动打开一个终端并且运行脚本

    第一步: 在 /home/pi/.config 下创建一个文件夹,名称为 autostart,如果已经存在则不需要创建 第二步: 在autostart文件夹中创建桌面启动项文件,(文件名以 .desk ...

  5. oracle 设置归档日模式

    首先关闭ORACLE SQL> shutdown immediate 把ORACLE启动为MOUNT模式 SQL:>startup mount sql:> alter databas ...

  6. 解决ifuse不支持24位UDID问题

    在使用ifuse对iphone设备进行沙盒挂载时, 对类似XXXXXXXX-XXXXXXXXXXXXXXXX UDID设备会提示Invalid device UDID specified, lengt ...

  7. SHELL脚本编程基础知识

    SHELL脚本编程基础知识 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. Linux之父Linus有一句话很经典:"Talk is cheap, show me the ...

  8. WebService知识点

    Web Services简介: 什么是Web Services? Web Services 是应用程序组件 Web Services 使用开放协议进行通信 Web Services 是独立的(self ...

  9. （Linux基础学习）第二章：CentOS7.4安装教程

    001 002 003 004 005 006 007 008 009 010 011 012 013 014 015 016 017 018 019 020 021 022 023 024 025 ...

  10. MyBatis_tp50_动态sql_sql标签_抽取可重用的sql片段_使用include标签进行引用

    笔记要点出错分析与总结 include内部使用自定的属性,之能使用$ {}来取值 ,#{}不能用 工程组织数据库组织0.重新修改Bean类1.定义接口 public interface Employe ...