简单介绍：

EFK 组合插件是k8s项目的一个日志解决方案，它包括三个组件：Elasticsearch, Fluentd, Kibana。相对于ELK这样的架构，k8s官方推行了EFK，可能Fluentd相对于Logstash更加轻量级吧。

Elasticsearch 是日志存储和日志搜索引擎，Fluentd 负责把k8s集群的日志发送给 Elasticsearch, Kibana 则是可视化界面查看和检索存储在 Elasticsearch 的数据。

GitHub官网：https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch

特别补充：

Kubernetes官方推荐的EFK组合，其实对于我们大多数公司来说并不合适，其中一点原因就是这个组合需要应用日志打印到标准输出。

推荐做法：还是使用传统的ELK组合，ELK可以部署在K8S集群之外，日志写入指定存储目录文件中，并采用filebeat方式采集。

测试环境：

System Hostname IP
CentOS 7.6 k8s-master 138.138.82.14
CentOS 7.6 k8s-node1 138.138.82.15
CentOS 7.6 k8s-node2 138.138.82.16

注：以下是简单的安装部署操作，实际用于生产环境的话还要根据不同的后端存储进而修改yaml中的相应存储路径。

操作步骤：

1. 获取yaml文件

从github官网下载：https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch

或者下载整个tar包：https://storage.googleapis.com/kubernetes-release/release/v1.14.1/kubernetes-server-linux-amd64.tar.gz

解压后yanl文件位置： kubernetes/cluster/addons/fluentd-elasticsearch/

复制yaml文件到另一处进行修改操作：

[root@k8s-master ~]# mkdir efk

[root@k8s-master ~]# cp kubernetes/cluster/addons/fluentd-elasticsearch/*.yaml  efk/

[root@k8s-master ~]# cd efk/

[root@k8s-master efk]# ll

总用量 36

-rw-r--r-- 1 root root   382 5月   5 13:55 es-service.yaml

-rw-r--r-- 1 root root  2906 5月   5 14:44 es-statefulset.yaml

-rw-r--r-- 1 root root 16124 5月   5 13:55 fluentd-es-configmap.yaml

-rw-r--r-- 1 root root  2702 5月   5 13:55 fluentd-es-ds.yaml

-rw-r--r-- 1 root root  1052 5月   5 13:55 kibana-deployment.yaml

-rw-r--r-- 1 root root   354 5月   5 13:55 kibana-service.yaml

2. 准备镜像（在所有主机上）

注：因为被墙的缘故，加上镜像文件比较大，启动会费时，可以提前单独下载好所需镜像。

①修改 es-statefulset.yaml （https://www.docker.elastic.co/#）

②修改 fluentd-es-ds.yaml

③无需修改 kibana-deployment.yaml

说明：关于kibana镜像以-oss结尾的版本好奇（default release that contains the commercially licensed code. The release that contains only open source will have the -oss appended to it.）：

https://www.elastic.co/guide/en/kibana/6.0/docker.html

https://discuss.elastic.co/t/what-are-the-differences-between-the-kibana-oss-and-non-oss-build/152364/3

附上镜像拉取文本：

~]# docker pull docker.elastic.co/elasticsearch/elasticsearch:6.6.

~]# docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/fluentd-elasticsearch:v2.4.0

~]# docker pull docker.elastic.co/kibana/kibana-oss:6.6.

3. 发布yaml文件到k8s中

[root@k8s-master efk]# kubectl apply -f .

4. 查看各pod状态

[root@k8s-master efk]# kubectl get po -n kube-system -o wide

NAME                                    READY   STATUS    RESTARTS   AGE     IP              NODE         NOMINATED NODE   READINESS GATES

calico-node-b78m4                       /     Running             7d1h    138.138.82.16   k8s-node2    <none>           <none>

calico-node-r5mlj                       /     Running             7d1h    138.138.82.14   k8s-master   <none>           <none>

calico-node-z5tdh                       /     Running             7d1h    138.138.82.15   k8s-node1    <none>           <none>

coredns-fb8b8dccf-6mgks                 /     Running             7d1h    192.168.0.7     k8s-master   <none>           <none>

coredns-fb8b8dccf-cbtlx                 /     Running             7d1h    192.168.0.6     k8s-master   <none>           <none>

elasticsearch-logging-0                 /     Running             10m     192.168.2.11    k8s-node2    <none>           <none>

elasticsearch-logging-1                 /     Running             10m     192.168.1.17    k8s-node1    <none>           <none>

etcd-k8s-master                         /     Running             7d1h    138.138.82.14   k8s-master   <none>           <none>

fluentd-es-v2.4.0-fm9n2                 /     Running             10m     192.168.1.16    k8s-node1    <none>           <none>

fluentd-es-v2.4.0-xjx6h                 /     Running             10m     192.168.2.10    k8s-node2    <none>           <none>

kibana-logging-f4d99b69f-hc4h9          /     Running             10m     192.168.1.15    k8s-node1    <none>           <none>

kube-apiserver-k8s-master               /     Running             7d1h    138.138.82.14   k8s-master   <none>           <none>

kube-controller-manager-k8s-master      /     Running             7d1h    138.138.82.14   k8s-master   <none>           <none>

kube-proxy-c9xd2                        /     Running             7d1h    138.138.82.14   k8s-master   <none>           <none>

kube-proxy-fp2r2                        /     Running             7d1h    138.138.82.15   k8s-node1    <none>           <none>

kube-proxy-lrsw7                        /     Running             7d1h    138.138.82.16   k8s-node2    <none>           <none>

kube-scheduler-k8s-master               /     Running             7d1h    138.138.82.14   k8s-master   <none>           <none>

kubernetes-dashboard-5f7b999d65-45vlk   /     Running             6d7h    192.168.1.14    k8s-node1    <none>           <none>

metrics-server-7579f696d8-pgcc4         /     Running             6d22h   192.168.1.12    k8s-node1    <none>           <none>

顺利的话，稍等一小会就成功running状态了。如果有问题，就查看容器日志，基本上可以定位到所有错误原因。

5. 查看kibana

留意kibana的日志输出，这里给出我的实际一段：

[root@k8s-master efk]# kubectl logs -n kube-system kibana-logging-f4d99b69f-hc4h9 -f

{"type":"log","@timestamp":"2019-05-05T08:02:58Z","tags":["warning","config","deprecation"],"pid":,"message":"You should set server.basePath along with server.rewriteBasePath. Starting in 7.0, Kibana will expect that all requests start with server.basePath rather than expecting you to rewrite the requests in your reverse proxy. Set server.rewriteBasePath to false to preserve the current behavior and silence this warning."}

{"type":"log","@timestamp":"2019-05-05T08:02:58Z","tags":["plugin","warning"],"pid":,"path":"/usr/share/kibana/src/legacy/core_plugins/ems_util","message":"Skipping non-plugin directory at /usr/share/kibana/src/legacy/core_plugins/ems_util"}

{"type":"log","@timestamp":"2019-05-05T08:02:59Z","tags":["warning","elasticsearch","config","deprecation"],"pid":,"message":"Config key \"url\" is deprecated. It has been replaced with \"hosts\""}

{"type":"log","@timestamp":"2019-05-05T08:03:00Z","tags":["status","plugin:kibana@6.6.1","info"],"pid":,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}

{"type":"log","@timestamp":"2019-05-05T08:03:00Z","tags":["status","plugin:elasticsearch@6.6.1","info"],"pid":,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}

{"type":"log","@timestamp":"2019-05-05T08:03:00Z","tags":["status","plugin:console@6.6.1","info"],"pid":,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}

{"type":"log","@timestamp":"2019-05-05T08:03:00Z","tags":["error","elasticsearch","admin"],"pid":,"message":"Request error, retrying\nHEAD http://elasticsearch-logging:9200/ => connect ECONNREFUSED 10.100.54.183:9200"}

{"type":"log","@timestamp":"2019-05-05T08:03:00Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:00Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:00Z","tags":["status","plugin:elasticsearch@6.6.1","error"],"pid":,"state":"red","message":"Status changed from yellow to red - Unable to connect to Elasticsearch.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}

{"type":"log","@timestamp":"2019-05-05T08:03:01Z","tags":["status","plugin:interpreter@6.6.1","info"],"pid":,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}

{"type":"log","@timestamp":"2019-05-05T08:03:01Z","tags":["status","plugin:metrics@6.6.1","info"],"pid":,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}

{"type":"log","@timestamp":"2019-05-05T08:03:01Z","tags":["status","plugin:timelion@6.6.1","info"],"pid":,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}

{"type":"log","@timestamp":"2019-05-05T08:03:03Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:03Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:06Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:06Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:08Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:08Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:11Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:11Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:13Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:13Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:16Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:16Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:18Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:18Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:21Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:21Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:23Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:23Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:26Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:26Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:28Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:28Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:31Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:31Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:33Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:33Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:36Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:36Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:38Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:38Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:41Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:41Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:44Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:44Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:46Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:46Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:49Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:49Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:51Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:51Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:54Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:54Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:56Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"Unable to revive connection: http://elasticsearch-logging:9200/"}

{"type":"log","@timestamp":"2019-05-05T08:03:56Z","tags":["warning","elasticsearch","admin"],"pid":,"message":"No living connections"}

{"type":"log","@timestamp":"2019-05-05T08:03:59Z","tags":["error","elasticsearch","admin"],"pid":,"message":"Request error, retrying\nHEAD http://elasticsearch-logging:9200/ => connect ECONNREFUSED 10.100.54.183:9200"}

{"type":"log","@timestamp":"2019-05-05T08:03:59Z","tags":["warning"],"pid":,"kibanaVersion":"6.6.1","nodes":[{"version":"6.6.2","http":{"publish_address":"192.168.2.11:9200"},"ip":"192.168.2.11"}],"message":"You're running Kibana 6.6.1 with some different versions of Elasticsearch. Update Kibana or Elasticsearch to the same version to prevent compatibility issues: v6.6.2 @ 192.168.2.11:9200 (192.168.2.11)"}

{"type":"log","@timestamp":"2019-05-05T08:03:59Z","tags":["status","plugin:elasticsearch@6.6.1","info"],"pid":,"state":"green","message":"Status changed from red to green - Ready","prevState":"red","prevMsg":"Unable to connect to Elasticsearch."}

{"type":"log","@timestamp":"2019-05-05T08:03:59Z","tags":["info","migrations"],"pid":,"message":"Creating index .kibana_1."}

{"type":"log","@timestamp":"2019-05-05T08:04:00Z","tags":["info","migrations"],"pid":,"message":"Pointing alias .kibana to .kibana_1."}

{"type":"log","@timestamp":"2019-05-05T08:04:00Z","tags":["info","migrations"],"pid":,"message":"Finished in 994ms."}

{"type":"log","@timestamp":"2019-05-05T08:04:00Z","tags":["listening","info"],"pid":,"message":"Server running at http://0:5601"}

以上显示了启动成功的一段过程。

查看kibana地址

[root@k8s-master ~]# kubectl cluster-info |grep kibana

Kibana is running at https://138.138.82.14:6443/api/v1/namespaces/kube-system/services/kibana-logging/proxy

在浏览器中输入以上链接即可，效果如下：

EFK插件简单部署到此结束，实际生产应用还需修改yaml文件相关参数。

结束.

kubernetes 1.14安装部署EFK日志收集系统的更多相关文章

  1. Kubernetes 系列（八）：搭建EFK日志收集系统

    Kubernetes 中比较流行的日志收集解决方案是 Elasticsearch.Fluentd 和 Kibana(EFK)技术栈,也是官方现在比较推荐的一种方案. Elasticsearch 是一个 ...

  2. 十九，基于helm搭建EFK日志收集系统

    目录 EFK日志系统 一,EFK日志系统简介: 二,EFK系统部署 1,EFK系统部署方式 2,基于Helm方式部署EFK EFK日志系统 一,EFK日志系统简介: 关于系统日志收集处理方案,其实有很 ...

  3. Docker搭建EFK日志收集系统，并自定义es索引名

    EFK架构图 一.EFK简介 EFK不是一个软件,而是一套解决方案,并且都是开源软件,之间互相配合使用,完美衔接,高效的满足了很多场合的应用,是目前主流的一种日志系统. EFK是三个开源软件的缩写,分 ...

  4. kubernetes 1.14安装部署dashboard

    简单介绍: Dashboard是一个基于web的Kubernetes用户界面.您可以使用Dashboard将容器化应用程序部署到Kubernetes集群,对容器化应用程序进行故障诊断,并管理集群资源. ...

  5. kubernetes 1.14安装部署metrics-server插件

    简单介绍: 如果使用kubernetes的自动扩容功能的话,那首先得有一个插件,然后该插件将收集到的信息(cpu.memory..)与自动扩容的设置的值进行比对,自动调整pod数量.关于该插件,在ku ...

  6. kubernetes 1.14安装部署helm插件

    简单介绍: Helm其实就是一个基于Kubernetes的程序包(资源包)管理器,它将一个应用的相关资源组织成为Charts,并通过Charts管理程序包.再简单点说,可以当做RHEL/CentOS系 ...

  7. kubernetes 1.14安装部署ingress

    简单介绍: Ingress是Kubernetes API的标准资源类型之一,它其实就是一组基于DNS名称或URL路径把请求转发至指定的Service资源的规则,用来将集群外部的请求流量转发至集群内部. ...

  8. 用fabric部署维护kle日志收集系统

    最近搞了一个logstash kafka elasticsearch kibana 整合部署的日志收集系统.部署参考lagstash + elasticsearch + kibana 3 + kafk ...

  9. 容器云平台No.9~kubernetes日志收集系统EFK

    EFK介绍 EFK,全称Elasticsearch Fluentd Kibana ,是kubernetes中比较常用的日志收集方案,也是官方比较推荐的方案. 通过EFK,可以把集群的所有日志收集到El ...

随机推荐

  1. css3可拖动的魔方3d

    css3可拖动的魔方3d 主要用到知识点: css3 3d转换 原生js鼠标拖动事件 display:grid 布局 实现的功能 3d魔方 可点击,可拖动 直接看效果 html: <div cl ...

  2. go语言实现限流器

    本文:https://chai2010.cn/advanced-go-programming-book/ch5-web/ch5-06-ratelimit.html Ratelimit 服务流量限制 计 ...

  3. C#中hashtable如何嵌套遍历

    嵌套hashtable的遍历取值怎么做 hastable中嵌套了hashtable,想用递归的方式把所有hashtable中的key和value取出来 foreach (DictionaryEntry ...

  4. 【HICP Gauss】数据库 数据库高级语法(数据类型函数)-3

    SQL高级语法:整型:   integer 2(-31) ~2(31)-1 4字节 intger unsigned 2(0)~2(32)-1 4字节 bigint 2(-63)~2(63)-1 8字节 ...

  5. python爬有道翻译

    在有道翻译页面中打开开发者工具,在Headers板块找到Request URL以及相应的data. import urllib.request import urllib.parse import j ...

  6. readme.txt 为什么我们不重视

    1.很多源码项目下面都有编译说明,readme 2.我们的电脑上每个目录都有很多文件,我们似乎可以建立一个readme.txt 知道是什么 节约一些时间

  7. Linux命令基础6-mkdir命令

    mkdir是英文单词make directory的缩写.mkdir就是用来创建路径,一般就是用来创建文件夹的. 语法 mkdir (选项)(参数) 选项 -Z:设置安全上下文,当使用SELinux时有 ...

  8. UVa 1671 语言的历史——判断两个DFA是否等价

    题意 一个DFA可以用一个5元组  $((Q, \sum , \delta , q_0, F))$ 表示,其中 $Q$ 为状态集,$\sum$ 为字母表,$\delta$ 为转移函数,$q_0$ 为起 ...

  9. 无向图的割点和桥 tarjan 模板

    #include <bits/stdc++.h> using namespace std; const int MAXN = 20005; const int MAXM = 100005; ...

  10. ML,DL核心数学及算法知识点总结

    ML,DL核心数学及算法知识点总结:https://mp.weixin.qq.com/s/bskyMQ2i1VMNiYKIvw_d7g