12-部署EFK插件
配置和安装 EFK
官方文件目录：cluster/addons/fluentd-elasticsearch
$ ls *.yaml
es-controller.yaml es-service.yaml fluentd-es-ds.yaml kibana-controller.yaml kibana-service.yaml efk-rbac.yaml
同样EFK服务也需要一个efk-rbac.yaml文件，配置serviceaccount为efk。
已经修改好的 yaml 文件见：EFK
配置 es-controller.yaml
# cat es-controller.yaml
apiVersion: v1
kind: ReplicationController
metadata:
name: elasticsearch-logging-v1
namespace: kube-system
labels:
k8s-app: elasticsearch-logging
version: v1
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
spec:
replicas: 2
selector:
k8s-app: elasticsearch-logging
version: v1
template:
metadata:
labels:
k8s-app: elasticsearch-logging
version: v1
kubernetes.io/cluster-service: "true"
spec:
serviceAccountName: efk
containers:
- image: index.tenxcloud.com/docker_library/elasticsearch:2.2.0
name: elasticsearch-logging
resources:
# need more cpu upon initialization, therefore burstable class
limits:
cpu: 1000m
requests:
cpu: 100m
ports:
- containerPort: 9200
name: db
protocol: TCP
- containerPort: 9300
name: transport
protocol: TCP
volumeMounts:
- name: es-persistent-storage
mountPath: /data
env:
- name: "NAMESPACE"
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumes:
- name: es-persistent-storage
emptyDir: {}
配置 es-service.yaml
无需修改
配置 fluentd-es-ds.yaml
# cat fluentd-es-ds.yaml
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: fluentd-es-v1.22
namespace: kube-system
labels:
k8s-app: fluentd-es
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
version: v1.22
spec:
template:
metadata:
labels:
k8s-app: fluentd-es
kubernetes.io/cluster-service: "true"
version: v1.22
# This annotation ensures that fluentd does not get evicted if the node
# supports critical pod annotation based priority scheme.
# Note that this does not guarantee admission on the nodes (#40573).
annotations:
scheduler.alpha.kubernetes.io/critical-pod: ''
spec:
serviceAccountName: fluentd
containers:
- name: fluentd-es
image: index.tenxcloud.com/zhangshun/fluentd-elasticsearch:v1
command:
- '/bin/sh'
- '-c'
- '/usr/sbin/td-agent 2>&1 >> /var/log/fluentd.log'
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 200Mi
volumeMounts:
- name: varlog
mountPath: /var/log
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
nodeSelector:
beta.kubernetes.io/fluentd-ds-ready: "true"
tolerations:
- key : "node.alpha.kubernetes.io/ismaster"
effect: "NoSchedule"
terminationGracePeriodSeconds: 30
volumes:
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
配置 kibana-controller.yaml
# cat kibana-controller.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: kibana-logging
namespace: kube-system
labels:
k8s-app: kibana-logging
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
spec:
replicas: 1
selector:
matchLabels:
k8s-app: kibana-logging
template:
metadata:
labels:
k8s-app: kibana-logging
spec:
serviceAccountName: efk
containers:
- name: kibana-logging
image: index.tenxcloud.com/docker_library/kibana:4.5.1
resources:
# keep request = limit to keep this container in guaranteed class
limits:
cpu: 100m
requests:
cpu: 100m
env:
- name: "ELASTICSEARCH_URL"
value: "http://elasticsearch-logging:9200"
- name: "KIBANA_BASE_URL"
value: "/api/v1/proxy/namespaces/kube-system/services/kibana-logging"
ports:
- containerPort: 5601
name: ui
protocol: TCP
给 Node 设置标签
定义 DaemonSet fluentd-es-v1.22 时设置了 nodeSelector beta.kubernetes.io/fluentd-ds-ready=true ，所以需要在期望运行 fluentd 的 Node 上设置该标签；
# kubectl get nodes
NAME STATUS AGE VERSION
192.168.1.122 Ready 22h v1.6.2
192.168.1.123 Ready 22h v1.6.2
# kubectl label nodes 192.168.1.122 beta.kubernetes.io/fluentd-ds-ready=true
node "172.20.0.112" labeled
# kubectl label nodes 192.168.1.123 beta.kubernetes.io/fluentd-ds-ready=true
node "172.20.0.123" labeled
给其他两台node打上同样的标签。
执行定义文件
$ kubectl create -f .
serviceaccount "efk" created
clusterrolebinding "efk" created
replicationcontroller "elasticsearch-logging-v1" created
service "elasticsearch-logging" created
daemonset "fluentd-es-v1.22" created
deployment "kibana-logging" created
service "kibana-logging" created
检查执行结果
$ kubectl get deployment -n kube-system|grep kibana
kibana-logging 1 1 1 1 2m
$ kubectl get pods -n kube-system|grep -E 'elasticsearch|fluentd|kibana'
elasticsearch-logging-v1-mlstp 1/1 Running 0 1m
elasticsearch-logging-v1-nfbbf 1/1 Running 0 1m
fluentd-es-v1.22-31sm0 1/1 Running 0 1m
fluentd-es-v1.22-bpgqs 1/1 Running 0 1m
fluentd-es-v1.22-qmn7h 1/1 Running 0 1m
kibana-logging-1432287342-0gdng 1/1 Running 0 1m
$ kubectl get service -n kube-system|grep -E 'elasticsearch|kibana'
elasticsearch-logging 10.254.77.62 <none> 9200/TCP 2m
kibana-logging 10.254.8.113 <none> 5601/TCP 2m
kibana Pod 第一次启动时会用较长时间(10-20分钟)来优化和 Cache 状态页面，可以 tailf 该 Pod 的日志观察进度：
$ kubectl logs kibana-logging-1432287342-0gdng -n kube-system -f
ELASTICSEARCH_URL=http://elasticsearch-logging:9200
server.basePath: /api/v1/proxy/namespaces/kube-system/services/kibana-logging
{"type":"log","@timestamp":"2017-07-26T13:08:06Z","tags":["info","optimize"],"pid":7,"message":"Optimizing and caching bundles for kibana and statusPage. This may take a few minutes"}
{"type":"log","@timestamp":"2017-07-26T13:18:17Z","tags":["info","optimize"],"pid":7,"message":"Optimization of bundles for kibana and statusPage complete in 610.40 seconds"}
{"type":"log","@timestamp":"2017-07-26T13:18:17Z","tags":["status","plugin:kibana@1.0.0","info"],"pid":7,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2017-07-26T13:18:18Z","tags":["status","plugin:elasticsearch@1.0.0","info"],"pid":7,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2017-07-26T13:18:19Z","tags":["status","plugin:kbn_vislib_vis_types@1.0.0","info"],"pid":7,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2017-07-26T13:18:19Z","tags":["status","plugin:markdown_vis@1.0.0","info"],"pid":7,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2017-07-26T13:18:19Z","tags":["status","plugin:metric_vis@1.0.0","info"],"pid":7,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2017-07-26T13:18:19Z","tags":["status","plugin:spyModes@1.0.0","info"],"pid":7,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2017-07-26T13:18:19Z","tags":["status","plugin:statusPage@1.0.0","info"],"pid":7,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2017-07-26T13:18:19Z","tags":["status","plugin:table_vis@1.0.0","info"],"pid":7,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2017-07-26T13:18:19Z","tags":["listening","info"],"pid":7,"message":"Server running at http://0.0.0.0:5601"}
{"type":"log","@timestamp":"2017-07-26T13:18:24Z","tags":["status","plugin:elasticsearch@1.0.0","info"],"pid":7,"state":"yellow","message":"Status changed from yellow to yellow - No existing Kibana index found","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2017-07-26T13:18:29Z","tags":["status","plugin:elasticsearch@1.0.0","info"],"pid":7,"state":"green","message":"Status changed from yellow to green - Kibana index ready","prevState":"yellow","prevMsg":"No existing Kibana index found"}
访问 kibana
- 通过 kube-apiserver 访问：
获取 monitoring-grafana 服务 URL
# kubectl cluster-info
Kubernetes master is running at https://192.168.1.121:6443
Elasticsearch is running at https://192.168.1.121:6443/api/v1/proxy/namespaces/kube-system/services/elasticsearch-logging
Heapster is running at https://192.168.1.121:6443/api/v1/proxy/namespaces/kube-system/services/heapster
Kibana is running at https://192.168.1.121:6443/api/v1/proxy/namespaces/kube-system/services/kibana-logging
KubeDNS is running at https://192.168.1.121:6443/api/v1/proxy/namespaces/kube-system/services/kube-dns
kubernetes-dashboard is running at https://192.168.1.121:6443/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard
monitoring-grafana is running at https://192.168.1.121:6443/api/v1/proxy/namespaces/kube-system/services/monitoring-grafana
monitoring-influxdb is running at https://192.168.1.121:6443/api/v1/proxy/namespaces/kube-system/services/monitoring-influxd
浏览器访问 URL： `https://192.168.1.121:6443/api/v1/proxy/namespaces/kube-system/services/kibana-logging/app/kibana`
通过 kubectl proxy 访问：
创建代理
$ kubectl proxy --address='192.168.1.121' --port=8086 --accept-hosts='^*$'
Starting to serve on 192.168.1.121:8086
浏览器访问 URL：
http://192.168.1.121:8086/api/v1/proxy/namespaces/kube-system/services/kibana-logging
在 Settings -> Indices 页面创建一个 index（相当于 mysql 中的一个 database），选中 Index contains time-based events，使用默认的 logstash-* pattern，点击 Create ;
可能遇到的问题
如果你在这里发现Create按钮是灰色的无法点击，且Time-filed name中没有选项，fluentd要读取/var/log/containers/目录下的log日志，这些日志是从/var/lib/docker/containers/${CONTAINER_ID}/${CONTAINER_ID}-json.log链接过来的，查看你的docker配置，—log-dirver需要设置为json-file格式，默认的可能是journald，参考docker logging。
创建Index后，可以在 Discover 下看到 ElasticSearch logging 中汇聚的日志；
12-部署EFK插件的更多相关文章
- 09-5.部署 EFK 插件
09-5.部署 EFK 插件 EFK 对应的目录:kubernetes/cluster/addons/fluentd-elasticsearch $ cd /opt/k8s/kubernetes/cl ...
- suse 12 二进制部署 Kubernetets 1.19.7 - 第13章 - 部署metrics-server插件
文章目录 1.13.0.创建metrics-server证书和私钥 1.13.1.生成metrics-server证书和私钥 1.13.2.开启kube-apiserver聚合配置 1.13.3.分发 ...
- suse 12 二进制部署 Kubernetets 1.19.7 - 第03章 - 部署flannel插件
文章目录 1.3.部署flannel网络 1.3.0.下载flannel二进制文件 1.3.1.创建flannel证书和私钥 1.3.2.生成flannel证书和私钥 1.3.3.将pod网段写入et ...
- suse 12 二进制部署 Kubernetets 1.19.7 - 第12章 - 部署dashboard插件
文章目录 1.12.0.创建namespace 1.12.1.创建Dashboard rbac文件 1.12.2.创建dashboard文件 1.12.3.查看pod以及svc 1.12.4.获取 d ...
- kubernetes 1.14安装部署EFK日志收集系统
简单介绍: EFK 组合插件是k8s项目的一个日志解决方案,它包括三个组件:Elasticsearch, Fluentd, Kibana.相对于ELK这样的架构,k8s官方推行了EFK,可能Fluen ...
- 安装IntelliJ IDEA热部署tomcat插件JreBel
最近试着使用IntelliJ IDEA这款IDE,网上说它是最好用的java开发工具~但奈何国内ecilpse市场占有率实在稳固,所以国内这个工具也就少数人在使用 当然使用起来跟ecilpse还是有很 ...
- 在kubernetes1.17.2上结合ceph部署efk
简绍 应用程序和系统日志可以帮助我们了解集群内部的运行情况,日志对于我们调试问题和监视集群情况也是非常有用的.而且大部分的应用都会有日志记录,对于传统的应用大部分都会写入到本地的日志文件之中.对于容器 ...
- Docker Compose部署 EFK（Elasticsearch + Fluentd + Kibana）收集日志
简述 本文用于记录如何使用Docker Compose部署 EFK(Elasticsearch + Fluentd + Kibana) 收集Docker容器日志,使用EFK,可以无侵入代码,获得灵活, ...
- zend stuido 12.5的插件安装和xdebug调试器的配置和和配置注意
参考: zend stuido 12.5的插件安装 zend 12.5 安装插件是按类别进行分类了的, 而且是在欢迎 界面就可以直接安装, 安装后,要重启zend才能生效 版式设计的一个基本点就是: ...
- Elasticsearch5.5 部署Head插件
Elasticsearch5.5 部署Head插件 1.git下载软件包 yum -y install git git clone git://github.com/mobz/elasticsearc ...
随机推荐
- hello2
String username = request.getParameter("username");//获取参数值 if (username != null && ...
- mongodb文件损坏的恢复--无可恢复数据
1.mongodb 启动异常error code 100,检查日志,数据文件损坏 2 检查collection-15-6548623434943640018.wt 可恢复数据,为空,不存在恢复的数据 ...
- 100道c++面试题(上)
1. new, delete, malloc, free关系 new/delete是c++的运算符,delete会调用对象的析构函数: malloc/free是c/c++的标准库函数,free只释放内 ...
- python数据类型分类
python数据分为可变类型和不可变类型,其中:可变类型:列表,字典不可变类型:布尔值,数字,字符串,元组 specidal:集合作为set是可变的,而作为frozenset是不可变集合 可变类型和不 ...
- Python local error
原来在python的函数中和全局同名的变量,如果你有修改变量的值就会变成局部变量,在修改之前对该变量的引用自然就会出现没定义这样的错误了,如果确定要引用全局变量,并且要对它修改,必须加上global关 ...
- python shell的交互模式和文本编辑模式
之前学python的时候,是拿<笨办法学python>练习的. 书里面基本都是以.py文件去写代码,也就是文本编辑模式. 而交互模式(也就是powershell),唯有在input用户输入 ...
- thinkphp 查表返回的数组，js解析有UNICode编码，解决办法
public function getDeviceMsg(){ $allDevicesMsg = M("newdevicesstatus")->getField(" ...
- Head First Servlets & JSP 学习笔记 第十一章 —— Web应用部署
jar:java archive(java归档) war:web archive(web归档) war文件只是Web应用结构的一个快照,采用了一种更可移植的压缩形式(它实际上就是一个jar文件).建立 ...
- 20175316盛茂淞 2018-2019-2 《Java程序设计》第2周课上测试总结
20175316 2018-2019-2 <Java程序设计>第2周课上测试总结 上周考试题目总结 题目1 题目要求: 在Ubuntu中用自己的有位学号建一个文件,教材p29 Exampl ...
- jquery如何在元素后面添加一个元素
jQuery添加插入元素技巧: jquery添加分为在指定元素的里面添加和外面添加两种: 里面添加使用(append 和prepend) 里面添加又分为在里面的前面添加和后面添加 里面的前面添加使用 ...