K8S之prometheus-operator监控
prometheus-operator
1. Prometheus Operator介绍
2016年年末,CoreOs
引入了Operator 模式,并发布了Prometheus Operator 作为Operator模式
的工作示例。Prometheus Operator
自动创建和管理Prometheus
监控实例。
Prometheus Operator
的任务是使得在Kubernetes
运行Prometheus
仅可能容易,同时保留可配置性以及使Kubernetes
配置原生。
Prometheus Operator
使我们的生活更容易——部署和维护。
2. 它如何工作
为了理解这个问题,我们首先需要了解Prometheus Operator
得工作原理。
Prometheus Operator
架构图.
我们成功部署 Prometheus Operator
后可以看到一个新的CRDs(Custom Resource Defination):
- Prometheus,定义一个期望的
Prometheus deployment
。 - ServiceMonitor,声明式指定应该如何监控服务组;
Operator
根据定义自动创建Prometheusscrape
配置。 - Alertmanager,定义期望的
Alertmanager deployment
。
当服务新版本更新时,将会常见一个新Pod
。Prometheus
监控k8s API
,因此当它检测到这种变化时,它将为这个新服务(pod)创建一组新的配置。
3. ServiceMonitor
Prometheus Operator
使用一个CRD
,叫做 ServiceMonitor 将配置抽象到目标。
下面是个ServiceMonitor
的示例:
apiVersion: monitoring.coreos.com/v1alpha1
kind: ServiceMonitor
metadata:
name: frontend
labels:
tier: frontend
spec:
selector:
matchLabels:
tier: frontend
endpoints:
- port: web # 指定exporter端口,这里指定的是endpoint的名称
interval: 10s # 刷新间隔时间
这仅仅是定义一组服务应该如何被监控。现在我们需要定义一个包含了该ServiceMonitor
的Prometheus
实例到其配置:
apiVersion: monitoring.coreos.com/v1alpha1
kind: Prometheus
metadata:
name: prometheus-frontend
labels:
prometheus: frontend
spec:
version: v1.3.0
#定义应包括标签为“tier=frontend”的所有ServiceMonitor 到服务器的配置中
serviceMonitors:
- selector:
matchLabels:
tier: frontend
现在Prometheus
将会监控每个带有tier: frontend
label的服务。
4. helm安装
先决条件:
- 部署了
Helm
准备好动手操作:
helm repo add coreos https://s3-eu-west-1.amazonaws.com/coreos-charts/stable/
helm install coreos/prometheus-operator --name prometheus-operator --namespace monitoring
到目前为止,我们已经在我们的集群中安装了Prometheus Operator
的TPR
。
现在我们来部署Prometheus
,Alertmanager
和Grafana
。
TIP: 当我使用一个庞大的
Helm Charts
时,我更倾向于创建一个独立的value.yaml
文件将包含我所有自定义的变更。这么做使我和同事为后期的变化和修改更容易。
helm install coreos/kube-prometheus --name kube-prometheus \
-f my_changes/prometheus.yaml \
-f my_changes/grafana.yaml \
-f my_changes/alertmanager.yaml
检查一切是否运行正常
kubectl -n monitoring get po
NAME READY STATUS RESTARTS AGE
alertmanager-kube-prometheus-0 2/2 Running 0 1h
kube-prometheus-exporter-kube-state-68dbb4f7c9-tr6rp 2/2 Running 0 1h
kube-prometheus-exporter-node-bqcj4 1/1 Running 0 1h
kube-prometheus-exporter-node-jmcq2 1/1 Running 0 1h
kube-prometheus-exporter-node-qnzsn 1/1 Running 0 1h
kube-prometheus-exporter-node-v4wn8 1/1 Running 0 1h
kube-prometheus-exporter-node-x5226 1/1 Running 0 1h
kube-prometheus-exporter-node-z996c 1/1 Running 0 1h
kube-prometheus-grafana-54c96ffc77-tjl6g 2/2 Running 0 1h
prometheus-kube-prometheus-0 2/2 Running 0 1h
prometheus-operator-1591343780-5vb5q 1/1 Running 0 1h
访问下Prometheus UI
看一下Targets
页面:
kubectl -n monitoring port-forward prometheus-kube-prometheus-0 9090
Forwarding from 127.0.0.1:9090 -> 9090
浏览器展示如下:
5. yaml文件安装
此安装方法本人亲测有效,用到的yaml文件都打包好了。解压之后直接kubectl apply即可用。会自动监控当前集群的所有node节点和pod。只需更改yaml文件中需要用到的镜像。我这里都推到了公司公网harbor仓库。部分镜像已经打成tar包。直接docker load -i即可用。
kube-state.tar.gz
webhook-dingtalk.tar.gz
prometheus-adapter.tar.gz
5.1 安装
#软件包集成了node Exporter alertmanager grafana prometheus ingress 所有服务的配置,只需解压到K8S master中。
[root@lecode-k8s-master monitor]# ll
total 1820
-rw-r--r-- 1 root root 875 Mar 11 2022 alertmanager-alertmanager.yaml
-rw-r--r-- 1 root root 515 Mar 11 2022 alertmanager-podDisruptionBudget.yaml
-rw-r--r-- 1 root root 4337 Mar 11 2022 alertmanager-prometheusRule.yaml
-rw-r--r-- 1 root root 1483 Mar 14 2022 alertmanager-secret.yaml
-rw-r--r-- 1 root root 301 Mar 11 2022 alertmanager-serviceAccount.yaml
-rw-r--r-- 1 root root 540 Mar 11 2022 alertmanager-serviceMonitor.yaml
-rw-r--r-- 1 root root 614 Mar 11 2022 alertmanager-service.yaml
drwxr-x--- 2 root root 4096 Oct 25 13:49 backsvc #这里是grafana的service配置。nodeport模式。用于外部访问。选择使用
-rw-r--r-- 1 root root 278 Mar 11 2022 blackbox-exporter-clusterRoleBinding.yaml
-rw-r--r-- 1 root root 287 Mar 11 2022 blackbox-exporter-clusterRole.yaml
-rw-r--r-- 1 root root 1392 Mar 11 2022 blackbox-exporter-configuration.yaml
-rw-r--r-- 1 root root 3081 Mar 11 2022 blackbox-exporter-deployment.yaml
-rw-r--r-- 1 root root 96 Mar 11 2022 blackbox-exporter-serviceAccount.yaml
-rw-r--r-- 1 root root 680 Mar 11 2022 blackbox-exporter-serviceMonitor.yaml
-rw-r--r-- 1 root root 540 Mar 11 2022 blackbox-exporter-service.yaml
-rw-r--r-- 1 root root 2521 Oct 25 13:36 dingtalk-dep.yaml
-rw-r--r-- 1 root root 721 Mar 11 2022 grafana-dashboardDatasources.yaml
-rw-r--r-- 1 root root 1448347 Mar 11 2022 grafana-dashboardDefinitions.yaml
-rw-r--r-- 1 root root 625 Mar 11 2022 grafana-dashboardSources.yaml
-rw-r--r-- 1 root root 8098 Mar 11 2022 grafana-deployment.yaml
-rw-r--r-- 1 root root 86 Mar 11 2022 grafana-serviceAccount.yaml
-rw-r--r-- 1 root root 398 Mar 11 2022 grafana-serviceMonitor.yaml
-rw-r--r-- 1 root root 468 Mar 30 2022 grafana-service.yaml
drwxr-xr-x 2 root root 4096 Oct 25 13:32 ingress #这里ingress资源也是可以直接用,可以把Prometheus和grafana服务暴露在外部。
-rw-r--r-- 1 root root 2639 Mar 14 2022 kube-prometheus-prometheusRule.yaml
-rw-r--r-- 1 root root 3380 Mar 14 2022 kube-prometheus-prometheusRule.yamlbak
-rw-r--r-- 1 root root 63531 Mar 11 2022 kubernetes-prometheusRule.yaml
-rw-r--r-- 1 root root 6912 Mar 11 2022 kubernetes-serviceMonitorApiserver.yaml
-rw-r--r-- 1 root root 425 Mar 11 2022 kubernetes-serviceMonitorCoreDNS.yaml
-rw-r--r-- 1 root root 6431 Mar 11 2022 kubernetes-serviceMonitorKubeControllerManager.yaml
-rw-r--r-- 1 root root 7629 Mar 11 2022 kubernetes-serviceMonitorKubelet.yaml
-rw-r--r-- 1 root root 530 Mar 11 2022 kubernetes-serviceMonitorKubeScheduler.yaml
-rw-r--r-- 1 root root 464 Mar 11 2022 kube-state-metrics-clusterRoleBinding.yaml
-rw-r--r-- 1 root root 1712 Mar 11 2022 kube-state-metrics-clusterRole.yaml
-rw-r--r-- 1 root root 2934 Oct 25 13:40 kube-state-metrics-deployment.yaml
-rw-r--r-- 1 root root 3082 Mar 11 2022 kube-state-metrics-prometheusRule.yaml
-rw-r--r-- 1 root root 280 Mar 11 2022 kube-state-metrics-serviceAccount.yaml
-rw-r--r-- 1 root root 1011 Mar 11 2022 kube-state-metrics-serviceMonitor.yaml
-rw-r--r-- 1 root root 580 Mar 11 2022 kube-state-metrics-service.yaml
-rw-r--r-- 1 root root 444 Mar 11 2022 node-exporter-clusterRoleBinding.yaml
-rw-r--r-- 1 root root 461 Mar 11 2022 node-exporter-clusterRole.yaml
-rw-r--r-- 1 root root 3047 Mar 11 2022 node-exporter-daemonset.yaml
-rw-r--r-- 1 root root 14356 Apr 11 2022 node-exporter-prometheusRule.yaml
-rw-r--r-- 1 root root 270 Mar 11 2022 node-exporter-serviceAccount.yaml
-rw-r--r-- 1 root root 850 Mar 11 2022 node-exporter-serviceMonitor.yaml
-rw-r--r-- 1 root root 492 Mar 11 2022 node-exporter-service.yaml
-rw-r--r-- 1 root root 482 Mar 11 2022 prometheus-adapter-apiService.yaml
-rw-r--r-- 1 root root 576 Mar 11 2022 prometheus-adapter-clusterRoleAggregatedMetricsReader.yaml
-rw-r--r-- 1 root root 494 Mar 11 2022 prometheus-adapter-clusterRoleBindingDelegator.yaml
-rw-r--r-- 1 root root 471 Mar 11 2022 prometheus-adapter-clusterRoleBinding.yaml
-rw-r--r-- 1 root root 378 Mar 11 2022 prometheus-adapter-clusterRoleServerResources.yaml
-rw-r--r-- 1 root root 409 Mar 11 2022 prometheus-adapter-clusterRole.yaml
-rw-r--r-- 1 root root 2204 Mar 11 2022 prometheus-adapter-configMap.yaml
-rw-r--r-- 1 root root 2530 Oct 25 13:39 prometheus-adapter-deployment.yaml
-rw-r--r-- 1 root root 506 Mar 11 2022 prometheus-adapter-podDisruptionBudget.yaml
-rw-r--r-- 1 root root 515 Mar 11 2022 prometheus-adapter-roleBindingAuthReader.yaml
-rw-r--r-- 1 root root 287 Mar 11 2022 prometheus-adapter-serviceAccount.yaml
-rw-r--r-- 1 root root 677 Mar 11 2022 prometheus-adapter-serviceMonitor.yaml
-rw-r--r-- 1 root root 501 Mar 11 2022 prometheus-adapter-service.yaml
-rw-r--r-- 1 root root 447 Mar 11 2022 prometheus-clusterRoleBinding.yaml
-rw-r--r-- 1 root root 394 Mar 11 2022 prometheus-clusterRole.yaml
-rw-r--r-- 1 root root 5000 Mar 11 2022 prometheus-operator-prometheusRule.yaml
-rw-r--r-- 1 root root 715 Mar 11 2022 prometheus-operator-serviceMonitor.yaml
-rw-r--r-- 1 root root 499 Mar 11 2022 prometheus-podDisruptionBudget.yaml
-rw-r--r-- 1 root root 14021 Mar 11 2022 prometheus-prometheusRule.yaml
-rw-r--r-- 1 root root 1184 Mar 11 2022 prometheus-prometheus.yaml
-rw-r--r-- 1 root root 471 Mar 11 2022 prometheus-roleBindingConfig.yaml
-rw-r--r-- 1 root root 1547 Mar 11 2022 prometheus-roleBindingSpecificNamespaces.yaml
-rw-r--r-- 1 root root 366 Mar 11 2022 prometheus-roleConfig.yaml
-rw-r--r-- 1 root root 2047 Mar 11 2022 prometheus-roleSpecificNamespaces.yaml
-rw-r--r-- 1 root root 271 Mar 11 2022 prometheus-serviceAccount.yaml
-rw-r--r-- 1 root root 531 Mar 11 2022 prometheus-serviceMonitor.yaml
-rw-r--r-- 1 root root 558 Mar 11 2022 prometheus-service.yaml
drw-r--r-- 2 root root 4096 Oct 24 12:31 setup
#先apply setup目录中的yaml文件。然后apply一级目录下的yaml文件。backsvc中的grafana的service资源清单。根据情况调整为nodeport或ClusterIP。K8S集群会自动在每台K8S节点部署node-exporter并收集数据。登录grafana后初始账号密码为admin admin。添加dashboard即可监控K8S集群
[root@lecode-k8s-master monitor]# cd setup/
[root@lecode-k8s-master setup]# kubectl apply -f .
[root@lecode-k8s-master setup]# cd ..
[root@lecode-k8s-master monitor]# kubectl apply -f .
[root@lecode-k8s-master monitor]# kubectl get po -n monitoring
NAME READY STATUS RESTARTS AGE
alertmanager-main-0 2/2 Running 0 74m
alertmanager-main-1 2/2 Running 0 74m
alertmanager-main-2 2/2 Running 0 74m
blackbox-exporter-6798fb5bb4-d9m7m 3/3 Running 0 74m
grafana-64668d8465-x7x9z 1/1 Running 0 74m
kube-state-metrics-569d89897b-hlqxj 3/3 Running 0 57m
node-exporter-6vqxg 2/2 Running 0 74m
node-exporter-7dxh6 2/2 Running 0 74m
node-exporter-9j5xk 2/2 Running 0 74m
node-exporter-ftrmn 2/2 Running 0 74m
node-exporter-qszkn 2/2 Running 0 74m
node-exporter-wjkgj 2/2 Running 0 74m
prometheus-adapter-5dd78c75c6-h2jf7 1/1 Running 0 58m
prometheus-adapter-5dd78c75c6-qpwzv 1/1 Running 0 58m
prometheus-k8s-0 2/2 Running 0 74m
prometheus-k8s-1 2/2 Running 0 74m
prometheus-operator-75d9b475d9-mmzgs 2/2 Running 0 80m
webhook-dingtalk-6ffc94b49-z9z6l 1/1 Running 0 61m
[root@lecode-k8s-master backsvc]# kubectl get svc -n monitoring
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
alertmanager-main NodePort 10.98.35.93 <none> 9093:30093/TCP 72m
alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 72m
blackbox-exporter ClusterIP 10.109.10.110 <none> 9115/TCP,19115/TCP 72m
grafana NodePort 10.110.48.214 <none> 3000:30300/TCP 72m
kube-state-metrics ClusterIP None <none> 8443/TCP,9443/TCP 72m
node-exporter ClusterIP None <none> 9100/TCP 72m
prometheus-adapter ClusterIP 10.97.23.176 <none> 443/TCP 72m
prometheus-k8s ClusterIP 10.100.92.254 <none> 9090/TCP 72m
prometheus-operated ClusterIP None <none> 9090/TCP 72m
prometheus-operator ClusterIP None <none> 8443/TCP 78m
webhook-dingtalk ClusterIP 10.100.131.63 <none> 80/TCP 72m
5.2 访问服务
暴露服务三种方法:用service资源的nodeport模式,或者用k8s的ingress暴露服务或者本地nginx代理。本地的nginx代理模式
这里我grafana用的是nodeport模式。Prometheus用的是nginx代理。附上nginx配置文件
[root@lecode-k8s-master setup]# cat /usr/local/nginx/conf/4-layer-conf.d/lecode-prometheus-operator.conf
#代理prometheus内置Dashboard UI
upstream prometheus-dashboard {
server 10.100.92.254:9090; #这里ip为prometheus-k8s svc资源的ip
} server {
listen 9090;
proxy_pass prometheus-dashboard;
} #代理grafana
upstream grafana {
server 10.1.82.89:3000; #这里ip为grafana svc资源的ip
} server {
listen 3000;
proxy_pass grafana;
}
访问Prometheus targets
5.3 接入grafana
访问grafana(默认密码是admin admin)
去grafana官网下载对应dashboard 地址:https://grafana.com/grafana/dashboards/
6. 监控集群外的服务
6.1 exporter安装
在对应服务的本地安装对应的exporter用于收集数据(这里以mysql为例)
#下载对应服务的exporter
#插件下载地址:https://www.modb.pro/db/216588
#插件下载地址:https://prometheus.io/download/
#下载完成后解压mysqld_exporter-0.13.0.linux-amd64.tar.gz
#配置mysql-exporter
在root路径下,创建.my.cnf文件。内容如下:
[root@lecode-test-001 ~]# cat /root/.my.cnf
[client]
user=mysql_monitor
password=Mysql@123
#创建mysql 用户并授权
CREATE USER ‘mysql_monitor’@‘localhost’ IDENTIFIED BY ‘Mysql@123’ WITH MAX_USER_CONNECTIONS 3;
GRANT PROCESS, REPLICATION CLIENT, SELECT ON . TO ‘mysql_monitor’@‘localhost’;
FLUSH PRIVILEGES;
EXIT
#启动mysqld_exporter
[root@lecode-test-001 mysql-exporter]# nohup mysqld_exporter &
#找到对应的端口
[root@lecode-test-001 mysql-exporter]# tail -f nohup.out
level=info ts=2022-10-25T09:26:54.464Z caller=mysqld_exporter.go:277 msg="Starting msqyld_exporter" version="(version=0.13.0, branch=HEAD, revision=ad2847c7fa67b9debafccd5a08bacb12fc9031f1)"
level=info ts=2022-10-25T09:26:54.464Z caller=mysqld_exporter.go:278 msg="Build context" (gogo1.16.4,userroot@e2043849cb1f,date20210531-07:30:16)=(MISSING)
level=info ts=2022-10-25T09:26:54.464Z caller=mysqld_exporter.go:293 msg="Scraper enabled" scraper=global_status
level=info ts=2022-10-25T09:26:54.464Z caller=mysqld_exporter.go:293 msg="Scraper enabled" scraper=global_variables
level=info ts=2022-10-25T09:26:54.464Z caller=mysqld_exporter.go:293 msg="Scraper enabled" scraper=slave_status
level=info ts=2022-10-25T09:26:54.464Z caller=mysqld_exporter.go:293 msg="Scraper enabled" scraper=info_schema.innodb_cmp
level=info ts=2022-10-25T09:26:54.464Z caller=mysqld_exporter.go:293 msg="Scraper enabled" scraper=info_schema.innodb_cmpmem
level=info ts=2022-10-25T09:26:54.464Z caller=mysqld_exporter.go:293 msg="Scraper enabled" scraper=info_schema.query_response_time
level=info ts=2022-10-25T09:26:54.464Z caller=mysqld_exporter.go:303 msg="Listening on address" address=:9104 #这是exporter的端口
level=info ts=2022-10-25T09:26:54.464Z caller=tls_config.go:191 msg="TLS is disabled." http2=false
#检查端口
[root@lecode-test-001 mysql-exporter]# ss -lntup |grep 9104
tcp LISTEN 0 128 :::9104 :::* users:(("mysqld_exporter",pid=26115,fd=3))
6.2 K8S配置
创建endpoint资源关联对应服务主机的exporter端口。绑定service资源,通过ServiceMonitor资源添加Prometheus targets,
1)官方格式
kubectl -n monitoring get prometheus kube-prometheus -o yaml
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
labels:
app: prometheus
chart: prometheus-0.0.14
heritage: Tiller
prometheus: kube-prometheus
release: kube-prometheus
name: kube-prometheus
namespace: monitoring
spec:
...
baseImage: quay.io/prometheus/prometheus
serviceMonitorSelector:
matchLabels:
prometheus: kube-prometheus
#接下来就是按照格式创建对应的ServiceMonitor资源
通过ep资源把外部服务关联到K8S内部,绑定对应的svc资源。在由serviceMonitor绑定对应的svc资源把数据提交给Prometheus,serviceMonitor通过标签选择器关联service,而service只需要通过端口关联ep,这里的标签和端口一定要注意一致.
2) 创建资源
[root@lecode-k8s-master monitor]# cat mysql.yaml
apiVersion: v1
kind: Endpoints
metadata:
name: mysql-test
namespace: monitoring
subsets:
- addresses:
- ip: 192.168.1.17 # ip为安装应用服务器的ip
ports:
- name: mysql
port: 9104 #export的端口
protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/component: mysql
app.kubernetes.io/name: mysql
app.kubernetes.io/part-of: kube-prometheus
app.kubernetes.io/version: 0.49.0
name: mysql-test
namespace: monitoring
spec:
clusterIP: None
clusterIPs:
- None
ports:
- name: mysql
port: 9104
protocol: TCP
sessionAffinity: None
type: ClusterIP
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
app.kubernetes.io/component: mysql
app.kubernetes.io/name: mysql
app.kubernetes.io/part-of: kube-prometheus
app.kubernetes.io/version: 0.49.0
name: mysql-test
namespace: monitoring
spec:
endpoints:
- bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
interval: 30s
port: mysql
tlsConfig:
insecureSkipVerify: true
selector:
matchLabels:
app.kubernetes.io/component: mysql
app.kubernetes.io/name: mysql
app.kubernetes.io/part-of: kube-prometheus
#创建
[root@lecode-k8s-master monitor]# kubectl apply -f mysql.yaml
endpoints/mysql-test created
service/mysql-test created
servicemonitor.monitoring.coreos.com/mysql-test created
#检查
[root@lecode-k8s-master monitor]# kubectl get -f mysql.yaml
NAME ENDPOINTS AGE
endpoints/mysql-test 192.168.1.17:9104 10m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/mysql-test ClusterIP None <none> 9104/TCP 10m
NAME AGE
servicemonitor.monitoring.coreos.com/mysql-test 10m
6.3 接入grafana
7 监控集群外的服务-redis
7.1 exporter安装
#部署redis-exporter
#插件下载地址:https://www.modb.pro/db/216588
[root@lecode-test-001 ~]# tar xf redis_exporter-v1.3.2.linux-amd64.tar.gz
[root@lecode-test-001 ~]# ll
drwxr-xr-x 2 root root 4096 Nov 6 2019 redis_exporter-v1.3.2.linux-amd64
-rw-r--r-- 1 root root 3376155 Oct 27 10:26 redis_exporter-v1.3.2.linux-amd64.tar.gz
[root@lecode-test-001 ~]# mv redis_exporter-v1.3.2.linux-amd64 redis_exporter
[root@lecode-test-001 ~]# cd redis_exporter
[root@lecode-test-001 redis_exporter]# ll
total 8488
-rw-r--r-- 1 root root 1063 Nov 6 2019 LICENSE
-rw-r--r-- 1 root root 10284 Nov 6 2019 README.md
-rwxr-xr-x 1 root root 8675328 Nov 6 2019 redis_exporter
[root@lecode-test-001 redis_exporter]# nohup ./redis_exporter -redis.addr 192.168.1.17:6379 -redis.password 'Redislecodetest@shuli123' &
[1] 4564
[root@lecode-test-001 redis_exporter]# nohup: ignoring input and appending output to ânohup.outâ
[root@lecode-test-001 redis_exporter]# tail -f nohup.out
time="2022-10-27T10:26:48+08:00" level=info msg="Redis Metrics Exporter v1.3.2 build date: 2019-11-06-02:25:20 sha1: 175a69f33e8267e0a0ba47caab488db5e83a592e Go: go1.13.4 GOOS: linux GOARCH: amd64"
time="2022-10-27T10:26:48+08:00" level=info msg="Providing metrics at :9121/metrics"
#端口为9121
7.2 K8S配置
#创建redis-serviceMonitor资源
[root@lecode-k8s-master monitor]# cat redis.yaml
apiVersion: v1
kind: Endpoints
metadata:
name: redis-test
namespace: monitoring
subsets:
- addresses:
- ip: 192.168.1.17 # ip为安装应用服务器的ip
ports:
- name: redis
port: 9121 #exporter端口
protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/component: redis
app.kubernetes.io/name: redis
app.kubernetes.io/part-of: kube-prometheus
app.kubernetes.io/version: 0.49.0
name: redis-test
namespace: monitoring
spec:
clusterIP: None
clusterIPs:
- None
ports:
- name: redis
port: 9121
protocol: TCP
sessionAffinity: None
type: ClusterIP
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
app.kubernetes.io/component: redis
app.kubernetes.io/name: redis
app.kubernetes.io/part-of: kube-prometheus
app.kubernetes.io/version: 0.49.0
name: redis-test
namespace: monitoring
spec:
endpoints:
- bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
interval: 30s
port: redis
tlsConfig:
insecureSkipVerify: true
selector:
matchLabels:
app.kubernetes.io/component: redis
app.kubernetes.io/name: redis
app.kubernetes.io/part-of: kube-prometheus
#创建资源
[root@lecode-k8s-master monitor]# kubectl apply -f redis.yaml
endpoints/redis-test created
service/redis-test created
servicemonitor.monitoring.coreos.com/redis-test created
[root@lecode-k8s-master monitor]# kubectl get ep,svc,serviceMonitor -n monitoring |grep redis
endpoints/redis-test 192.168.1.17:9121 6m2s
service/redis-test ClusterIP None <none> 9121/TCP 6m2s
servicemonitor.monitoring.coreos.com/redis-test 6m2s
7.3 Prometheus检查
7.4 接入grafana
dashboard ID: 11835
8. 监控集群外的服务-kafka
8.1 安装exporter
#插件下载地址:https://www.modb.pro/db/216588
[root@lecode-test-001 ~]# tar xf kafka_exporter-1.4.2.linux-amd64.tar.gz
[root@lecode-test-001 ~]# ll
drwxrwxr-x 2 2000 2000 4096 Sep 16 2021 kafka_exporter-1.4.2.linux-amd64
-rw-r--r-- 1 root root 8499720 Oct 27 15:30 kafka_exporter-1.4.2.linux-amd64.tar.gz
[root@lecode-test-001 ~]# mv kafka_exporter-1.4.2.linux-amd64 kafka_exporter
[root@lecode-test-001 ~]# cd kafka_exporter
[root@lecode-test-001 kafka_exporter]# ll
total 17676
-rwxr-xr-x 1 2000 2000 18086208 Sep 16 2021 kafka_exporter
-rw-rw-r-- 1 2000 2000 11357 Sep 16 2021 LICENSE
[root@lecode-test-001 kafka_exporter]# nohup ./kafka_exporter --kafka.server=192.168.1.17:9092 &
[1] 20777
[root@lecode-test-001 kafka_exporter]# nohup: ignoring input and appending output to ‘nohup.out’
[root@lecode-test-001 kafka_exporter]# tail -f nohup.out
I1027 15:32:38.904075 20777 kafka_exporter.go:769] Starting kafka_exporter (version=1.4.2, branch=HEAD, revision=0d5d4ac4ba63948748cc2c53b35ed95c310cd6f2)
I1027 15:32:38.905515 20777 kafka_exporter.go:929] Listening on HTTP :9308
#exporter端口为9308
8.2 K8S配置
[root@lecode-k8s-master monitor]# cat kafka.yaml
apiVersion: v1
kind: Endpoints
metadata:
name: kafka-test
namespace: monitoring
subsets:
- addresses:
- ip: 192.168.1.17 # ip为安装应用服务器的ip
ports:
- name: kafka
port: 9308 #export的端口
protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/component: kafka
app.kubernetes.io/name: kafka
app.kubernetes.io/part-of: kube-prometheus
app.kubernetes.io/version: 0.49.0
name: kafka-test
namespace: monitoring
spec:
clusterIP: None
clusterIPs:
- None
ports:
- name: kafka
port: 9308
protocol: TCP
sessionAffinity: None
type: ClusterIP
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
app.kubernetes.io/component: kafka
app.kubernetes.io/name: kafka
app.kubernetes.io/part-of: kube-prometheus
app.kubernetes.io/version: 0.49.0
name: kafka-test
namespace: monitoring
spec:
endpoints:
- bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
interval: 30s
port: kafka
tlsConfig:
insecureSkipVerify: true
selector:
matchLabels:
app.kubernetes.io/component: kafka
app.kubernetes.io/name: kafka
app.kubernetes.io/part-of: kube-prometheus
#创建
[root@lecode-k8s-master monitor]# kubectl apply -f kafka.yaml
endpoints/kafka-test created
service/kafka-test created
servicemonitor.monitoring.coreos.com/kafka-test created
[root@lecode-k8s-master monitor]# kubectl get -f kafka.yaml
NAME ENDPOINTS AGE
endpoints/kafka-test 192.168.1.17:9308 8m49s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kafka-test ClusterIP None <none> 9308/TCP 8m49s
NAME AGE
servicemonitor.monitoring.coreos.com/kafka-test 8m48s
8.3 Prometheus检查
8.4 接入grafana
dashboard ID:7589
9. 监控集群外的服务-zookeeper
9.1 安装exporter
exporter下载地址:https://github.com/carlpett/zookeeper_exporter/releases/download/v1.0.2/zookeeper_exporter
[root@lecode-test-001 zookeeper_exporter]# nohup ./zookeeper_exporter -zookeeper 192.168.1.17:2181 -bind-addr :9143 &
[2] 8310
[root@lecode-test-001 zookeeper_exporter]# nohup: ignoring input and appending output to ‘nohup.out’
[root@lecode-test-001 zookeeper_exporter]# tail -f nohup.out
time="2022-10-27T15:58:27+08:00" level=info msg="zookeeper_exporter, version v1.0.2 (branch: HEAD, revision: d6e929223f6b3bf5ff25dd0340e8194cbd4d04fc)\n build user: @bd731f434d23\n build date: 2018-05-01T20:40:14+0000\n go version: go1.10.1"
time="2022-10-27T15:58:27+08:00" level=info msg="Starting zookeeper_exporter"
time="2022-10-27T15:58:27+08:00" level=info msg="Starting metric http endpoint on :9143"
#exporter端口为9143
9.2 K8S配置
[root@lecode-k8s-master monitor]# cat zookeeper.yaml
apiVersion: v1
kind: Endpoints
metadata:
name: zookeeper-test
namespace: monitoring
subsets:
- addresses:
- ip: 192.168.1.17
ports:
- name: zookeeper
port: 9143
protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/component: zookeeper
app.kubernetes.io/name: zookeeper
app.kubernetes.io/part-of: kube-prometheus
app.kubernetes.io/version: 0.49.0
name: zookeeper-test
namespace: monitoring
spec:
clusterIP: None
clusterIPs:
- None
ports:
- name: zookeeper
port: 9143
protocol: TCP
sessionAffinity: None
type: ClusterIP
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
app.kubernetes.io/component: zookeeper
app.kubernetes.io/name: zookeeper
app.kubernetes.io/part-of: kube-prometheus
app.kubernetes.io/version: 0.49.0
name: zookeeper-test
namespace: monitoring
spec:
endpoints:
- bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
interval: 30s
port: zookeeper
tlsConfig:
insecureSkipVerify: true
selector:
matchLabels:
app.kubernetes.io/component: zookeeper
app.kubernetes.io/name: zookeeper
app.kubernetes.io/part-of: kube-prometheus
#创建
[root@lecode-k8s-master monitor]# kubectl apply -f zookeeper.yaml
endpoints/zookeeper-test created
service/zookeeper-test created
servicemonitor.monitoring.coreos.com/zookeeper-test created
[root@lecode-k8s-master monitor]# kubectl get -f zookeeper.yaml
NAME ENDPOINTS AGE
endpoints/zookeeper-test 192.168.1.17:9143 9m55s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/zookeeper-test ClusterIP None <none> 9143/TCP 9m55s
NAME AGE
servicemonitor.monitoring.coreos.com/zookeeper-test 9m55s
9.3 Prometheus检查
9.4 接入grafana
dashboard ID:15026
K8S之prometheus-operator监控的更多相关文章
- Prometheus Operator 监控Kubernetes
Prometheus Operator 监控Kubernetes 1. Prometheus的基本架构 Prometheus是一个开源的完整监控解决方案,涵盖数据采集.查询.告警.展示整个监控流程 ...
- 使用Prometheus Operator 监控Kubernetes(15)
一.Prometheus概述: Prometheus是一个开源系统监测和警报工具箱. Prometheus Operator 是 CoreOS 开发的基于 Prometheus 的 Kubernete ...
- k8s Helm安装Prometheus Operator
Ubuntu 18 Kubernetes集群的安装和部署 以及Helm的安装完成了k8s的集群和helm的安装,今天我们来看看Prometheus的监控怎么搞.Prometheus Operator ...
- Kubernetes 监控方案之 Prometheus Operator(十九)
目录 一.Prometheus 介绍 1.1.Prometheus 架构 1.2.Prometheus Operator 架构 二.Helm 安装部署 2.1.Helm 客户端安装 2.2.Tille ...
- Prometheus Operator自定义监控项
Prometheus Operator默认的监控指标并不能完全满足实际的监控需求,这时候就需要我们自己根据业务添加自定义监控.添加一个自定义监控的步骤如下: 1.创建一个ServiceMonitor对 ...
- 简单4步,利用Prometheus Operator实现自定义指标监控
本文来自Rancher Labs 在过去的文章中,我们花了相当大的篇幅来聊关于监控的话题.这是因为当你正在管理Kubernetes集群时,一切都会以极快的速度发生变化.因此有一个工具来监控集群的健康状 ...
- 容器编排系统K8s之Prometheus监控系统+Grafana部署
前文我们聊到了k8s的apiservice资源结合自定义apiserver扩展原生apiserver功能的相关话题,回顾请参考:https://www.cnblogs.com/qiuhom-1874/ ...
- K8s之Prometheus监控
目录 容器监控与报警 Prometheus prometheus简介 prometheus系统架构 prometheus 安装方式 容器方式安装prometheus operator部署 克隆项目 创 ...
- Kubernetes 监控:Prometheus Operator
安装 前面的章节中我们学习了用自定义的方式来对 Kubernetes 集群进行监控,基本上也能够完成监控报警的需求了.但实际上对上 Kubernetes 来说,还有更简单方式来监控报警,那就是 Pro ...
- Kubernetes 监控:Prometheus Operator + Thanos ---实践篇
具体参考网址:https://www.cnblogs.com/sanduzxcvbnm/p/16291296.html 本章用到的yaml文件地址:https://files.cnblogs.com/ ...
随机推荐
- SSH免密登录的配置
ssh登录 登录ssh一般情况有两种方法 密码登录 秘钥登录(免密) 大部分情况我们选择都是输入密码登录,平常使用暂时没有遇到什么问题.最近我编写了一些使用scp来传输文件的脚本,每一次scp都需要输 ...
- 在 C# CLR 中学习 C++ 之了解 extern
一:背景 在 CLR 源码中有很多的 extern 和 extern "C" 这样的关键词,比如下面这些代码: extern size_t gc_global_mechanisms ...
- 【Traefik二次开发】中间件 Middleware 开发
本篇只讨论HTTP中间件 中间件定义 https://doc.traefik.io/traefik/middlewares/overview/ Attached to the routers, pie ...
- GNSS模块使用笔记
目录 目录 GNSS芯片 NMEA0183 协议 指令 GNSS TO MCU MCU TO GNSS GNSS芯片 ATGM336H-5N31(GPS+BDS双模) 原理图 NMEA0183 协议 ...
- docker-compose总结
docker-compose.yml 样例: 各个标签的含义在注释里 version: '3' # 选择的docker-compose 版本 services: # 要编排的一组服务 fim-mysq ...
- Sync包
sync同步包 Mutex互斥锁: 能够保证在同一时间段内仅有一个goroutine持有锁,有且仅有一个goroutine访问共享资源,其他申请锁的goroutine将会被阻塞直到锁被释放.然后重新争 ...
- MySQL数据备份 mysqldump 详解
MySQL数据备份流程 1 打开cmd窗口 通过命令进行数据备份与恢复: 需要在Windows的命令行窗口中进行: l 开始菜单,在运行中输入cmd回车: l 或者win+R,然后输入cmd回车,即可 ...
- Elasticsearch:inverted index,doc_values及source
以后会用到的相关知识:索引中某些字段禁止搜索,排序等操作 当我们学习Elasticsearch时,经常会遇到如下的几个概念: Reverted index doc_values source? 这个几 ...
- Elasticsearch:ICU分词器介绍
ICU Analysis插件是一组将Lucene ICU模块集成到Elasticsearch中的库. 本质上,ICU的目的是增加对Unicode和全球化的支持,以提供对亚洲语言更好的文本分割分析. 从 ...
- Elasticsearch:fielddata 介绍
默认情况下,大多数字段都已编入索引,这使它们可搜索. 但是,脚本中的排序,聚合和访问字段值需要与搜索不同的访问模式. 搜索需要回答"哪个文档包含该术语?"这个问题,而排序和汇总则需 ...