写在前面

之前部署web网站的时候，架构图中有一环节是监控部分，并且搭建一套有效的监控平台对于运维来说非常之重要，只有这样才能更有效率的保证我们的服务器和服务的稳定运行，常见的开源监控软件有好几种，如zabbix、Nagios、open-flcon还有prometheus,每一种有着各自的优劣势，感谢的童鞋可以自行百度，但是与k8s集群监控，相对于而已更加友好的是Prometheus，今天我们就看看如何部署一套Prometheus全方位监控K8S

主要内容

  • 1.Prometheus架构

  • 2.K8S监控指标及实现思路

  • 3.在K8S平台部署Prometheus

  • 4.基于K8S服务发现的配置解析

  • 5.在K8S平台部署Grafana

  • 6.监控K8S集群中Pod、Node、资源对象

  • 7.使用Grafana可视化展示Prometheus监控数据

  • 8.告警规则与告警通知

1 Prometheus架构

Prometheus 是什么

Prometheus（普罗米修斯）是一个最初在SoundCloud上构建的监控系统。自2012年成为社区开源项目，拥有非常活跃的开发人员和用户社区。为强调开源及独立维护，Prometheus于2016年加入云原生云计算基金会（CNCF），成为继Kubernetes之后的第二个托管项目。

官网地址:

https://prometheus.io

https://github.com/prometheus

Prometheus 组成及架构

  • Prometheus Server：收集指标和存储时间序列数据，并提供查询接口
  • ClientLibrary： 客户端库
  • Push Gateway：短期存储指标数据。主要用于临时性的任务
  • Exporters：采集已有的第三方服务监控指标并暴露metrics
  • Alertmanager：告警
  • Web UI：简单的Web控制台

数据模型

Prometheus将所有数据存储为时间序列；具有相同度量名称以及标签属于同一个指标。

每个时间序列都由度量标准名称和一组键值对（也成为标签）唯一标识。

时间序列格式：

{=, ...}

示例：api_http_requests_total{method="POST", handler="/messages"}

作业和实例

实例：可以抓取的目标称为实例（Instances）

作业：具有相同目标的实例集合称为作业（Job）

scrape_configs:

-job_name: 'prometheus'

static_configs:

-targets: ['localhost:9090']

-job_name: 'node'

static_configs:

-targets: ['192.168.1.10:9090']

2 K8S监控指标及实现思路

k8S监控指标

Kubernetes本身监控

  • Node资源利用率
  • Node数量
  • Pods数量（Node）
  • 资源对象状态

Pod监控

  • Pod数量（项目）
  • 容器资源利用率
  • 应用程序

Prometheus监控K8S架构

监控指标 具体实现 举例
Pod性能 cAdvisor 容器CPU
Node性能 node-exporter 节点CPU 内存利用率
K8S资源对象 kube-state-metrics Pod/Deployment/Service
服务发现：
https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config

3 在K8S平台部署Prometheus

3.1集群环境

ip地址 角色 备注
192.168.73.136 nfs
192.168.73.138 k8s-master
192.168.73.139 k8s-node01
192.168.73.140 k8s-node02
192.168.73.135 k8s-node03

3.2 项目地址：

  1. [root@k8s-master src]# git clone https://github.com/zhangdongdong7/k8s-prometheus.git
  2. Cloning into 'k8s-prometheus'...
  3. remote: Enumerating objects: 3, done.
  4. remote: Counting objects: 100% (3/3), done.
  5. remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
  6. Unpacking objects: 100% (3/3), done.
  7. [root@k8s-master src]# cd k8s-prometheus/
  8. [root@k8s-master k8s-prometheus]# ls
  9. alertmanager-configmap.yaml         kube-state-metrics-rbac.yaml             prometheus-rbac.yaml
  10. alertmanager-deployment.yaml        kube-state-metrics-service.yaml          prometheus-rules.yaml
  11. alertmanager-pvc.yaml               node_exporter-0.17.0.linux-amd64.tar.gz  prometheus-service.yaml
  12. alertmanager-service.yaml           node_exporter.sh                         prometheus-statefulset-static-pv.yaml
  13. grafana.yaml                        OWNERS                                   prometheus-statefulset.yaml
  14. kube-state-metrics-deployment.yaml  prometheus-configmap.yaml                README.md

3.3 授权

RBAC（Role-Based Access Control，基于角色的访问控制）：负责完成授权（Authorization）工作。

编写授权yaml

  1. [root@k8s-master prometheus-k8s]# vim prometheus-rbac.yaml
  2. apiVersion: v1
  3. kind: ServiceAccount
  4. metadata:
  5.   name: prometheus
  6.   namespace: kube-system
  7.   labels:
  8.     kubernetes.io/cluster-service: "true"
  9.     addonmanager.kubernetes.io/mode: Reconcile
  10. ---
  11. apiVersion: rbac.authorization.k8s.io/v1beta1
  12. kind: ClusterRole
  13. metadata:
  14.   name: prometheus
  15.   labels:
  16.     kubernetes.io/cluster-service: "true"
  17.     addonmanager.kubernetes.io/mode: Reconcile
  18. rules:
  19.   - apiGroups:
  20.       - ""
  21.     resources:
  22.       - nodes
  23.       - nodes/metrics
  24.       - services
  25.       - endpoints
  26.       - pods
  27.     verbs:
  28.       - get
  29.       - list
  30.       - watch
  31.   - apiGroups:
  32.       - ""
  33.     resources:
  34.       - configmaps
  35.     verbs:
  36.       - get
  37.   - nonResourceURLs:
  38.       - "/metrics"
  39.     verbs:
  40.       - get
  41. ---
  42. apiVersion: rbac.authorization.k8s.io/v1beta1
  43. kind: ClusterRoleBinding
  44. metadata:
  45.   name: prometheus
  46.   labels:
  47.     kubernetes.io/cluster-service: "true"
  48.     addonmanager.kubernetes.io/mode: Reconcile
  49. "prometheus-rbac.yaml" 55L, 1080C                                                                                 1,1           Top
  50. apiVersion: v1
  51. kind: ServiceAccount
  52. metadata:
  53.   name: prometheus
  54.   namespace: kube-system
  55.   labels:
  56.     kubernetes.io/cluster-service: "true"
  57.     addonmanager.kubernetes.io/mode: Reconcile
  58. ---
  59. apiVersion: rbac.authorization.k8s.io/v1beta1
  60. kind: ClusterRole
  61. metadata:
  62.   name: prometheus
  63.   labels:
  64.     kubernetes.io/cluster-service: "true"
  65.     addonmanager.kubernetes.io/mode: Reconcile
  66.                                                               1,1           Top
  67. apiVersion: v1
  68. kind: ServiceAccount
  69. metadata:
  70.   name: prometheus
  71.   namespace: kube-system
  72.   labels:
  73.     kubernetes.io/cluster-service: "true"
  74.     addonmanager.kubernetes.io/mode: Reconcile
  75. ---
  76. apiVersion: rbac.authorization.k8s.io/v1beta1
  77. kind: ClusterRole
  78. metadata:
  79.   name: prometheus
  80.   labels:
  81.     kubernetes.io/cluster-service: "true"
  82.     addonmanager.kubernetes.io/mode: Reconcile
  83. rules:
  84.   - apiGroups:
  85.       - ""
  86.     resources:
  87.       - nodes
  88.       - nodes/metrics
  89.       - services
  90.       - endpoints
  91.       - pods
  92.     verbs:
  93.       - get
  94.       - list
  95.       - watch
  96.   - apiGroups:
  97.       - ""
  98.     resources:
  99.       - configmaps
  100.     verbs:
  101.       - get
  102.   - nonResourceURLs:
  103.       - "/metrics"
  104.     verbs:
  105.       - get
  106. ---
  107. apiVersion: rbac.authorization.k8s.io/v1beta1
  108. kind: ClusterRoleBinding
  109. metadata:
  110.   name: prometheus
  111.   labels:
  112.     kubernetes.io/cluster-service: "true"
  113.     addonmanager.kubernetes.io/mode: Reconcile
  114. roleRef:
  115.   apiGroup: rbac.authorization.k8s.io
  116.   kind: ClusterRole
  117.   name: prometheus
  118. subjects:
  119. - kind: ServiceAccount
  120.   name: prometheus
  121.   namespace: kube-system

创建

  1. [root@k8s-master prometheus-k8s]# kubectl apply -f prometheus-rbac.yaml

3.4 配置管理

使用Configmap保存不需要加密配置信息

其中需要把nodes中ip地址根据自己的地址进行修改

  1. [root@k8s-master prometheus-k8s]# vim prometheus-configmap.yaml
  2. # Prometheus configuration format https://prometheus.io/docs/prometheus/latest/configuration/configuration/
  3. apiVersion: v1
  4. kind: ConfigMap #
  5. metadata:
  6.   name: prometheus-config
  7.   namespace: kube-system
  8.   labels:
  9.     kubernetes.io/cluster-service: "true"
  10.     addonmanager.kubernetes.io/mode: EnsureExists
  11. data:
  12.   prometheus.yml: |
  13.     rule_files:
  14.     - /etc/config/rules/*.rules
  16.     scrape_configs:
  17.     - job_name: prometheus
  18.       static_configs:
  19.       - targets:
  20.         - localhost:9090
  22.     - job_name: kubernetes-nodes
  23.       scrape_interval: 30s
  24.       static_configs:
  25.       - targets:
  26.         - 192.168.73.135:9100
  27.         - 192.168.73.138:9100
  28.         - 192.168.73.139:9100
  29.         - 192.168.73.140:9100
  31.     - job_name: kubernetes-apiservers
  32.       kubernetes_sd_configs:
  33.       - role: endpoints
  34.       relabel_configs:
  35.       - action: keep
  36.         regex: default;kubernetes;https
  37.         source_labels:
  38.         - __meta_kubernetes_namespace
  39.         - __meta_kubernetes_service_name
  40.         - __meta_kubernetes_endpoint_port_name
  41.       scheme: https
  42.       tls_config:
  43.         ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  44.         insecure_skip_verify: true
  45.       bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  47.     - job_name: kubernetes-nodes-kubelet
  48.       kubernetes_sd_configs:
  49.       - role: node
  50.       relabel_configs:
  51.       - action: labelmap
  52.         regex: __meta_kubernetes_node_label_(.+)
  53.       scheme: https
  54.       tls_config:
  55.         ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  56.         insecure_skip_verify: true
  57.       bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  59.     - job_name: kubernetes-nodes-cadvisor
  60.       kubernetes_sd_configs:
  61.       - role: node
  62.       relabel_configs:
  63.       - action: labelmap
  64.         regex: __meta_kubernetes_node_label_(.+)
  65.       - target_label: __metrics_path__
  66.         replacement: /metrics/cadvisor
  67.       scheme: https
  68.       tls_config:
  69.         ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  70.         insecure_skip_verify: true
  71.       bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  73.     - job_name: kubernetes-service-endpoints
  74.       kubernetes_sd_configs:
  75.       - role: endpoints
  76.       relabel_configs:
  77.       - action: keep
  78.         regex: true
  79.         source_labels:
  80.         - __meta_kubernetes_service_annotation_prometheus_io_scrape
  81.       - action: replace
  82.         regex: (https?)
  83.         source_labels:
  84.         - __meta_kubernetes_service_annotation_prometheus_io_scheme
  85.         target_label: __scheme__
  86.       - action: replace
  87.         regex: (.+)
  88.         source_labels:
  89.         - __meta_kubernetes_service_annotation_prometheus_io_path
  90.         target_label: __metrics_path__
  91.       - action: replace
  92.         regex: ([^:]+)(?::\d+)?;(\d+)
  93.         replacement: $1:$2
  94.         source_labels:
  95.         - __address__
  96.         - __meta_kubernetes_service_annotation_prometheus_io_port
  97.         target_label: __address__
  98.       - action: labelmap
  99.         regex: __meta_kubernetes_service_label_(.+)
  100.       - action: replace
  101.         source_labels:
  102.         - __meta_kubernetes_namespace
  103.         target_label: kubernetes_namespace
  104.       - action: replace
  105.         source_labels:
  106.         - __meta_kubernetes_service_name
  107.         target_label: kubernetes_name
  109.     - job_name: kubernetes-services
  110.       kubernetes_sd_configs:
  111.       - role: service
  112.       metrics_path: /probe
  113.       params:
  114.         module:
  115.         - http_2xx
  116.       relabel_configs:
  117.       - action: keep
  118.         regex: true
  119.         source_labels:
  120.         - __meta_kubernetes_service_annotation_prometheus_io_probe
  121.       - source_labels:
  122.         - __address__
  123.         target_label: __param_target
  124.       - replacement: blackbox
  125.         target_label: __address__
  126.       - source_labels:
  127.         - __param_target
  128.         target_label: instance
  129.       - action: labelmap
  130.         regex: __meta_kubernetes_service_label_(.+)
  131.       - source_labels:
  132.         - __meta_kubernetes_namespace
  133.         target_label: kubernetes_namespace
  134.       - source_labels:
  135.         - __meta_kubernetes_service_name
  136.         target_label: kubernetes_name
  138.     - job_name: kubernetes-pods
  139.       kubernetes_sd_configs:
  140.       - role: pod
  141.       relabel_configs:
  142.       - action: keep
  143.         regex: true
  144.         source_labels:
  145.         - __meta_kubernetes_pod_annotation_prometheus_io_scrape
  146.       - action: replace
  147.         regex: (.+)
  148.         source_labels:
  149.         - __meta_kubernetes_pod_annotation_prometheus_io_path
  150.         target_label: __metrics_path__
  151.       - action: replace
  152.         regex: ([^:]+)(?::\d+)?;(\d+)
  153.         replacement: $1:$2
  154.         source_labels:
  155.         - __address__
  156.         - __meta_kubernetes_pod_annotation_prometheus_io_port
  157.         target_label: __address__
  158.       - action: labelmap
  159.         regex: __meta_kubernetes_pod_label_(.+)
  160.       - action: replace
  161.         source_labels:
  162.         - __meta_kubernetes_namespace
  163.         target_label: kubernetes_namespace
  164.       - action: replace
  165.         source_labels:
  166.         - __meta_kubernetes_pod_name
  167.         target_label: kubernetes_pod_name
  168.     alerting:
  169.       alertmanagers:
  170.       - static_configs:
  171.           - targets: ["alertmanager:80"]

创建

  1. [root@k8s-master prometheus-k8s]# kubectl apply -f prometheus-configmap.yaml

3.5 有状态部署prometheus

这里使用storageclass进行动态供给，给prometheus的数据进行持久化，具体实现办法，可以查看之前的文章《k8s中的NFS动态存储供给》，除此之外可以使用静态供给的prometheus-statefulset-static-pv.yaml进行持久化

  1. [root@k8s-master prometheus-k8s]# vim prometheus-statefulset.yaml
  2. apiVersion: apps/v1
  3. kind: StatefulSet
  4. metadata:
  5.   name: prometheus
  6.   namespace: kube-system
  7.   labels:
  8.     k8s-app: prometheus
  9.     kubernetes.io/cluster-service: "true"
  10.     addonmanager.kubernetes.io/mode: Reconcile
  11.     version: v2.2.1
  12. spec:
  13.   serviceName: "prometheus"
  14.   replicas: 1
  15.   podManagementPolicy: "Parallel"
  16.   updateStrategy:
  17.    type: "RollingUpdate"
  18.   selector:
  19.     matchLabels:
  20.       k8s-app: prometheus
  21.   template:
  22.     metadata:
  23.       labels:
  24.         k8s-app: prometheus
  25.       annotations:
  26.         scheduler.alpha.kubernetes.io/critical-pod: ''
  27.     spec:
  28.       priorityClassName: system-cluster-critical
  29.       serviceAccountName: prometheus
  30.       initContainers:
  31.       - name: "init-chown-data"
  32.         image: "busybox:latest"
  33.         imagePullPolicy: "IfNotPresent"
  34.         command: ["chown", "-R", "65534:65534", "/data"]
  35.         volumeMounts:
  36.         - name: prometheus-data
  37.           mountPath: /data
  38.           subPath: ""
  39.       containers:
  40.         - name: prometheus-server-configmap-reload
  41.           image: "jimmidyson/configmap-reload:v0.1"
  42.           imagePullPolicy: "IfNotPresent"
  43.           args:
  44.             - --volume-dir=/etc/config
  45.             - --webhook-url=http://localhost:9090/-/reload
  46.           volumeMounts:
  47.             - name: config-volume
  48.               mountPath: /etc/config
  49.               readOnly: true
  50.           resources:
  51.             limits:
  52.               cpu: 10m
  53.               memory: 10Mi
  54.             requests:
  55.               cpu: 10m
  56.               memory: 10Mi
  58.         - name: prometheus-server
  59.           image: "prom/prometheus:v2.2.1"
  60.           imagePullPolicy: "IfNotPresent"
  61.           args:
  62.             - --config.file=/etc/config/prometheus.yml
  63.             - --storage.tsdb.path=/data
  64.             - --web.console.libraries=/etc/prometheus/console_libraries
  65.             - --web.console.templates=/etc/prometheus/consoles
  66.             - --web.enable-lifecycle
  67.           ports:
  68.             - containerPort: 9090
  69.           readinessProbe:
  70.             httpGet:
  71.               path: /-/ready
  72.               port: 9090
  73.             initialDelaySeconds: 30
  74.             timeoutSeconds: 30
  75.           livenessProbe:
  76.             httpGet:
  77.               path: /-/healthy
  78.               port: 9090
  79.             initialDelaySeconds: 30
  80.             timeoutSeconds: 30
  81.           # based on 10 running nodes with 30 pods each
  82.           resources:
  83.             limits:
  84.               cpu: 200m
  85.               memory: 1000Mi
  86.             requests:
  87.               cpu: 200m
  88.               memory: 1000Mi
  90.           volumeMounts:
  91.             - name: config-volume
  92.               mountPath: /etc/config
  93.             - name: prometheus-data
  94.               mountPath: /data
  95.               subPath: ""
  96.             - name: prometheus-rules
  97.               mountPath: /etc/config/rules
  99.       terminationGracePeriodSeconds: 300
  100.       volumes:
  101.         - name: config-volume
  102.           configMap:
  103.             name: prometheus-config
  104.         - name: prometheus-rules
  105.           configMap:
  106.             name: prometheus-rules
  108.   volumeClaimTemplates:
  109.   - metadata:
  110.       name: prometheus-data
  111.     spec:
  112.       storageClassName: managed-nfs-storage
  113.       accessModes:
  114.         - ReadWriteOnce
  115.       resources:
  116.         requests:
  117.           storage: "16Gi"

创建

  1. [root@k8s-master prometheus-k8s]# kubectl apply -f prometheus-statefulset.yaml

检查状态

  1. [root@k8s-master prometheus-k8s]# kubectl get pod -n kube-system
  2. NAME                                    READY   STATUS    RESTARTS   AGE
  3. alertmanager-5d75d5688f-fmlq6           2/2     Running   0          8d
  4. coredns-5bd5f9dbd9-wv45t                1/1     Running   1          8d
  5. grafana-0                               1/1     Running   2          14d
  6. kube-state-metrics-7c76bdbf68-kqqgd     2/2     Running   6          13d
  7. kubernetes-dashboard-7d77666777-d5ng4   1/1     Running   5          14d
  8. prometheus-0                            2/2     Running   6          14d

可以看到一个prometheus-0的pod，这就刚才使用statefulset控制器进行的有状态部署，状态为Runing则是正常，如果不为Runing可以使用kubectl describe pod prometheus-0 -n kube-system查看报错详情

3.6 创建service暴露访问端口

此处使用nodePort固定一个访问端口，便于记忆

  1. [root@k8s-master prometheus-k8s]# vim prometheus-service.yaml
  2. kind: Service
  3. apiVersion: v1
  4. metadata:
  5.   name: prometheus
  6.   namespace: kube-system
  7.   labels:
  8.     kubernetes.io/name: "Prometheus"
  9.     kubernetes.io/cluster-service: "true"
  10.     addonmanager.kubernetes.io/mode: Reconcile
  11. spec:
  12.   type: NodePort
  13.   ports:
  14.     - name: http
  15.       port: 9090
  16.       protocol: TCP
  17.       targetPort: 9090
  18.       nodePort: 30090
  19.   selector:
  20.     k8s-app: prometheus

创建

  1. [root@k8s-master prometheus-k8s]# kubectl apply -f prometheus-service.yaml

检查

  1. [root@k8s-master prometheus-k8s]# kubectl get pod,svc -n kube-system
  2. NAME                                        READY   STATUS    RESTARTS   AGE
  3. pod/coredns-5bd5f9dbd9-wv45t                1/1     Running   1          8d
  4. pod/kubernetes-dashboard-7d77666777-d5ng4   1/1     Running   5          14d
  5. pod/prometheus-0                            2/2     Running   6          14d
  7. NAME                           TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)             AGE
  8. service/kube-dns               ClusterIP   10.0.0.2     <none>        53/UDP,53/TCP       13d
  9. service/kubernetes-dashboard   NodePort    10.0.0.127   <none>        443:30001/TCP       16d
  10. service/prometheus             NodePort    10.0.0.33    <none>        9090:30090/TCP      13d

3.7 web访问

使用任意一个NodeIP加端口进行访问，访问地址：http://NodeIP:Port ,此例就是：http://192.168.73.139:30090

访问成功的界面如图所示：

4 在K8S平台部署Grafana

通过上面的web访问，可以看出prometheus自带的UI界面是没有多少功能的，可视化展示的功能不完善，不能满足日常的监控所需，因此常常我们需要再结合Prometheus+Grafana的方式来进行可视化的数据展示

官网地址：

https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/prometheus

https://grafana.com/grafana/download

刚才下载的项目中已经写好了Grafana的yaml，根据自己的环境进行修改

4.1 使用StatefulSet部署grafana

  1. [root@k8s-master prometheus-k8s]# vim grafana.yaml
  2. apiVersion: apps/v1
  3. kind: StatefulSet
  4. metadata:
  5.   name: grafana
  6.   namespace: kube-system
  7. spec:
  8.   serviceName: "grafana"
  9.   replicas: 1
  10.   selector:
  11.     matchLabels:
  12.       app: grafana
  13.   template:
  14.     metadata:
  15.       labels:
  16.         app: grafana
  17.     spec:
  18.       containers:
  19.       - name: grafana
  20.         image: grafana/grafana
  21.         ports:
  22.           - containerPort: 3000
  23.             protocol: TCP
  24.         resources:
  25.           limits:
  26.             cpu: 100m
  27.             memory: 256Mi
  28.           requests:
  29.             cpu: 100m
  30.             memory: 256Mi
  31.         volumeMounts:
  32.           - name: grafana-data
  33.             mountPath: /var/lib/grafana
  34.             subPath: grafana
  35.       securityContext:
  36.         fsGroup: 472
  37.         runAsUser: 472
  38.   volumeClaimTemplates:
  39.   - metadata:
  40.       name: grafana-data
  41.     spec:
  42.       storageClassName: managed-nfs-storage
  43.       accessModes:
  44.         - ReadWriteOnce
  45.       resources:
  46.         requests:
  47.           storage: "1Gi"
  49. ---
  51. apiVersion: v1
  52. kind: Service
  53. metadata:
  54.   name: grafana
  55.   namespace: kube-system
  56. spec:
  57.   type: NodePort
  58.   ports:
  59.   - port : 80
  60.     targetPort: 3000
  61.     nodePort: 30091
  62.   selector:
  63.     app: grafana

4.2 Grafana的web访问

使用任意一个NodeIP加端口进行访问，访问地址：http://NodeIP:Port ,此例就是：http://192.168.73.139:30091

成功访问界面如下，会需要进行账号密码登陆，默认账号密码都为admin，登陆之后会让修改密码

登陆之后的界面如下

第一步需要进行数据源添加，点击create your first data source数据库图标，根据下图所示进行添加即可

第二步，添加完了之后点击底部的绿色的Save&Test，会成功提示Data sourse is working，则表示数据源添加成功

4.3 监控K8S集群中Pod、Node、资源对象

Pod

kubelet的节点使用cAdvisor提供的metrics接口获取该节点所有Pod和容器相关的性能指标数据。

暴露接口地址：

https://NodeIP:10255/metrics/cadvisor

https://NodeIP:10250/metrics/cadvisor

Node

使用node_exporter收集器采集节点资源利用率。

https://github.com/prometheus/node_exporter

使用文档：https://prometheus.io/docs/guides/node-exporter/

使用node_exporter.sh脚本分别在三台服务器上部署node_exporter收集器，不需要修改可直接运行脚本

  1. [root@k8s-master prometheus-k8s]# cat node_exporter.sh
  2. #!/bin/bash
  4. wget https://github.com/prometheus/node_exporter/releases/download/v0.17.0/node_exporter-0.17.0.linux-amd64.tar.gz
  6. tar zxf node_exporter-0.17.0.linux-amd64.tar.gz
  7. mv node_exporter-0.17.0.linux-amd64 /usr/local/node_exporter
  9. cat <<EOF >/usr/lib/systemd/system/node_exporter.service
  10. [Unit]
  11. Description=https://prometheus.io
  13. [Service]
  14. Restart=on-failure
  15. ExecStart=/usr/local/node_exporter/node_exporter --collector.systemd --collector.systemd.unit-whitelist=(docker|kubelet|kube-proxy|flanneld).service
  17. [Install]
  18. WantedBy=multi-user.target
  19. EOF
  21. systemctl daemon-reload
  22. systemctl enable node_exporter
  23. systemctl restart node_exporter
  24. [root@k8s-master prometheus-k8s]# ./node_exporter.sh
  • 检测node_exporter的进程，是否生效
  1. [root@k8s-master prometheus-k8s]# ps -ef|grep node_exporter
  2. root       6227      1  0 Oct08 ?        00:06:43 /usr/local/node_exporter/node_exporter --collector.systemd --collector.systemd.unit-whitelist=(docker|kubelet|kube-proxy|flanneld).service
  3. root     118269 117584  0 23:27 pts/0    00:00:00 grep --color=auto node_exporter

资源对象

kube-state-metrics采集了k8s中各种资源对象的状态信息，只需要在master节点部署就行

https://github.com/kubernetes/kube-state-metrics

  1. 创建rbac的yaml对metrics进行授权
  1. [root@k8s-master prometheus-k8s]# vim kube-state-metrics-rbac.yaml
  2. apiVersion: v1
  3. kind: ServiceAccount
  4. metadata:
  5.   name: kube-state-metrics
  6.   namespace: kube-system
  7.   labels:
  8.     kubernetes.io/cluster-service: "true"
  9.     addonmanager.kubernetes.io/mode: Reconcile
  10. ---
  11. apiVersion: rbac.authorization.k8s.io/v1
  12. kind: ClusterRole
  13. metadata:
  14.   name: kube-state-metrics
  15.   labels:
  16.     kubernetes.io/cluster-service: "true"
  17.     addonmanager.kubernetes.io/mode: Reconcile
  18. rules:
  19. - apiGroups: [""]
  20.   resources:
  21.   - configmaps
  22.   - secrets
  23.   - nodes
  24.   - pods
  25.   - services
  26.   - resourcequotas
  27.   - replicationcontrollers
  28.   - limitranges
  29.   - persistentvolumeclaims
  30.   - persistentvolumes
  31.   - namespaces
  32.   - endpoints
  33.   verbs: ["list", "watch"]
  34. - apiGroups: ["extensions"]
  35.   resources:
  36.   - daemonsets
  37.   - deployments
  38.   - replicasets
  39.   verbs: ["list", "watch"]
  40. - apiGroups: ["apps"]
  41.   resources:
  42.   - statefulsets
  43.   verbs: ["list", "watch"]
  44. - apiGroups: ["batch"]
  45.   resources:
  46.   - cronjobs
  47.   - jobs
  48.   verbs: ["list", "watch"]
  49. - apiGroups: ["autoscaling"]
  50.   resources:
  51.   - horizontalpodautoscalers
  52.   verbs: ["list", "watch"]
  53. ---
  54. apiVersion: rbac.authorization.k8s.io/v1
  55. kind: Role
  56. metadata:
  57.   name: kube-state-metrics-resizer
  58.   namespace: kube-system
  59.   labels:
  60.     kubernetes.io/cluster-service: "true"
  61.     addonmanager.kubernetes.io/mode: Reconcile
  62. rules:
  63. - apiGroups: [""]
  64.   resources:
  65.   - pods
  66.   verbs: ["get"]
  67. - apiGroups: ["extensions"]
  68.   resources:
  69.   - deployments
  70.   resourceNames: ["kube-state-metrics"]
  71.   verbs: ["get", "update"]
  72. ---
  73. apiVersion: rbac.authorization.k8s.io/v1
  74. kind: ClusterRoleBinding
  75. metadata:
  76.   name: kube-state-metrics
  77.   labels:
  78.     kubernetes.io/cluster-service: "true"
  79.     addonmanager.kubernetes.io/mode: Reconcile
  80. roleRef:
  81.   apiGroup: rbac.authorization.k8s.io
  82.   kind: ClusterRole
  83.   name: kube-state-metrics
  84. subjects:
  85. - kind: ServiceAccount
  86.   name: kube-state-metrics
  87.   namespace: kube-system
  88. ---
  89. apiVersion: rbac.authorization.k8s.io/v1
  90. kind: RoleBinding
  91. metadata:
  92.   name: kube-state-metrics
  93.   namespace: kube-system
  94.   labels:
  95.     kubernetes.io/cluster-service: "true"
  96.     addonmanager.kubernetes.io/mode: Reconcile
  97. roleRef:
  98.   apiGroup: rbac.authorization.k8s.io
  99.   kind: Role
  100.   name: kube-state-metrics-resizer
  101. subjects:
  102. - kind: ServiceAccount
  103.   name: kube-state-metrics
  104.   namespace: kube-system
  105. [root@k8s-master prometheus-k8s]# kubectl apply -f kube-state-metrics-rbac.yaml
  1. 编写Deployment和ConfigMap的yaml进行metrics pod部署，不需要进行修改
  1. [root@k8s-master prometheus-k8s]# cat kube-state-metrics-deployment.yaml
  2. apiVersion: apps/v1
  3. kind: Deployment
  4. metadata:
  5.   name: kube-state-metrics
  6.   namespace: kube-system
  7.   labels:
  8.     k8s-app: kube-state-metrics
  9.     kubernetes.io/cluster-service: "true"
  10.     addonmanager.kubernetes.io/mode: Reconcile
  11.     version: v1.3.0
  12. spec:
  13.   selector:
  14.     matchLabels:
  15.       k8s-app: kube-state-metrics
  16.       version: v1.3.0
  17.   replicas: 1
  18.   template:
  19.     metadata:
  20.       labels:
  21.         k8s-app: kube-state-metrics
  22.         version: v1.3.0
  23.       annotations:
  24.         scheduler.alpha.kubernetes.io/critical-pod: ''
  25.     spec:
  26.       priorityClassName: system-cluster-critical
  27.       serviceAccountName: kube-state-metrics
  28.       containers:
  29.       - name: kube-state-metrics
  30.         image: lizhenliang/kube-state-metrics:v1.3.0
  31.         ports:
  32.         - name: http-metrics
  33.           containerPort: 8080
  34.         - name: telemetry
  35.           containerPort: 8081
  36.         readinessProbe:
  37.           httpGet:
  38.             path: /healthz
  39.             port: 8080
  40.           initialDelaySeconds: 5
  41.           timeoutSeconds: 5
  42.       - name: addon-resizer
  43.         image: lizhenliang/addon-resizer:1.8.3
  44.         resources:
  45.           limits:
  46.             cpu: 100m
  47.             memory: 30Mi
  48.           requests:
  49.             cpu: 100m
  50.             memory: 30Mi
  51.         env:
  52.           - name: MY_POD_NAME
  53.             valueFrom:
  54.               fieldRef:
  55.                 fieldPath: metadata.name
  56.           - name: MY_POD_NAMESPACE
  57.             valueFrom:
  58.               fieldRef:
  59.                 fieldPath: metadata.namespace
  60.         volumeMounts:
  61.           - name: config-volume
  62.             mountPath: /etc/config
  63.         command:
  64.           - /pod_nanny
  65.           - --config-dir=/etc/config
  66.           - --container=kube-state-metrics
  67.           - --cpu=100m
  68.           - --extra-cpu=1m
  69.           - --memory=100Mi
  70.           - --extra-memory=2Mi
  71.           - --threshold=5
  72.           - --deployment=kube-state-metrics
  73.       volumes:
  74.         - name: config-volume
  75.           configMap:
  76.             name: kube-state-metrics-config
  77. ---
  78. # Config map for resource configuration.
  79. apiVersion: v1
  80. kind: ConfigMap
  81. metadata:
  82.   name: kube-state-metrics-config
  83.   namespace: kube-system
  84.   labels:
  85.     k8s-app: kube-state-metrics
  86.     kubernetes.io/cluster-service: "true"
  87.     addonmanager.kubernetes.io/mode: Reconcile
  88. data:
  89.   NannyConfiguration: |-
  90.     apiVersion: nannyconfig/v1alpha1
  91.     kind: NannyConfiguration
  92. [root@k8s-master prometheus-k8s]# kubectl apply -f kube-state-metrics-deployment.yaml
  1. 编写Service的yaml对metrics进行端口暴露
  1. [root@k8s-master prometheus-k8s]# cat kube-state-metrics-service.yaml
  2. apiVersion: v1
  3. kind: Service
  4. metadata:
  5.   name: kube-state-metrics
  6.   namespace: kube-system
  7.   labels:
  8.     kubernetes.io/cluster-service: "true"
  9.     addonmanager.kubernetes.io/mode: Reconcile
  10.     kubernetes.io/name: "kube-state-metrics"
  11.   annotations:
  12.     prometheus.io/scrape: 'true'
  13. spec:
  14.   ports:
  15.   - name: http-metrics
  16.     port: 8080
  17.     targetPort: http-metrics
  18.     protocol: TCP
  19.   - name: telemetry
  20.     port: 8081
  21.     targetPort: telemetry
  22.     protocol: TCP
  23.   selector:
  24.     k8s-app: kube-state-metrics
  25. [root@k8s-master prometheus-k8s]# kubectl apply -f kube-state-metrics-service.yaml
  1. 检查pod和svc的状态，可以看到正常运行了pod/kube-state-metrics-7c76bdbf68-kqqgd 和对外暴露了8080和8081端口
  1. [root@k8s-master prometheus-k8s]# kubectl get pod,svc -n kube-system
  2. NAME                                        READY   STATUS    RESTARTS   AGE
  3. pod/alertmanager-5d75d5688f-fmlq6           2/2     Running   0          9d
  4. pod/coredns-5bd5f9dbd9-wv45t                1/1     Running   1          9d
  5. pod/grafana-0                               1/1     Running   2          15d
  6. pod/kube-state-metrics-7c76bdbf68-kqqgd     2/2     Running   6          14d
  7. pod/kubernetes-dashboard-7d77666777-d5ng4   1/1     Running   5          16d
  8. pod/prometheus-0                            2/2     Running   6          15d
  10. NAME                           TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)             AGE
  11. service/alertmanager           ClusterIP   10.0.0.207   <none>        80/TCP              13d
  12. service/grafana                NodePort    10.0.0.74    <none>        80:30091/TCP        15d
  13. service/kube-dns               ClusterIP   10.0.0.2     <none>        53/UDP,53/TCP       14d
  14. service/kube-state-metrics     ClusterIP   10.0.0.194   <none>        8080/TCP,8081/TCP   14d
  15. service/kubernetes-dashboard   NodePort    10.0.0.127   <none>        443:30001/TCP       17d
  16. service/prometheus             NodePort    10.0.0.33    <none>        9090:30090/TCP      14d
  17. [root@k8s-master prometheus-k8s]#

5 使用Grafana可视化展示Prometheus监控数据

通常在使用Prometheus采集数据的时候我们需要监控K8S集群中Pod、Node、资源对象，因此我们需要安装对应的插件和资源采集器来提供api进行数据获取，在4.3中我们已经配置好，我们也可以使用Prometheus的UI界面中的Staus菜单下的Target中的各个采集器的状态情况，如图所示：

只有当我们各个Target的状态都是UP状态时，我们可以使用自带的的界面去获取到某一监控项的相关的数据，如图所示：

从上面的图中可以看出Prometheus的界面可视化展示的功能较单一，不能满足需求，因此我们需要结合Grafana来进行可视化展示Prometheus监控数据，在上一章节，已经成功部署了Granfana，因此需要在使用的时候添加dashboard和Panel来设计展示相关的监控项，但是实际上在Granfana社区里面有很多成熟的模板，我们可以直接使用，然后根据自己的环境修改Panel中的查询语句来获取数据

https://grafana.com/grafana/dashboards

推荐模板：

  • 集群资源监控：3119

当模板添加之后如果某一个Panel不显示数据，可以点击Panel上的编辑，查询PromQL语句，然后去Prometheus自己的界面上进行调试PromQL语句是否可以获取到值，最后调整之后的监控界面如图所示

  • 资源状态监控：6417

    同理，添加资源状态的监控模板，然后经过调整之后的监控界面如图所示，可以获取到k8s中各种资源状态的监控展示

  • Node监控：9276

    同理，添加资源状态的监控模板，然后经过调整之后的监控界面如图所示，可以获取到各个node上的基本情况

6 在K8S中部署Alertmanager

6.1 部署Alertmanager

6.2 部署告警

我们以Email来进行实现告警信息的发送

  1. 首先需要准备一个发件邮箱，开启stmp发送功能
  2. 使用configmap存储告警规则，编写报警规则的yaml文件，可根据自己的实际情况进行修改和添加报警的规则，prometheus比zabbix就麻烦在这里，所有的告警规则需要自己去定义
  1. [root@k8s-master prometheus-k8s]# vim prometheus-rules.yaml
  2. apiVersion: v1
  3. kind: ConfigMap
  4. metadata:
  5.   name: prometheus-rules
  6.   namespace: kube-system
  7. data:
  8.   general.rules: |
  9.     groups:
  10.     - name: general.rules
  11.       rules:
  12.       - alert: InstanceDown
  13.         expr: up == 0
  14.         for: 1m
  15.         labels:
  16.           severity: error
  17.         annotations:
  18.           summary: "Instance {{ $labels.instance }} 停止工作"
  19.           description: "{{ $labels.instance }} job {{ $labels.job }} 已经停止5分钟以上."
  20.   node.rules: |
  21.     groups:
  22.     - name: node.rules
  23.       rules:
  24.       - alert: NodeFilesystemUsage
  25.         expr: 100 - (node_filesystem_free_bytes{fstype=~"ext4|xfs"} / node_filesystem_size_bytes{fstype=~"ext4|xfs"} * 100) > 80
  26.         for: 1m
  27.         labels:
  28.           severity: warning
  29.         annotations:
  30.           summary: "Instance {{ $labels.instance }} : {{ $labels.mountpoint }} 分区使用率过高"
  31.           description: "{{ $labels.instance }}: {{ $labels.mountpoint }} 分区使用大于80% (当前值: {{ $value }})"
  33.       - alert: NodeMemoryUsage
  34.         expr: 100 - (node_memory_MemFree_bytes+node_memory_Cached_bytes+node_memory_Buffers_bytes) / node_memory_MemTotal_bytes * 10
  35. 0 > 80
  36.         for: 1m
  37.         labels:
  38.           severity: warning
  39.         annotations:
  40.           summary: "Instance {{ $labels.instance }} 内存使用率过高"
  41.           description: "{{ $labels.instance }}内存使用大于80% (当前值: {{ $value }})"
  43.       - alert: NodeCPUUsage
  44.         expr: 100 - (avg(irate(node_cpu_seconds_total{mode="idle"}[5m])) by (instance) * 100) > 60
  45.         for: 1m
  46.         labels:
  47.           severity: warning
  48.         annotations:
  49.           summary: "Instance {{ $labels.instance }} CPU使用率过高"
  50.           description: "{{ $labels.instance }}CPU使用大于60% (当前值: {{ $value }})"
  51. [root@k8s-master prometheus-k8s]# kubectl apply -f prometheus-rules.yaml
  1. 编写告警configmap的yaml文件部署，增加alertmanager告警配置，进行配置邮箱发送地址
  1. [root@k8s-master prometheus-k8s]# vim alertmanager-configmap.yaml
  2. apiVersion: v1
  3. kind: ConfigMap
  4. metadata:
  5.   name: alertmanager-config
  6.   namespace: kube-system
  7.   labels:
  8.     kubernetes.io/cluster-service: "true"
  9.     addonmanager.kubernetes.io/mode: EnsureExists
  10. data:
  11.   alertmanager.yml: |
  12.     global:
  13.       resolve_timeout: 5m
  14.       smtp_smarthost: 'mail.goldwind.com.cn:587'  #登陆邮件进行查看
  15.       smtp_from: 'goldwindscada@goldwind.com.cn' #根据自己申请的发件邮箱进行配置
  16.       smtp_auth_username: 'goldwindscada@goldwind.com.cn'
  17.       smtp_auth_password: 'Dbadmin@123'
  19.     receivers:
  20.     - name: default-receiver
  21.       email_configs:
  22.       - to: "zhangdongdong27459@goldwind.com.cn"
  24.     route:
  25.       group_interval: 1m
  26.       group_wait: 10s
  27.       receiver: default-receiver
  28.       repeat_interval: 1m
  29. [root@k8s-master prometheus-k8s]# kubectl apply -f alertmanager-configmap.yaml
  1. 创建PVC进行数据持久化，我这个yaml文件使用的跟Prometheus安装时用的存储类来进行自动供给，需要根据自己的实际情况修改
  1. [root@k8s-master prometheus-k8s]# vim alertmanager-pvc.yaml
  2. apiVersion: v1
  3. kind: PersistentVolumeClaim
  4. metadata:
  5.   name: alertmanager
  6.   namespace: kube-system
  7.   labels:
  8.     kubernetes.io/cluster-service: "true"
  9.     addonmanager.kubernetes.io/mode: EnsureExists
  10. spec:
  11.   storageClassName: managed-nfs-storage
  12.   accessModes:
  13.     - ReadWriteOnce
  14.   resources:
  15.     requests:
  16.       storage: "2Gi"
  17. [root@k8s-master prometheus-k8s]# kubectl apply -f alertmanager-pvc.yaml
  1. 编写deployment的yaml来部署alertmanager的pod
  1. [root@k8s-master prometheus-k8s]# vim alertmanager-deployment.yaml
  2. apiVersion: apps/v1
  3. kind: Deployment
  4. metadata:
  5.   name: alertmanager
  6.   namespace: kube-system
  7.   labels:
  8.     k8s-app: alertmanager
  9.     kubernetes.io/cluster-service: "true"
  10.     addonmanager.kubernetes.io/mode: Reconcile
  11.     version: v0.14.0
  12. spec:
  13.   replicas: 1
  14.   selector:
  15.     matchLabels:
  16.       k8s-app: alertmanager
  17.       version: v0.14.0
  18.   template:
  19.     metadata:
  20.       labels:
  21.         k8s-app: alertmanager
  22.         version: v0.14.0
  23.       annotations:
  24.         scheduler.alpha.kubernetes.io/critical-pod: ''
  25.     spec:
  26.       priorityClassName: system-cluster-critical
  27.       containers:
  28.         - name: prometheus-alertmanager
  29.           image: "prom/alertmanager:v0.14.0"
  30.           imagePullPolicy: "IfNotPresent"
  31.           args:
  32.             - --config.file=/etc/config/alertmanager.yml
  33.             - --storage.path=/data
  34.             - --web.external-url=/
  35.           ports:
  36.             - containerPort: 9093
  37.           readinessProbe:
  38.             httpGet:
  39.               path: /#/status
  40.               port: 9093
  41.             initialDelaySeconds: 30
  42.             timeoutSeconds: 30
  43.           volumeMounts:
  44.             - name: config-volume
  45.               mountPath: /etc/config
  46.             - name: storage-volume
  47.               mountPath: "/data"
  48.               subPath: ""
  49.           resources:
  50.             limits:
  51.               cpu: 10m
  52.               memory: 50Mi
  53.             requests:
  54.               cpu: 10m
  55.               memory: 50Mi
  56.         - name: prometheus-alertmanager-configmap-reload
  57.           image: "jimmidyson/configmap-reload:v0.1"
  58.           imagePullPolicy: "IfNotPresent"
  59.           args:
  60.             - --volume-dir=/etc/config
  61.             - --webhook-url=http://localhost:9093/-/reload
  62.           volumeMounts:
  63.             - name: config-volume
  64.               mountPath: /etc/config
  65.               readOnly: true
  66.           resources:
  67.             limits:
  68.               cpu: 10m
  69.               memory: 10Mi
  70.             requests:
  71.               cpu: 10m
  72.               memory: 10Mi
  73.       volumes:
  74.         - name: config-volume
  75.           configMap:
  76.             name: alertmanager-config
  77.         - name: storage-volume
  78.           persistentVolumeClaim:
  79.             claimName: alertmanager
  80. [root@k8s-master prometheus-k8s]# kubectl apply -f alertmanager-deployment.yaml
  1. 创建 alertmanager的service对外暴露的端口
  1. [root@k8s-master prometheus-k8s]# vim alertmanager-service.yaml
  2. apiVersion: v1
  3. kind: Service
  4. metadata:
  5.   name: alertmanager
  6.   namespace: kube-system
  7.   labels:
  8.     kubernetes.io/cluster-service: "true"
  9.     addonmanager.kubernetes.io/mode: Reconcile
  10.     kubernetes.io/name: "Alertmanager"
  11. spec:
  12.   ports:
  13.     - name: http
  14.       port: 80
  15.       protocol: TCP
  16.       targetPort: 9093
  17.   selector:
  18.     k8s-app: alertmanager
  19.   type: "ClusterIP"
  20.   [root@k8s-master prometheus-k8s]# kubectl apply -f alertmanager-service.yaml
  1. 检测部署状态，可以发现pod/alertmanager-5d75d5688f-fmlq6和service/alertmanager正常运行
  1. [root@k8s-master prometheus-k8s]# kubectl get pod,svc -n kube-system -o wide
  2. NAME                                        READY   STATUS    RESTARTS   AGE   IP            NODE             NOMINATED NODE   READINESS GATES
  3. pod/alertmanager-5d75d5688f-fmlq6           2/2     Running   4          10d   172.17.15.2   192.168.73.140   <none>           <none>
  4. pod/coredns-5bd5f9dbd9-qxvmz                1/1     Running   0          42m   172.17.33.2   192.168.73.138   <none>           <none>
  5. pod/grafana-0                               1/1     Running   3          16d   172.17.31.2   192.168.73.139   <none>           <none>
  6. pod/kube-state-metrics-7c76bdbf68-hv56m     2/2     Running   0          23h   172.17.15.3   192.168.73.140   <none>           <none>
  7. pod/kubernetes-dashboard-7d77666777-d5ng4   1/1     Running   6          17d   172.17.31.4   192.168.73.139   <none>           <none>
  8. pod/prometheus-0                            2/2     Running   8          16d   172.17.83.2   192.168.73.135   <none>           <none>
  10. NAME                           TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)             AGE   SELECTOR
  11. service/alertmanager           ClusterIP   10.0.0.207   <none>        80/TCP              14d   k8s-app=alertmanager
  12. service/grafana                NodePort    10.0.0.74    <none>        80:30091/TCP        16d   app=grafana
  13. service/kube-dns               ClusterIP   10.0.0.2     <none>        53/UDP,53/TCP       42m   k8s-app=kube-dns
  14. service/kube-state-metrics     ClusterIP   10.0.0.194   <none>        8080/TCP,8081/TCP   15d   k8s-app=kube-state-metrics
  15. service/kubernetes-dashboard   NodePort    10.0.0.127   <none>        443:30001/TCP       18d   k8s-app=kubernetes-dashboard
  16. service/prometheus             NodePort    10.0.0.33    <none>        9090:30090/TCP      15d   k8s-app=prometheus

6.3 测试告警发送

k8s实战之部署Prometheus+Grafana可视化监控告警平台的更多相关文章

  1. Spring Boot 微服务应用集成Prometheus + Grafana 实现监控告警

    Spring Boot 微服务应用集成Prometheus + Grafana 实现监控告警 一.添加依赖 1.1 Actuator 的 /prometheus端点 二.Prometheus 配置 部 ...

  2. Prometheus+Grafana企业监控系统

    Prometheus+Grafana企业监控系统 作者 刘畅 实验配置: 主机名称 Ip地址 controlnode 172.16.1.70/24 slavenode1 172.16.1.71/24 ...

  3. 基于k8s集群部署prometheus监控ingress nginx

    目录 基于k8s集群部署prometheus监控ingress nginx 1.背景和环境概述 2.修改prometheus配置 3.检查是否生效 4.配置grafana图形 基于k8s集群部署pro ...

  4. 基于k8s集群部署prometheus监控etcd

    目录 基于k8s集群部署prometheus监控etcd 1.背景和环境概述 2.修改prometheus配置 3.检查是否生效 4.配置grafana图形 基于k8s集群部署prometheus监控 ...

  5. go-zero docker-compose 搭建课件服务（七）：prometheus+grafana服务监控

    0.转载 go-zero docker-compose 搭建课件服务(七):prometheus+grafana服务监控 0.1源码地址 https://github.com/liuyuede123/ ...

  6. Prometheus Grafana可视化展示Linux资源使用率

    Prometheus Grafana可视化展示Linux资源使用率  Grfana官方仪表盘下载:https://grafana.com/dashboards 数据源推荐:https://grafan ...

  7. 部署Prometheus+Grafana监控

    Prometheus 1.不是很友好,各种配置都手写 2.对docker和k8s监控有成熟解决方案 Prometheus(普罗米修斯) 是一个最初在SoudCloud上构建的监控系统,开源项目,拥有非 ...

  8. Rancher2.x 一键式部署 Prometheus + Grafana 监控 Kubernetes 集群

    目录 1.Prometheus & Grafana 介绍 2.环境.软件准备 3.Rancher 2.x 应用商店 4.一键式部署 Prometheus 5.验证 Prometheus + G ...

  9. k8b部署prometheus+grafana

    来源: https://juejin.im/post/5c36054251882525a50bbdf0 https://github.com/redhatxl/k8s-prometheus-grafa ...

随机推荐

  1. 「MySql高级查询与编程」练习：企业员工管理

    题目:企业员工管理 一.语言和环境 1.实现语言:SQL. 2.开发环境:MySQL,navicat for mysql. 二.题目(100分): 1.创建数据库及数据表: (1)创建数据库,名称为d ...

  2. x86-2-保护模式

    x86-2-保护模式 操作系统负责计算机上的所有软件和硬件的管理,它可以百分百操作计算机的所有内容.但是,操作系统上编写的用户程序却应当有所限制,只允许用户程序访问属于自己程序的内容,不然整个生态就很 ...

  3. Windows实现桌面录屏、指定窗口录制直播，低延时，H5页面播放

    接着前面记录的3种方式实现桌面推流直播: 1.Windows 11实现录屏直播,搭建Nginx的rtmp服务 的方式需要依赖与Flash插件,使用场景有限 2.Windows 11实现直播,VLC超简 ...

  4. Python_获取全部异常信息

    import traceback try: os.getcwd('exc') except Exception: exc = traceback.format_exc() print(exc)

  5. vue2.0与vue3.0项目创建

    脚手架安装与卸载 安装 npm install -g vue-cli //or npm install -g @vue/cli 卸载 npm uninstall -g vue-cli //or npm ...

  6. Typora+PicGo-Core实现图片自动上传gitee图床

    说明: 使用gitee作为图床: 客户机为Mac M1: Typora版本:1.0.2 (5990). gitee配置步骤 需要拥有一个gitee账号,创建一个公有仓库用于存储图片,然后需要生成一个t ...

  7. 使用Crossplane构建专属PaaS体验：Kubernetes，OAM和CoreWorkflows

    关键点:•许多组织的目标是构建自己的云平台,这些平台通常由内部部署架构和云供应商共建完成.•虽然Kubernetes没有提供开箱即用的完整PaaS平台式服务,但其具备完整的API,清晰的技术抽取和易理 ...

  8. 手把手教你分析解决MySQL死锁问题

    在生产环境中如果出现MySQL死锁问题该如何排查和解决呢,本文将模拟真实死锁场景进行排查,最后总结下实际开发中如何尽量避免死锁发生. 一.准备好相关数据和环境 当前自己的数据版本是8.0.22 mys ...

  9. 【分享】让prometheus支持PUSH模式，可以使用remote write协议推送数据

    2021-10-21补充: 我通过修改源码来让prometheus支持remote-write,纯属多此一举: --enable-feature=remote-write-receiver这个命令行参 ...

  10. CesiumJS新增官方TypeScript类型定义

    Cesium中文网:http://cesiumcn.org/ | 国内快速访问:http://cesium.coinidea.com/ 在当前的1.70版本中,CesiumJS现在附带了正式的Type ...