来源：

https://juejin.im/post/5c36054251882525a50bbdf0

https://github.com/redhatxl/k8s-prometheus-grafana

Prometheus

master/node节点环境部署

git clone https://github.com/redhatxl/k8s-prometheus-grafana.git

在node节点下载监控所需镜像（在所有节点也下载了）

  1. docker pull prom/node-exporter
  2. docker pull prom/prometheus:v2.0.0
  3. docker pull grafana/grafana:4.2.0

采用daemonset方式部署node-exporter组件

kubectl create -f node-exporter.yaml

部署prometheus组件

rbac文件

kubectl create -f k8s-prometheus-grafana/prometheus/rbac-setup.yaml

以configmap的形式管理prometheus组件的配置文件

kubectl create -f k8s-prometheus-grafana/prometheus/configmap.yaml

Prometheus deployment 文件

kubectl create -f k8s-prometheus-grafana/prometheus/prometheus.deploy.yml

Prometheus service文件

kubectl create -f k8s-prometheus-grafana/prometheus/prometheus.svc.yml

部署grafana组件

grafana deployment配置文件

kubectl create -f k8s-prometheus-grafana/grafana/grafana-deploy.yaml

grafana service配置文件

kubectl create -f k8s-prometheus-grafana/grafana/grafana-svc.yaml

grafana ingress配置文件

kubectl create -f k8s-prometheus-grafana/grafana/grafana-ing.yaml

访问

  1.  k get po -n kube-system   -owide
  2. NAME                                 READY   STATUS    RESTARTS   AGE     IP                NODE         NOMINATED NODE   READINESS GATES
  3. coredns-5c98db65d4-j2925             1/1     Running   4          2d23h   10.244.0.13       k8s-master   <none>           <none>
  4. coredns-5c98db65d4-k2rcj             1/1     Running   4          2d23h   10.244.0.12       k8s-master   <none>           <none>
  5. etcd-k8s-master                      1/1     Running   4          2d23h   192.168.190.140   k8s-master   <none>           <none>
  6. grafana-core-6ff599bfdc-pbdzf        1/1     Running   0          99m     10.244.1.66       k8s-node1    <none>           <none>
  7. kube-apiserver-k8s-master            1/1     Running   4          2d23h   192.168.190.140   k8s-master   <none>           <none>
  8. kube-controller-manager-k8s-master   1/1     Running   4          2d23h   192.168.190.140   k8s-master   <none>           <none>
  9. kube-flannel-ds-amd64-6rq62          1/1     Running   5          2d23h   192.168.190.140   k8s-master   <none>           <none>
  10. kube-flannel-ds-amd64-ctmdz          1/1     Running   4          2d23h   192.168.190.141   k8s-node1    <none>           <none>
  11. kube-proxy-kmgc5                     1/1     Running   4          2d23h   192.168.190.140   k8s-master   <none>           <none>
  12. kube-proxy-ss8jr                     1/1     Running   4          2d23h   192.168.190.141   k8s-node1    <none>           <none>
  13. kube-scheduler-k8s-master            1/1     Running   4          2d23h   192.168.190.140   k8s-master   <none>           <none>
  14. node-exporter-fsfkb                  1/1     Running   0          102m    10.244.1.64       k8s-node1    <none>           <none>
  15. prometheus-68545d4fd8-hnltb          1/1     Running   0          101m    10.244.1.65       k8s-node1    <none>           <none>
  17. k get svc  -n kube-system
  18. NAME            TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                  AGE
  19. grafana         NodePort    10.106.70.200    <none>        3000:30592/TCP           98m
  20. kube-dns        ClusterIP   10.96.0.10       <none>        53/UDP,53/TCP,9153/TCP   2d23h
  21. node-exporter   NodePort    10.102.245.168   <none>        9100:31672/TCP           101m
  22. prometheus      NodePort    10.101.205.6     <none>        9090:30003/TCP           99m

访问prometheus ：

任一个节点的物理地址+暴露的端口（9090:30003/TCP ）

访问grafana:

任一个节点的物理地址+暴露的端口（3000:30592/TCP）

输入密码登录：

admin/admin

添加数据源

类型：prometheus

url: http://prometheus:9090

save& test

导入模板

模板下载地址https:///dashboards/315， 或在线导入315 /1621

各配置文件

daemonset方式部署的node-exporter

  1. cat   node-exporter.yaml
  2. ---
  3. apiVersion: extensions/v1beta1
  4. kind: DaemonSet
  5. metadata:
  6.   name: node-exporter
  7.   namespace: kube-system
  8.   labels:
  9.     k8s-app: node-exporter
  10. spec:
  11.   template:
  12.     metadata:
  13.       labels:
  14.         k8s-app: node-exporter
  15.     spec:
  16.       containers:
  17.       - image: prom/node-exporter
  18.         name: node-exporter
  19.         ports:
  20.         - containerPort: 9100
  21.           protocol: TCP
  22.           name: http
  23. ---
  24. apiVersion: v1
  25. kind: Service
  26. metadata:
  27.   labels:
  28.     k8s-app: node-exporter
  29.   name: node-exporter
  30.   namespace: kube-system
  31. spec:
  32.   ports:
  33.   - name: http
  34.     port: 9100
  35.     nodePort: 31672
  36.     protocol: TCP
  37.   type: NodePort
  38.   selector:
  39.     k8s-app: node-exporter

prometheus的rbac-setup.yaml

  1.  cat  rbac-setup.yaml
  2. apiVersion: rbac.authorization.k8s.io/v1
  3. kind: ClusterRole
  4. metadata:
  5.   name: prometheus
  6. rules:
  7. - apiGroups: [""]
  8.   resources:
  9.   - nodes
  10.   - nodes/proxy
  11.   - services
  12.   - endpoints
  13.   - pods
  14.   verbs: ["get", "list", "watch"]
  15. - apiGroups:
  16.   - extensions
  17.   resources:
  18.   - ingresses
  19.   verbs: ["get", "list", "watch"]
  20. - nonResourceURLs: ["/metrics"]
  21.   verbs: ["get"]
  22. ---
  23. apiVersion: v1
  24. kind: ServiceAccount
  25. metadata:
  26.   name: prometheus
  27.   namespace: kube-system
  28. ---
  29. apiVersion: rbac.authorization.k8s.io/v1
  30. kind: ClusterRoleBinding
  31. metadata:
  32.   name: prometheus
  33. roleRef:
  34.   apiGroup: rbac.authorization.k8s.io
  35.   kind: ClusterRole
  36.   name: prometheus
  37. subjects:
  38. - kind: ServiceAccount
  39.   name: prometheus
  40.   namespace: kube-system

prometheus的configmap

  1.  cat  configmap.yaml
  2. apiVersion: v1
  3. kind: ConfigMap
  4. metadata:
  5.   name: prometheus-config
  6.   namespace: kube-system
  7. data:
  8.   prometheus.yml: |
  9.     global:
  10.       scrape_interval:     15s
  11.       evaluation_interval: 15s
  12.     scrape_configs:
  14.     - job_name: 'kubernetes-apiservers'
  15.       kubernetes_sd_configs:
  16.       - role: endpoints
  17.       scheme: https
  18.       tls_config:
  19.         ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  20.       bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  21.       relabel_configs:
  22.       - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
  23.         action: keep
  24.         regex: default;kubernetes;https
  26.     - job_name: 'kubernetes-nodes'
  27.       kubernetes_sd_configs:
  28.       - role: node
  29.       scheme: https
  30.       tls_config:
  31.         ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  32.       bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  33.       relabel_configs:
  34.       - action: labelmap
  35.         regex: __meta_kubernetes_node_label_(.+)
  36.       - target_label: __address__
  37.         replacement: kubernetes.default.svc:443
  38.       - source_labels: [__meta_kubernetes_node_name]
  39.         regex: (.+)
  40.         target_label: __metrics_path__
  41.         replacement: /api/v1/nodes/${1}/proxy/metrics
  43.     - job_name: 'kubernetes-cadvisor'
  44.       kubernetes_sd_configs:
  45.       - role: node
  46.       scheme: https
  47.       tls_config:
  48.         ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  49.       bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  50.       relabel_configs:
  51.       - action: labelmap
  52.         regex: __meta_kubernetes_node_label_(.+)
  53.       - target_label: __address__
  54.         replacement: kubernetes.default.svc:443
  55.       - source_labels: [__meta_kubernetes_node_name]
  56.         regex: (.+)
  57.         target_label: __metrics_path__
  58.         replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
  60.     - job_name: 'kubernetes-service-endpoints'
  61.       kubernetes_sd_configs:
  62.       - role: endpoints
  63.       relabel_configs:
  64.       - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
  65.         action: keep
  66.         regex: true
  67.       - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
  68.         action: replace
  69.         target_label: __scheme__
  70.         regex: (https?)
  71.       - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
  72.         action: replace
  73.         target_label: __metrics_path__
  74.         regex: (.+)
  75.       - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
  76.         action: replace
  77.         target_label: __address__
  78.         regex: ([^:]+)(?::\d+)?;(\d+)
  79.         replacement: $1:$2
  80.       - action: labelmap
  81.         regex: __meta_kubernetes_service_label_(.+)
  82.       - source_labels: [__meta_kubernetes_namespace]
  83.         action: replace
  84.         target_label: kubernetes_namespace
  85.       - source_labels: [__meta_kubernetes_service_name]
  86.         action: replace
  87.         target_label: kubernetes_name
  89.     - job_name: 'kubernetes-services'
  90.       kubernetes_sd_configs:
  91.       - role: service
  92.       metrics_path: /probe
  93.       params:
  94.         module: [http_2xx]
  95.       relabel_configs:
  96.       - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
  97.         action: keep
  98.         regex: true
  99.       - source_labels: [__address__]
  100.         target_label: __param_target
  101.       - target_label: __address__
  102.         replacement: blackbox-exporter.example.com:9115
  103.       - source_labels: [__param_target]
  104.         target_label: instance
  105.       - action: labelmap
  106.         regex: __meta_kubernetes_service_label_(.+)
  107.       - source_labels: [__meta_kubernetes_namespace]
  108.         target_label: kubernetes_namespace
  109.       - source_labels: [__meta_kubernetes_service_name]
  110.         target_label: kubernetes_name
  112.     - job_name: 'kubernetes-ingresses'
  113.       kubernetes_sd_configs:
  114.       - role: ingress
  115.       relabel_configs:
  116.       - source_labels: [__meta_kubernetes_ingress_annotation_prometheus_io_probe]
  117.         action: keep
  118.         regex: true
  119.       - source_labels: [__meta_kubernetes_ingress_scheme,__address__,__meta_kubernetes_ingress_path]
  120.         regex: (.+);(.+);(.+)
  121.         replacement: ${1}://${2}${3}
  122.         target_label: __param_target
  123.       - target_label: __address__
  124.         replacement: blackbox-exporter.example.com:9115
  125.       - source_labels: [__param_target]
  126.         target_label: instance
  127.       - action: labelmap
  128.         regex: __meta_kubernetes_ingress_label_(.+)
  129.       - source_labels: [__meta_kubernetes_namespace]
  130.         target_label: kubernetes_namespace
  131.       - source_labels: [__meta_kubernetes_ingress_name]
  132.         target_label: kubernetes_name
  134.     - job_name: 'kubernetes-pods'
  135.       kubernetes_sd_configs:
  136.       - role: pod
  137.       relabel_configs:
  138.       - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
  139.         action: keep
  140.         regex: true
  141.       - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
  142.         action: replace
  143.         target_label: __metrics_path__
  144.         regex: (.+)
  145.       - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
  146.         action: replace
  147.         regex: ([^:]+)(?::\d+)?;(\d+)
  148.         replacement: $1:$2
  149.         target_label: __address__
  150.       - action: labelmap
  151.         regex: __meta_kubernetes_pod_label_(.+)
  152.       - source_labels: [__meta_kubernetes_namespace]
  153.         action: replace
  154.         target_label: kubernetes_namespace
  155.       - source_labels: [__meta_kubernetes_pod_name]
  156.         action: replace
  157.         target_label: kubernetes_pod_name

配置说明：来源官网

https://prometheus.io/docs/prometheus/latest/configuration/configuration/

<kubernetes_sd_config>

Kubernetes SD配置允许从Kubernetes的 REST API 检索抓取目标， 并始终与集群状态保持同步。

role可以将以下类型之一配置为发现目标：

1，node

该node角色为每个群集节点发现一个目标，其地址默认为Kubelet的HTTP端口。目标地址默认为的地址类型顺序Kubernetes节点对象的第一个现有地址NodeInternalIP，NodeExternalIP， NodeLegacyHostIP，和NodeHostName。

可用的元标签:

__meta_kubernetes_node_name：节点对象的名称。

_meta_kubernetes_node_label：来自节点对象的每个标签。

_meta_kubernetes_node_labelpresent：true对于节点对象中的每个标签。

_meta_kubernetes_node_annotation：来自节点对象的每个注释。

_meta_kubernetes_node_annotationpresent：true用于节点对象的每个注释。

_meta_kubernetes_node_address<address_type>：每个节点地址类型的第一个地址（如果存在）。

此外，该instance节点的标签将设置为从API服务器检索到的节点名称

2,service

该service角色发现每一个服务端口为每个服务的目标。这通常用于监视服务的黑盒。该地址将设置为服务的Kubernetes DNS名称以及相应的服务端口

可用的元标签：

__meta_kubernetes_namespace：服务对象的名称空间。

_meta_kubernetes_service_annotation：来自服务对象的每个注释。

_meta_kubernetes_service_annotationpresent：对于服务对象的每个注释为“ true”。

__meta_kubernetes_service_cluster_ip：服务的群集IP地址。（不适用于外部名称类型的服务）

__meta_kubernetes_service_external_name：服务的DNS名称。（适用于外部名称类型的服务）

_meta_kubernetes_service_label：服务对象中的每个标签。

_meta_kubernetes_service_labelpresent：true用于服务对象的每个标签。

__meta_kubernetes_service_name：服务对象的名称。

__meta_kubernetes_service_port_name：目标服务端口的名称。

__meta_kubernetes_service_port_protocol：目标服务端口的协议。

3,pod

该pod角色发现所有pods并将其内容器公开为目标。对于容器的每个声明的端口，将生成一个目标。如果容器没有指定的端口，则会为每个容器创建无端口目标，以通过重新标记手动添加端口

可用的元标签：

__meta_kubernetes_namespace：pod对象的名称空间。

__meta_kubernetes_pod_name：pod对象的名称。

__meta_kubernetes_pod_ip：pod对象的pod IP。

_meta_kubernetes_pod_label：来自pod对象的每个标签。

_meta_kubernetes_pod_labelpresent：true用于pod对象中的每个标签。

_meta_kubernetes_pod_annotation：来自pod对象的每个注释。

_meta_kubernetes_pod_annotationpresent：true用于pod对象的每个注释。

__meta_kubernetes_pod_container_init：true如果容器是InitContainer

__meta_kubernetes_pod_container_name：目标地址指向的容器的名称。

__meta_kubernetes_pod_container_port_name：容器端口的名称。

__meta_kubernetes_pod_container_port_number：容器端口号。

__meta_kubernetes_pod_container_port_protocol：容器端口的协议。

__meta_kubernetes_pod_ready：设置为true或false吊舱的就绪状态。

__meta_kubernetes_pod_phase：设置为Pending，Running，Succeeded，Failed或Unknown 在生命周期。

__meta_kubernetes_pod_node_name：将Pod调度到的节点的名称。

__meta_kubernetes_pod_host_ip：pod对象的当前主机IP。

__meta_kubernetes_pod_uid：pod对象的UID。

__meta_kubernetes_pod_controller_kind：pod控制器的对象种类。

__meta_kubernetes_pod_controller_name：pod控制器的名称。

４，endpoints

该endpoints角色从服务的列表中的终端发现目标。对于每个端点地址，每个端口都发现一个目标。如果端点由Pod支持，则该Pod的所有其他未绑定到端点端口的容器端口也将被发现为目标

可用的元标签：

__meta_kubernetes_namespace：端点对象的名称空间。

__meta_kubernetes_endpoints_name：端点对象的名称。

对于直接从端点列表中发现的所有目标（未从基础容器额外推断出的所有目标），将附加以下标签：

__meta_kubernetes_endpoint_hostname：端点的主机名。

__meta_kubernetes_endpoint_node_name：托管端点的节点的名称。

__meta_kubernetes_endpoint_ready：设置为true或false为端点的就绪状态。

__meta_kubernetes_endpoint_port_name：端点端口的名称。

__meta_kubernetes_endpoint_port_protocol：端点端口的协议。

__meta_kubernetes_endpoint_address_target_kind：端点地址目标的种类。

__meta_kubernetes_endpoint_address_target_name：端点地址目标的名称。

如果端点属于服务，role: service则会附加发现的所有标签。

对于由Pod支持的所有目标，role: pod将附加发现的所有标签。

５，ingress

该ingress角色发现了一个目标，为每个进入的每个路径。这通常对黑盒监视入口很有用。该地址将设置为入口规范中指定的主机。

可用的元标签：

__meta_kubernetes_namespace：入口对象的名称空间。

__meta_kubernetes_ingress_name：入口对象的名称。

_meta_kubernetes_ingress_label：来自入口对象的每个标签。

_meta_kubernetes_ingress_labelpresent：true用于来自入口对象的每个标签。

_meta_kubernetes_ingress_annotation：来自入口对象的每个注释。

_meta_kubernetes_ingress_annotationpresent：true用于来自入口对象的每个注释。

__meta_kubernetes_ingress_scheme：https如果设置了TLS配置，则为入口的协议方案。默认为http。

__meta_kubernetes_ingress_path：来自入口规范的路径。默认为/

有关Kubernetes发现的配置选项：

  1. # The information to access the Kubernetes API.
  3. # The API server addresses. If left empty, Prometheus is assumed to run inside
  4. # of the cluster and will discover API servers automatically and use the pod's
  5. # CA certificate and bearer token file at /var/run/secrets/kubernetes.io/serviceaccount/.
  6. [ api_server: <host> ]
  8. # The Kubernetes role of entities that should be discovered.
  9. role: <role>
  11. # Optional authentication information used to authenticate to the API server.
  12. # Note that `basic_auth`, `bearer_token` and `bearer_token_file` options are
  13. # mutually exclusive.
  14. # password and password_file are mutually exclusive.
  16. # Optional HTTP basic authentication information.
  17. basic_auth:
  18.   [ username: <string> ]
  19.   [ password: <secret> ]
  20.   [ password_file: <string> ]
  22. # Optional bearer token authentication information.
  23. [ bearer_token: <secret> ]
  25. # Optional bearer token file authentication information.
  26. [ bearer_token_file: <filename> ]
  28. # Optional proxy URL.
  29. [ proxy_url: <string> ]
  31. # TLS configuration.
  32. tls_config:
  33.   [ <tls_config> ]
  35. # Optional namespace discovery. If omitted, all namespaces are used.
  36. namespaces:
  37.   names:
  38.     [ - <string> ]

prometheus-kubernetes.yml:

来源：https://github.com/prometheus/prometheus/blob/release-2.14/documentation/examples/prometheus-kubernetes.yml

  1. # A scrape configuration for running Prometheus on a Kubernetes cluster.
  2. # This uses separate scrape configs for cluster components (i.e. API server, node)
  3. # and services to allow each to use different authentication configs.
  4. #
  5. # Kubernetes labels will be added as Prometheus labels on metrics via the
  6. # `labelmap` relabeling action.
  7. #
  8. # If you are using Kubernetes 1.7.2 or earlier, please take note of the comments
  9. # for the kubernetes-cadvisor job; you will need to edit or remove this job.
  11. # Scrape config for API servers.
  12. #
  13. # Kubernetes exposes API servers as endpoints to the default/kubernetes
  14. # service so this uses `endpoints` role and uses relabelling to only keep
  15. # the endpoints associated with the default/kubernetes service using the
  16. # default named port `https`. This works for single API server deployments as
  17. # well as HA API server deployments.
  18. scrape_configs:
  19. - job_name: 'kubernetes-apiservers'
  21.   kubernetes_sd_configs:
  22.   - role: endpoints
  24.   # Default to scraping over https. If required, just disable this or change to
  25.   # `http`.
  26.   scheme: https
  28.   # This TLS & bearer token file config is used to connect to the actual scrape
  29.   # endpoints for cluster components. This is separate to discovery auth
  30.   # configuration because discovery & scraping are two separate concerns in
  31.   # Prometheus. The discovery auth config is automatic if Prometheus runs inside
  32.   # the cluster. Otherwise, more config options have to be provided within the
  33.   # <kubernetes_sd_config>.
  34.   tls_config:
  35.     ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  36.     # If your node certificates are self-signed or use a different CA to the
  37.     # master CA, then disable certificate verification below. Note that
  38.     # certificate verification is an integral part of a secure infrastructure
  39.     # so this should only be disabled in a controlled environment. You can
  40.     # disable certificate verification by uncommenting the line below.
  41.     #
  42.     # insecure_skip_verify: true
  43.   bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  45.   # Keep only the default/kubernetes service endpoints for the https port. This
  46.   # will add targets for each API server which Kubernetes adds an endpoint to
  47.   # the default/kubernetes service.
  48.   relabel_configs:
  49.   - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
  50.     action: keep
  51.     regex: default;kubernetes;https
  53. # Scrape config for nodes (kubelet).
  54. #
  55. # Rather than connecting directly to the node, the scrape is proxied though the
  56. # Kubernetes apiserver.  This means it will work if Prometheus is running out of
  57. # cluster, or can't connect to nodes for some other reason (e.g. because of
  58. # firewalling).
  59. - job_name: 'kubernetes-nodes'
  61.   # Default to scraping over https. If required, just disable this or change to
  62.   # `http`.
  63.   scheme: https
  65.   # This TLS & bearer token file config is used to connect to the actual scrape
  66.   # endpoints for cluster components. This is separate to discovery auth
  67.   # configuration because discovery & scraping are two separate concerns in
  68.   # Prometheus. The discovery auth config is automatic if Prometheus runs inside
  69.   # the cluster. Otherwise, more config options have to be provided within the
  70.   # <kubernetes_sd_config>.
  71.   tls_config:
  72.     ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  73.   bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  75.   kubernetes_sd_configs:
  76.   - role: node
  78.   relabel_configs:
  79.   - action: labelmap
  80.     regex: __meta_kubernetes_node_label_(.+)
  81.   - target_label: __address__
  82.     replacement: kubernetes.default.svc:443
  83.   - source_labels: [__meta_kubernetes_node_name]
  84.     regex: (.+)
  85.     target_label: __metrics_path__
  86.     replacement: /api/v1/nodes/${1}/proxy/metrics
  88. # Scrape config for Kubelet cAdvisor.
  89. #
  90. # This is required for Kubernetes 1.7.3 and later, where cAdvisor metrics
  91. # (those whose names begin with 'container_') have been removed from the
  92. # Kubelet metrics endpoint.  This job scrapes the cAdvisor endpoint to
  93. # retrieve those metrics.
  94. #
  95. # In Kubernetes 1.7.0-1.7.2, these metrics are only exposed on the cAdvisor
  96. # HTTP endpoint; use "replacement: /api/v1/nodes/${1}:4194/proxy/metrics"
  97. # in that case (and ensure cAdvisor's HTTP server hasn't been disabled with
  98. # the --cadvisor-port=0 Kubelet flag).
  99. #
  100. # This job is not necessary and should be removed in Kubernetes 1.6 and
  101. # earlier versions, or it will cause the metrics to be scraped twice.
  102. - job_name: 'kubernetes-cadvisor'
  104.   # Default to scraping over https. If required, just disable this or change to
  105.   # `http`.
  106.   scheme: https
  108.   # This TLS & bearer token file config is used to connect to the actual scrape
  109.   # endpoints for cluster components. This is separate to discovery auth
  110.   # configuration because discovery & scraping are two separate concerns in
  111.   # Prometheus. The discovery auth config is automatic if Prometheus runs inside
  112.   # the cluster. Otherwise, more config options have to be provided within the
  113.   # <kubernetes_sd_config>.
  114.   tls_config:
  115.     ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  116.   bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  118.   kubernetes_sd_configs:
  119.   - role: node
  121.   relabel_configs:
  122.   - action: labelmap
  123.     regex: __meta_kubernetes_node_label_(.+)
  124.   - target_label: __address__
  125.     replacement: kubernetes.default.svc:443
  126.   - source_labels: [__meta_kubernetes_node_name]
  127.     regex: (.+)
  128.     target_label: __metrics_path__
  129.     replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
  131. # Example scrape config for service endpoints.
  132. #
  133. # The relabeling allows the actual service scrape endpoint to be configured
  134. # for all or only some endpoints.
  135. - job_name: 'kubernetes-service-endpoints'
  137.   kubernetes_sd_configs:
  138.   - role: endpoints
  140.   relabel_configs:
  141.   # Example relabel to scrape only endpoints that have
  142.   # "example.io/should_be_scraped = true" annotation.
  143.   #  - source_labels: [__meta_kubernetes_service_annotation_example_io_should_be_scraped]
  144.   #    action: keep
  145.   #    regex: true
  146.   #
  147.   # Example relabel to customize metric path based on endpoints
  148.   # "example.io/metric_path = <metric path>" annotation.
  149.   #  - source_labels: [__meta_kubernetes_service_annotation_example_io_metric_path]
  150.   #    action: replace
  151.   #    target_label: __metrics_path__
  152.   #    regex: (.+)
  153.   #
  154.   # Example relabel to scrape only single, desired port for the service based
  155.   # on endpoints "example.io/scrape_port = <port>" annotation.
  156.   #  - source_labels: [__address__, __meta_kubernetes_service_annotation_example_io_scrape_port]
  157.   #    action: replace
  158.   #    regex: ([^:]+)(?::\d+)?;(\d+)
  159.   #    replacement: $1:$2
  160.   #    target_label: __address__
  161.   #
  162.   # Example relabel to configure scrape scheme for all service scrape targets
  163.   # based on endpoints "example.io/scrape_scheme = <scheme>" annotation.
  164.   #  - source_labels: [__meta_kubernetes_service_annotation_example_io_scrape_scheme]
  165.   #    action: replace
  166.   #    target_label: __scheme__
  167.   #    regex: (https?)
  168.   - action: labelmap
  169.     regex: __meta_kubernetes_service_label_(.+)
  170.   - source_labels: [__meta_kubernetes_namespace]
  171.     action: replace
  172.     target_label: kubernetes_namespace
  173.   - source_labels: [__meta_kubernetes_service_name]
  174.     action: replace
  175.     target_label: kubernetes_name
  177. # Example scrape config for probing services via the Blackbox Exporter.
  178. #
  179. # The relabeling allows the actual service scrape endpoint to be configured
  180. # for all or only some services.
  181. - job_name: 'kubernetes-services'
  183.   metrics_path: /probe
  184.   params:
  185.     module: [http_2xx]
  187.   kubernetes_sd_configs:
  188.   - role: service
  190.   relabel_configs:
  191.   # Example relabel to probe only some services that have "example.io/should_be_probed = true" annotation
  192.   #  - source_labels: [__meta_kubernetes_service_annotation_example_io_should_be_probed]
  193.   #    action: keep
  194.   #    regex: true
  195.   - source_labels: [__address__]
  196.     target_label: __param_target
  197.   - target_label: __address__
  198.     replacement: blackbox-exporter.example.com:9115
  199.   - source_labels: [__param_target]
  200.     target_label: instance
  201.   - action: labelmap
  202.     regex: __meta_kubernetes_service_label_(.+)
  203.   - source_labels: [__meta_kubernetes_namespace]
  204.     target_label: kubernetes_namespace
  205.   - source_labels: [__meta_kubernetes_service_name]
  206.     target_label: kubernetes_name
  208. # Example scrape config for probing ingresses via the Blackbox Exporter.
  209. #
  210. # The relabeling allows the actual ingress scrape endpoint to be configured
  211. # for all or only some services.
  212. - job_name: 'kubernetes-ingresses'
  214.   metrics_path: /probe
  215.   params:
  216.     module: [http_2xx]
  218.   kubernetes_sd_configs:
  219.   - role: ingress
  221.   relabel_configs:
  222.   # Example relabel to probe only some ingresses that have "example.io/should_be_probed = true" annotation
  223.   #  - source_labels: [__meta_kubernetes_ingress_annotation_example_io_should_be_probed]
  224.   #    action: keep
  225.   #    regex: true
  226.   - source_labels: [__meta_kubernetes_ingress_scheme,__address__,__meta_kubernetes_ingress_path]
  227.     regex: (.+);(.+);(.+)
  228.     replacement: ${1}://${2}${3}
  229.     target_label: __param_target
  230.   - target_label: __address__
  231.     replacement: blackbox-exporter.example.com:9115
  232.   - source_labels: [__param_target]
  233.     target_label: instance
  234.   - action: labelmap
  235.     regex: __meta_kubernetes_ingress_label_(.+)
  236.   - source_labels: [__meta_kubernetes_namespace]
  237.     target_label: kubernetes_namespace
  238.   - source_labels: [__meta_kubernetes_ingress_name]
  239.     target_label: kubernetes_name
  241. # Example scrape config for pods
  242. #
  243. # The relabeling allows the actual pod scrape to be configured
  244. # for all the declared ports (or port-free target if none is declared)
  245. # or only some ports.
  246. - job_name: 'kubernetes-pods'
  248.   kubernetes_sd_configs:
  249.   - role: pod
  251.   relabel_configs:
  252.   # Example relabel to scrape only pods that have
  253.   # "example.io/should_be_scraped = true" annotation.
  254.   #  - source_labels: [__meta_kubernetes_pod_annotation_example_io_should_be_scraped]
  255.   #    action: keep
  256.   #    regex: true
  257.   #
  258.   # Example relabel to customize metric path based on pod
  259.   # "example.io/metric_path = <metric path>" annotation.
  260.   #  - source_labels: [__meta_kubernetes_pod_annotation_example_io_metric_path]
  261.   #    action: replace
  262.   #    target_label: __metrics_path__
  263.   #    regex: (.+)
  264.   #
  265.   # Example relabel to scrape only single, desired port for the pod
  266.   # based on pod "example.io/scrape_port = <port>" annotation.
  267.   # Note that __address__ is modified here, so if pod containers' ports
  268.   # are declared, they all will be ignored.
  269.   #  - source_labels: [__address__, __meta_kubernetes_pod_annotation_example_io_scrape_port]
  270.   #    action: replace
  271.   #    regex: ([^:]+)(?::\d+)?;(\d+)
  272.   #    replacement: $1:$2
  273.   #    target_label: __address__
  274.   - action: labelmap
  275.     regex: __meta_kubernetes_pod_label_(.+)
  276.   - source_labels: [__meta_kubernetes_namespace]
  277.     action: replace
  278.     target_label: kubernetes_namespace
  279.   - source_labels: [__meta_kubernetes_pod_name]
  280.     action: replace
  281.     target_label: kubernetes_pod_name

Prometheus deployment

  1. cat  prometheus.deploy.yml
  2. ---
  3. apiVersion: apps/v1beta2
  4. kind: Deployment
  5. metadata:
  6.   labels:
  7.     name: prometheus-deployment
  8.   name: prometheus
  9.   namespace: kube-system
  10. spec:
  11.   replicas: 1
  12.   selector:
  13.     matchLabels:
  14.       app: prometheus
  15.   template:
  16.     metadata:
  17.       labels:
  18.         app: prometheus
  19.     spec:
  20.       containers:
  21.       - image: prom/prometheus:v2.0.0
  22.         name: prometheus
  23.         command:
  24.         - "/bin/prometheus"
  25.         args:
  26.         - "--config.file=/etc/prometheus/prometheus.yml"
  27.         - "--storage.tsdb.path=/prometheus"
  28.         - "--storage.tsdb.retention=24h"
  29.         ports:
  30.         - containerPort: 9090
  31.           protocol: TCP
  32.         volumeMounts:
  33.         - mountPath: "/prometheus"
  34.           name: data
  35.         - mountPath: "/etc/prometheus"
  36.           name: config-volume
  37.         resources:
  38.           requests:
  39.             cpu: 100m
  40.             memory: 100Mi
  41.           limits:
  42.             cpu: 500m
  43.             memory: 2500Mi
  44.       serviceAccountName: prometheus
  45.       volumes:
  46.       - name: data
  47.         emptyDir: {}
  48.       - name: config-volume
  49.         configMap:
  50.           name: prometheus-config

prometheus的service

  1. cat  prometheus.svc.yml
  2. ---
  3. kind: Service
  4. apiVersion: v1
  5. metadata:
  6.   labels:
  7.     app: prometheus
  8.   name: prometheus
  9.   namespace: kube-system
  10. spec:
  11.   type: NodePort
  12.   ports:
  13.   - port: 9090
  14.     targetPort: 9090
  15.     nodePort: 30003
  16.   selector:
  17.     app: prometheus

grafana-deployment配置文件

  1. cat grafana-deploy.yaml
  2. apiVersion: extensions/v1beta1
  3. kind: Deployment
  4. metadata:
  5.   name: grafana-core
  6.   namespace: kube-system
  7.   labels:
  8.     app: grafana
  9.     component: core
  10. spec:
  11.   replicas: 1
  12.   template:
  13.     metadata:
  14.       labels:
  15.         app: grafana
  16.         component: core
  17.     spec:
  18.       containers:
  19.       - image: grafana/grafana:4.2.0
  20.         name: grafana-core
  21.         imagePullPolicy: IfNotPresent
  22.         # env:
  23.         resources:
  24.           # keep request = limit to keep this container in guaranteed class
  25.           limits:
  26.             cpu: 100m
  27.             memory: 100Mi
  28.           requests:
  29.             cpu: 100m
  30.             memory: 100Mi
  31.         env:
  32.           # The following env variables set up basic auth twith the default admin user and admin password.
  33.           - name: GF_AUTH_BASIC_ENABLED
  34.             value: "true"
  35.           - name: GF_AUTH_ANONYMOUS_ENABLED
  36.             value: "false"
  37.           # - name: GF_AUTH_ANONYMOUS_ORG_ROLE
  38.           #   value: Admin
  39.           # does not really work, because of template variables in exported dashboards:
  40.           # - name: GF_DASHBOARDS_JSON_ENABLED
  41.           #   value: "true"
  42.         readinessProbe:
  43.           httpGet:
  44.             path: /login
  45.             port: 3000
  46.           # initialDelaySeconds: 30
  47.           # timeoutSeconds: 1
  48.         volumeMounts:
  49.         - name: grafana-persistent-storage
  50.           mountPath: /var
  51.       volumes:
  52.       - name: grafana-persistent-storage
  53.         emptyDir: {}

grafana service

  1. cat  grafana-svc.yaml
  2. apiVersion: v1
  3. kind: Service
  4. metadata:
  5.   name: grafana
  6.   namespace: kube-system
  7.   labels:
  8.     app: grafana
  9.     component: core
  10. spec:
  11.   type: NodePort
  12.   ports:
  13.     - port: 3000
  14.   selector:
  15.     app: grafana
  16.     component: core

grafana ingress

  1.  cat   grafana-ing.yaml
  2. apiVersion: extensions/v1beta1
  3. kind: Ingress
  4. metadata:
  5.    name: grafana
  6.    namespace: kube-system
  7. spec:
  8.    rules:
  9.    - host: k8s.grafana
  10.      http:
  11.        paths:
  12.        - path: /
  13.          backend:
  14.           serviceName: grafana
  15.           servicePort: 3000

k8b部署prometheus+grafana的更多相关文章

  1. Rancher2.x 一键式部署 Prometheus + Grafana 监控 Kubernetes 集群

    目录 1.Prometheus & Grafana 介绍 2.环境.软件准备 3.Rancher 2.x 应用商店 4.一键式部署 Prometheus 5.验证 Prometheus + G ...

  2. 部署Prometheus+Grafana监控

    Prometheus 1.不是很友好,各种配置都手写 2.对docker和k8s监控有成熟解决方案 Prometheus(普罗米修斯) 是一个最初在SoudCloud上构建的监控系统,开源项目,拥有非 ...

  3. k8s实战之部署Prometheus+Grafana可视化监控告警平台

    写在前面 之前部署web网站的时候,架构图中有一环节是监控部分,并且搭建一套有效的监控平台对于运维来说非常之重要,只有这样才能更有效率的保证我们的服务器和服务的稳定运行,常见的开源监控软件有好几种,如 ...

  4. kubenetes部署prometheus+grafana

    文章目录 环境介绍 创建node-exporter 创建Prometheus 创建Grafana 测试 环境介绍 # 关于k8s的集群部署,可以查看我其他博客 [root@master ~]# cat ...

  5. 群晖-使用docker套件部署Prometheus+Grafana

    Docker 部署 Prometheus 说明: 先在群辉管理界面安装好docker套件,修改一下镜像源(更快一点) 所需容器如下 Prometheus Server(普罗米修斯监控主服务器 ) No ...

  6. K8s 部署 Prometheus + Grafana

    一.简介 1. Prometheus 一款开源的监控&报警&时间序列数据库的组合,起始是由 SoundCloud 公司开发的 基本原理是通过 HTTP 协议周期性抓取被监控组件的状态, ...

  7. Kubernetes部署Prometheus+Grafana(非存储持久化方式部署)

    1.在master节点处新建一个文件夹,用于保存下载prometheus+granfana的yaml文件 mkdir /root/prometheus cd /root/prometheus git ...

  8. 【Linux】【Services】【SaaS】Docker+kubernetes（12. 部署prometheus/grafana/Influxdb实现监控）

    1.简介 1.1. 官方网站: promethos:https://prometheus.io/ grafana:https://grafana.com/ 1.2. 架构图 2. 环境 2.1. 机器 ...

  9. CentOS7安装部署Prometheus+Grafana （转）

    转自:https://www.jianshu.com/p/967cb76cd5ca 作为监控系统的后起之秀,prometheus的安装可谓非常简单,不需要第三方的依赖(数据库.缓存.PHP之类的).下 ...

随机推荐

  1. Django2.2 Mysql数据表一对一映射（字段绑定） --Model拓展学习笔记shen

    我:本次继上次的Model继续延伸一下数据表之间的一对一映射关系 访客:有什么用? 我:众所周知,一张数据表的字段太多会显得冗余.杂乱,那么就需要将一张表进行拆分成多个表,这样不会太杂乱,也比较好维护 ...

  2. Elasticsearch-如何控制存储和索引文档(_source、_all、返回源文档的某些字段)

    Elasticsearch-如何控制存储和索引文档(_source._all) _source:可以在索引中存储文档._all:可以在单个字段上索引所有内容. 1. 存储原有内容的_source _s ...

  3. JS小知识--获取当前日期的时间和上周五时间

    获取当前日期的时间和上周五时间 var today=new Date();//获取当前时间var weekday=today.getDay();//获取星期几    var monday=new Da ...

  4. Python内置函数Object

    英文文档 class object Return a new featureless object. object is a base for all classes. It has the meth ...

  5. C++多线程基础学习笔记（八）

    shared_futrue和futrue_status的用法 shared_futrue是一个类模板,类似于futrue,不同的是它的成员函数get()可以使用多次,因为是复制数据,而futrue的g ...

  6. 对C++类的继承和派生的理解

    C++中的继承是类与类之间的关系,是一个很简单很直观的概念,与现实世界中的继承类似,例如儿子继承父亲的财产. 1.继承(Inheritance)可以理解为一个类从另一个类获取成员变量和成员函数的过程. ...

  7. python-day42(正式学习)

    目录 数据库 卸载 安装 连接数据库 用户信息查看 数据库的基本操作 表的基本操作 记录的基本操作 复习 今日内容 数据库配置 数据库修改信息 用户操作:重点 表的修改 创建表的完整语法 数据库表的引 ...

  8. 分布式---Raft算法

    6.Raft   Raft也是分布式一致性协议,主要是用来竞选主节点. 单个Candidate的竞选   有三种节点:Follower,Candidate和Leader.Leader会周期性的发送心跳 ...

  9. C#中的编译为什么不叫Compile而叫build

    是因为Build包含了compile,build既compile了你自己的代码,又把你compile的Assembly和引用别人的Assembly放在一起buiid.

  10. pycharm terminal打开在虚拟环境

    pycharm调试是虚拟环境,terminal不是虚拟环境,搞了好久,原来需要激活一下 cd   venv\Scripts> 去虚拟环境的Script目录下,运行activate.bat激活环境 ...