prometheus如何使用blackbox-exporter来获取k8s的网络性能

如果学会了如何抓取app的metrics，则blackbox-exporter抓取网络性能套路是相同的。

一，在blackbox-exporter的blackbox.yml里配置好抓取模块，常见的是http_get,http_post,tcp,icmp。

二，在prometheus的平台侧，配置好抓取http-probe或tcp-probe的job规则。

三，应用侧的service里加上相关annotation注释，配合prometheus的metrics抓取。

下面根据网上文档，记录相关的yaml文件。

一，blackbox-exporter的blackbox.yam的configmap配置。

apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app: prometheus-blackbox-exporter
  name: prometheus-blackbox-exporter
  namespace: kube-system
data:
  blackbox.yml: |-
    modules:
      http_2xx:
        prober: http
        timeout: 10s
        http:
          valid_http_versions: ["HTTP/1.1", "HTTP/2"]　　　　　　# 这里最好作一个返回状态码，在grafana作图时，有明示---陈刚注释。
          valid_status_codes: [200]
          method: GET
          preferred_ip_protocol: "ip4"
      http_post_2xx: # http post 监测模块
        prober: http
        timeout: 10s
        http:
          valid_http_versions: ["HTTP/1.1", "HTTP/2"]
          method: POST
          preferred_ip_protocol: "ip4"
      tcp_connect:
        prober: tcp
        timeout: 10s
      icmp:
        prober: icmp
        timeout: 10s
        icmp:
          preferred_ip_protocol: "ip4"

备注：

blackbox-exporter的配置文件为/etc/blackbox_exporter/blackbox.yml，可以运行时动态的重新加载配置文件，当重新加载配置文件失败时，不影响在运行的配置。重载方式：curl -XPOST http://IP:9115/-/reload
二，blackbox-exporter的yaml文件。

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: prometheus-blackbox-exporter
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: prometheus-blackbox-exporter
  replicas: 1
  template:
    metadata:
      labels:
        app: prometheus-blackbox-exporter
    spec:
      restartPolicy: Always
      containers:
      - name: prometheus-blackbox-exporter
        image: prom/blackbox-exporter:v0.12.0
        imagePullPolicy: IfNotPresent
        ports:
        - name: blackbox-port
          containerPort: 9115
        readinessProbe:
          tcpSocket:
            port: 9115
          initialDelaySeconds: 5
          timeoutSeconds: 5
        resources:
          requests:
            memory: 50Mi
            cpu: 100m
          limits:
            memory: 60Mi
            cpu: 200m
        volumeMounts:
        - name: config
          mountPath: /etc/blackbox_exporter
        args:
        - --config.file=/etc/blackbox_exporter/blackbox.yml
        - --log.level=debug
        - --web.listen-address=:9115
      volumes:
      - name: config
        configMap:
          name: prometheus-blackbox-exporter
      nodeSelector:
        node-role.kubernetes.io/master: "true"
      tolerations:
      - key: "node-role.kubernetes.io/master"
        effect: "NoSchedule"
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: prometheus-blackbox-exporter
  name: prometheus-blackbox-exporter
  namespace: kube-system
  annotations:
    prometheus.io/scrape: 'true'
spec:
  type: NodePort
  selector:
    app: prometheus-blackbox-exporter
  ports:
  - name: blackbox
    port: 9115
    targetPort: 9115
    nodePort: 30009
    protocol: TCP

三，prometheus的job配置。

- job_name: 'kubernetes-service-http-probe'
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        kubernetes_sd_configs:
        - role: service
        # 将metrics_path由默认的/metrics改为/probe
        metrics_path: /probe
        # Optional HTTP URL parameters.
        # 生成__param_module="http_2xx"的label
        params:
          module: [http_2xx]
        relabel_configs:
        # 只保留含有label为prometheus/io=scrape的service
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape, __meta_kubernetes_service_annotation_prometheus_io_http_probe]
          regex: true;true
          action: keep
        - source_labels: [__meta_kubernetes_service_name, __meta_kubernetes_namespace, __meta_kubernetes_service_annotation_prometheus_io_http_probe_port, __meta_kubernetes_service_annotation_prometheus_io_http_probe_path]
          action: replace
          target_label: __param_target
          regex: (.+);(.+);(.+);(.+)
          replacement: $1.$2:$3$4
        # 用__address__这个label的值创建一个名为__param_target的label为blackbox-exporter,值为内部service的访问地址，作为blackbox-exporter采集用
        #- source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_http_probe_path]
        #  action: replace
        #  target_label: __param_target
        #  regex: (.+);(.+)
        #  replacement: $1$2
        # 用blackbox-exporter的service地址值”prometheus-blackbox-exporter:9115"替换原__address__的值
        - target_label: __address__
          replacement: prometheus-blackbox-exporter:9115
        - source_labels: [__param_target]
          target_label: instance
        # 去掉label name中的前缀__meta_kubernetes_service_annotation_prometheus_io_app_info_
        - action: labelmap
          regex: __meta_kubernetes_service_annotation_prometheus_io_app_info_(.+)
        #- source_labels: [__meta_kubernetes_namespace]
        #  target_label: kubernetes_namespace
        #- source_labels: [__meta_kubernetes_service_name]
        #  target_label: kubernetes_name
      ## kubernetes-services and kubernetes-ingresses are blackbox_exporter related

      # Example scrape config for probing services via the Blackbox Exporter.
      #
      # The relabeling allows the actual service scrape endpoint to be configured
      # for all or only some services.
      - job_name: 'kubernetes-service-tcp-probe'
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        kubernetes_sd_configs:
        - role: service
        # 将metrics_path由默认的/metrics改为/probe
        metrics_path: /probe
        # Optional HTTP URL parameters.
        # 生成__param_module="tcp_connect"的label
        params:
          module: [tcp_connect]
        relabel_configs:
        # 只保留含有label为prometheus/io=scrape的service
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape, __meta_kubernetes_service_annotation_prometheus_io_tcp_probe]
          regex: true;true
          action: keep
        - source_labels: [__meta_kubernetes_service_name, __meta_kubernetes_namespace, __meta_kubernetes_service_annotation_prometheus_io_tcp_probe_port]
          action: replace
          target_label: __param_target
          regex: (.+);(.+);(.+)
          replacement: $1.$2:$3
        # 用__address__这个label的值创建一个名为__param_target的label为blackbox-exporter,值为内部service的访问地址，作为blackbox-exporter采集用
        #- source_labels: [__address__]
        #  target_label: __param_target
        # 用blackbox-exporter的service地址值”prometheus-blackbox-exporter:9115"替换原__address__的值
        - target_label: __address__
          replacement: prometheus-blackbox-exporter:9115
        - source_labels: [__param_target]
          target_label: instance
        # 去掉label name中的前缀__meta_kubernetes_service_annotation_prometheus_io_app_info_
        - action: labelmap
          regex: __meta_kubernetes_service_annotation_prometheus_io_app_info_(.+)

四，应用侧的service的annotation段配合的配置。

# http_probe
prometheus.io/scrape: 'true'
prometheus.io/http-probe: 'true'
prometheus.io/http-probe-port: '8080'
prometheus.io/http-probe-path: '/healthz'
# tcp_probe
prometheus.io/scrape: 'true'
prometheus.io/tcp-probe: 'true'
prometheus.io/tcp-probe-port: '80'

有了这些操作，以后要增加http或tcp的探测时，直接操作应用service即可接入。