Prometheus + Consul 自动发现服务监控

一、Prometheus支持的多种服务发现机制（常用如下）

1. static_configs: 静态服务发现
2. file_sd_configs: 文件服务发现
3. dns_sd_configs: DNS 服务发现
4. kubernetes_sd_configs: Kubernetes 服务发现
5. consul_sd_configs: Consul 服务发现

二、Prometheus主要配置prometheus.yml中的scrape_configs以及consul_sd_configs如下

scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'
    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.
    static_configs:
    - targets: ['localhost:9090']

  - job_name: 'consul'
    consul_sd_configs:
      - server: 'localhost:8500'
    relabel_configs:
      - source_labels: [__meta_consul_tags]
        regex: .*,prome,.*
        action: keep
      - source_labels: [__meta_consul_service]
        target_label: job

通过重新加载prometheus.yml配置添加

./prometheus --config.file="prometheus.yml" --web.listen-address="0.0.0.0:9090" --web.enable-admin-api --web.enable-lifecycle &

--web.enable-admin-api 运行通过web方式管理prometheus（删除清空数据等操作）
--web.enable-lifecycle 运行通过web方式重新加载prometheus配置（相当于reload）

三、k8s环境实现

将配置制作成secret挂在到prometheus中

1、编写prometheus-additional-consul.yaml

scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'
    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.
    static_configs:
    - targets: ['localhost:9090']

  - job_name: 'consul'
    consul_sd_configs:
      - server: 'localhost:8500'
    relabel_configs:
      - source_labels: [__meta_consul_tags]
        regex: .*,prome,.*
        action: keep
      - source_labels: [__meta_consul_service]
        target_label: job

2、制作secret

kubectl create secret generic additional-consul-configs --from-file=prometheus-additional-consul.yaml -n monitoring

3、添加配置到prometheus的pod中，修改prometheus-prometheus.yaml，并更新资源

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  labels:
    prometheus: k8s
  name: k8s
  namespace: monitoring
spec:
  alerting:
    alertmanagers:
    - name: alertmanager-main
      namespace: monitoring
      port: web
  baseImage: quay.io/prometheus/prometheus
  nodeSelector:
    kubernetes.io/os: linux
  podMonitorNamespaceSelector: {}
  podMonitorSelector: {}
  replicas:
  resources:
    requests:
      memory: 400Mi
  ruleSelector:
    matchLabels:
      prometheus: k8s
      role: alert-rules
  securityContext:
    fsGroup:
    runAsNonRoot: true
    runAsUser:
  serviceAccountName: prometheus-k8s
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector: {}
  version: v2.11.0
  additionalScrapeConfigs:
    name: additional-consul-configs
    key: prometheus-additional-consul.yaml

4、注意事项：权限不够（配置更新，但是没有对应的监控任务生成）

修改名为prometheus-k8s的ClusterRole权限，并更新资源

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus-k8s
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  - services
  - endpoints
  - pods
  - nodes/proxy
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - configmaps
  - nodes/metrics
  verbs:
  - get
- nonResourceURLs:
  - /metrics
  verbs:
  - get