思路一

统一区域的监控目标,prometheus server两台监控相同的目标群体。

改变后

上面这个变化对于监控目标端,会多出一倍的查询请求,但在一台prometheus server宕机的情况下,可以不影响监控。

思路二

这是一个金字塔式的层次结构,而不是分布式层次结构。Prometheus 的抓取请求也会加载到prometheus work节点上,这是需要考虑的。

上面这种模式,准备3台prometheus server进行搭建,这种方式work节点一台宕机后,其它wokr节点不会去接手故障work节点的机器。

1、环境准备

192.168.31.151(primary)

192.168.31.144 (worker)

192.168.31.82(worker)

2、部署prometheus

cd /usr/loacl
tar -xvf prometheus-2.8.0.linux-amd64.tar.gz
ln -s /usr/local/prometheus-2.8.0.linux-amd64 /usr/local/prometheus
cd /usr/local/prometheus;mkdir bin conf data
mv ./promtool bin
mv ./prometheus bin
mv ./prometheus.yml conf

3、worker节点配置(192.168.31.144)

prometheus.yml

# my global config
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
external_labels:
worker: 0 # Alertmanager configuration
alerting:
alertmanagers:
- static_configs:
- targets:
# - alertmanager:9093 # Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
- "rules/*_rules.yml" # A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: 'prometheus'
static_configs:
- targets:
- 192.168.31.151:9090
- 192.168.31.144:9090
- 192.168.31.82:9090
relabel_configs:
- source_labels: [__address__]
modulus: 2
target_label: __tmp_hash
action: hashmod
- source_labels: [__tmp_hash]
regex: ^0$
action: keep
- job_name: 'node_exporter'
file_sd_configs:
- files:
- targets/nodes/*.json
refresh_interval: 1m
relabel_configs:
- source_labels: [__address__]
modulus: 2
target_label: __tmp_hash
action: hashmod
- source_labels: [__tmp_hash]
regex: ^0$
action: keep
- job_name: 'docker'
file_sd_configs:
- files:
- targets/docker/*.json
refresh_interval: 1m
relabel_configs:
- source_labels: [__address__]
modulus: 2
target_label: __tmp_hash
action: hashmod
- source_labels: [__tmp_hash]
regex: ^0$
action: keep
- job_name: 'alertmanager'
static_configs:
- targets:
- 192.168.31.151:9093
- 192.168.31.144:9093
relabel_configs:
- source_labels: [__address__]
modulus: 2
target_label: __tmp_hash
action: hashmod
- source_labels: [__tmp_hash]
regex: ^0$
action: keep

worker节点配置(192.168.31.82)

prometheus.yml

# my global config
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
external_labels:
worker: 1 # Alertmanager configuration
alerting:
alertmanagers:
- static_configs:
- targets:
# - alertmanager:9093 # Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
- "rules/*_rules.yml" # A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: 'prometheus'
static_configs:
- targets:
- 192.168.31.151:9090
- 192.168.31.144:9090
- 192.168.31.82:9090
relabel_configs:
- source_labels: [__address__]
modulus: 2
target_label: __tmp_hash
action: hashmod
- source_labels: [__tmp_hash]
regex: ^1$
action: keep
- job_name: 'node_exporter'
file_sd_configs:
- files:
- targets/nodes/*.json
refresh_interval: 1m
relabel_configs:
- source_labels: [__address__]
modulus: 2
target_label: __tmp_hash
action: hashmod
- source_labels: [__tmp_hash]
regex: ^1$
action: keep
- job_name: 'docker'
file_sd_configs:
- files:
- targets/docker/*.json
refresh_interval: 1m
relabel_configs:
- source_labels: [__address__]
modulus: 2
target_label: __tmp_hash
action: hashmod
- source_labels: [__tmp_hash]
regex: ^1$
action: keep
- job_name: 'alertmanager'
static_configs:
- targets:
- 192.168.31.151:9093
- 192.168.31.144:9093
relabel_configs:
- source_labels: [__address__]
modulus: 2
target_label: __tmp_hash
action: hashmod
- source_labels: [__tmp_hash]
regex: ^1$
action: keep

primary节点配置(192.168.31.151)

prometheus.yml

# my global config
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
# scrape_timeout is set to the global default (10s). # Alertmanager configuration
alerting:
alertmanagers:
- static_configs:
- targets:
- 192.168.31.151:9093
- 192.168.31.144:9093 # Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
- "rules/*_alerts.yml" # A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
- job_name: 'node_workers'
file_sd_configs:
- files:
- 'targets/workers/*.json'
refresh_interval: 5m
honor_labels: true
metrics_path: /federate
params:
'match[]':
- '{__name__=~"^instance:.*"}'

cat ./targets/workers/workers.json

[{
"targets": [
"192.168.31.144:9090",
"192.168.31.82:9090"
]
}]

prometheus 集群的更多相关文章

  1. Prometheus集群介绍-1

    Prometheus监控介绍 公司做教育的,要迁移上云,所以需要我这边从零开始调研加后期维护Prometheus:近期看过二本方面的prometheus书籍,一本是深入浅出一般是实战方向的:官方文档主 ...

  2. Thanos prometheus 集群以及多租户解决方案docker-compose 试用(一)

    prometheus 是一个非常不多的metrics 监控解决方案,但是对于ha 以及多租户的处理并不是很好,当前有好多解决方案 cortex Thanos prometheus+ influxdb ...

  3. 部署prometheus监控kubernetes集群并存储到ceph

    简介 Prometheus 最初是 SoundCloud 构建的开源系统监控和报警工具,是一个独立的开源项目,于2016年加入了 CNCF 基金会,作为继 Kubernetes 之后的第二个托管项目. ...

  4. 如何扩展单个Prometheus实现近万Kubernetes集群监控?

    引言 TKE团队负责公有云,私有云场景下近万个集群,数百万核节点的运维管理工作.为了监控规模如此庞大的集群联邦,TKE团队在原生Prometheus的基础上进行了大量探索与改进,研发出一套可扩展,高可 ...

  5. 如何用Prometheus监控十万container的Kubernetes集群

    概述 不久前,我们在文章<如何扩展单个Prometheus实现近万Kubernetes集群监控?>中详细介绍了TKE团队大规模Kubernetes联邦监控系统Kvass的演进过程,其中介绍 ...

  6. vivo 容器集群监控系统架构与实践

    vivo 互联网服务器团队-YuanPeng 一.概述 从容器技术的推广以及 Kubernetes成为容器调度管理领域的事实标准开始,云原生的理念和技术架构体系逐渐在生产环境中得到了越来越广泛的应用实 ...

  7. 如何使用helm优雅安装prometheus-operator,并监控k8s集群微服务

    前言:随着云原生概念盛行,对于容器.服务.节点以及集群的监控变得越来越重要.Prometheus 作为 Kubernetes 监控的事实标准,有着强大的功能和良好的生态.但是它不支持分布式,不支持数据 ...

  8. Kubernetes集群部署史上最详细(二)Prometheus监控Kubernetes集群

    使用Prometheus监控Kubernetes集群 监控方面Grafana采用YUM安装通过服务形式运行,部署在Master上,而Prometheus则通过POD运行,Grafana通过使用Prom ...

  9. Prometheus监控elasticsearch集群(以elasticsearch-6.4.2版本为例)

    部署elasticsearch集群,配置文件可"浓缩"为以下: cluster.name: es_cluster node.name: node1 path.data: /app/ ...

随机推荐

  1. Intel MKL FATAL ERROR: Cannot load mkl_intel_thread.dll

    Intel MKL FATAL ERROR: Cannot load mkl_intel_thread.dll 在使用Anaconda创建一个虚拟环境出来,然后安装了scikit-learn.nump ...

  2. java----静态代理

    静态代理没啥好说的,直接上代码 package com.yk.aop.staticproxy; import org.junit.jupiter.api.Test; //1.接口 public int ...

  3. Android深入理解Context(一)Context关联类和Application Context创建过程

    前言 Context也就是上下文对象,是Android较为常用的类,但是对于Context,很多人都停留在会用的阶段,这个系列会带大家从源码角度来分析Context,从而更加深入的理解它. 1.Con ...

  4. 如何修改Recovery的字符串资源

    前言:在实际的ROM修改中,Recovery的修改还是会经常遇到的,这篇文章主要讲解如何修改Recovery字符串.   首先我们先了解下大概的流程. 1.screen_ui.cpp 中的Screen ...

  5. <3>Centos系统完整安装python流程

    一.环境 系统:Centos7 Python:3.6.5  自带pip.setuptools 二.命令 介绍:因为yum是依赖于python2,所以千万别删除自带的python2,下面的方法就是py2 ...

  6. QT中使用google breakpad捕获程序崩溃异常

    今天给大家介绍一个在linux下如何捕获程序崩溃异常的方法 一.google breakpad源码的下载和编译 1.https://github.com/google/breakpad.git,源码地 ...

  7. rocketmq有序消息

    RocketMQ提供的顺序消费消息实现是使用的FIFO 先进先出算法 Producer消息发送 public class Producer { public static void main(Stri ...

  8. memcached 学习

    memcached 是什么 特点 协议简单 基于 libevent 的事件处理 内置内存存储方式 memcached 不互相通信的分布式 启动 安装 依赖 libevent 安装命令 下载地址在这个网 ...

  9. Scrapy框架-Spider

    目录 1. Spider 2.Scrapy源代码 2.1. Scrapy主要属性和方法 3.parse()方法的工作机制 1. Spider Spider类定义了如何爬取某个(或某些)网站.包括了爬取 ...

  10. excel身份证验证(附带防止粘贴导致校验失效的函数)

    一.检验函数 =IF(LEN(A1)=18,MID("10X98765432",MOD(SUMPRODUCT(MID(A1,ROW(INDIRECT("1:17" ...