prometheus使用4
安装pushgateway
下载地址： https://github.com/prometheus/pushgateway/releases
下载这个
[root@mcw04 ~]# tar xf pushgateway-1.7.0.linux-amd64.tar.gz
[root@mcw04 ~]# ls
\ apache-tomcat-8.5.88 hadoop-2.8.5.tar.gz nohup.out usr
1.py apache-tomcat-8.5.88.tar.gz ip_forward~ original-ks.cfg zabbix-release-4.0-1.el7.noarch.rpm
a filebeat-6.5.2-x86_64.rpm jdk-8u191-linux-x64.tar.gz pushgateway-1.7.0.linux-amd64
alertmanager.yml grafana-9.2.3 mcw.txt pushgateway-1.7.0.linux-amd64.tar.gz
anaconda-ks.cfg grafana-9.2.3.linux-amd64.tar.gz node_exporter-0.16.0.linux-amd64.tar.gz python3yizhuang.tar.gz
[root@mcw04 ~]# cd pushgateway-1.7.0.linux-amd64/
[root@mcw04 pushgateway-1.7.0.linux-amd64]# ls
LICENSE NOTICE pushgateway
[root@mcw04 pushgateway-1.7.0.linux-amd64]# echo $PATH
/usr/local/jdk/bin:/opt/hadoop/bin:/opt/hadoop/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin
[root@mcw04 pushgateway-1.7.0.linux-amd64]# cp pushgateway /usr/local/bin/
[root@mcw04 pushgateway-1.7.0.linux-amd64]# pushgateway --version
pushgateway, version 1.7.0 (branch: HEAD, revision: 109280c17d29059623c6f5dbf1d6babab34166cf)
build user: root@c05cb3457dcb
build date: 20240119-13:28:37
go version: go1.21.6
platform: linux/amd64
tags: unknown
[root@mcw04 pushgateway-1.7.0.linux-amd64]#
配置和运行
监听地址默认就是--web.listen-address="0.0.0.0:9091" 这个
[root@mcw04 pushgateway-1.7.0.linux-amd64]# cd
[root@mcw04 ~]# pushgateway --web.listen-address="0.0.0.0:9091"
ts=2024-02-13T04:31:47.165Z caller=main.go:86 level=info msg="starting pushgateway" version="(version=1.7.0, branch=HEAD, revision=109280c17d29059623c6f5dbf1d6babab34166cf)"
ts=2024-02-13T04:31:47.165Z caller=main.go:87 level=info build_context="(go=go1.21.6, platform=linux/amd64, user=root@c05cb3457dcb, date=20240119-13:28:37, tags=unknown)"
ts=2024-02-13T04:31:47.213Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9091
ts=2024-02-13T04:31:47.213Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9091
访问：http://10.0.0.14:9091/
指标默认存储在内存，持久化存储指标需要加下面参数，指定持久化文件
[root@mcw04 ~]# pushgateway --persistence.file="/tmp/pushgateway_persist"
ts=2024-02-13T04:37:54.994Z caller=main.go:86 level=info msg="starting pushgateway" version="(version=1.7.0, branch=HEAD, revision=109280c17d29059623c6f5dbf1d6babab34166cf)"
ts=2024-02-13T04:37:54.995Z caller=main.go:87 level=info build_context="(go=go1.21.6, platform=linux/amd64, user=root@c05cb3457dcb, date=20240119-13:28:37, tags=unknown)"
ts=2024-02-13T04:37:54.998Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9091
ts=2024-02-13T04:37:54.998Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9091
默认5分钟写入一次，可以用--persistence.interval参数覆盖掉
使用systemd管理服务
[root@mcw04 ~]# vim /usr/lib/systemd/system/pushgateway.service
[root@mcw04 ~]# cat /usr/lib/systemd/system/pushgateway.service
[Unit]
Description=pushgateway
Documentation=https://prometheus.io/docs/introduction/overview
After=network-online.target remote-fs.target nss-lookup.target
Wants=network-online.target [Service]
Type=simple
PIDFile==/var/run/pushgateway.pid
ExecStart=/usr/local/pushgateway \
--persistence.file="/usr/local/mcwpushgateway/pushgateway_persist_file" \
--persistence.interval=5m \
--web.listen-address=:9091
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s TERM $MAINPID [Install]
WantedBy=multi-user.target
[root@mcw04 ~]# systemctl daemon-reload
[root@mcw04 ~]# systemctl start pushgateway
[root@mcw04 ~]# systemctl status pushgateway
● pushgateway.service - pushgateway
Loaded: loaded (/usr/lib/systemd/system/pushgateway.service; disabled; vendor preset: disabled)
Active: active (running) since Tue 2024-02-13 20:36:16 CST; 6s ago
Docs: https://prometheus.io/docs/introduction/overview
Main PID: 45252 (pushgateway)
CGroup: /system.slice/pushgateway.service
└─45252 /usr/local/pushgateway --persistence.file="/usr/local/mcwpushgateway/pushgateway_persist_file" --persistence.interval=5m --web.listen-address=:9091 Feb 13 20:36:16 mcw04 systemd[1]: Started pushgateway.
Feb 13 20:36:16 mcw04 systemd[1]: Starting pushgateway...
Feb 13 20:36:17 mcw04 pushgateway[45252]: ts=2024-02-13T12:36:17.022Z caller=main.go:86 level=info msg="starting pushgateway" version="(version=1.7.0, branch=HEAD, re...ab34166cf)"
Feb 13 20:36:17 mcw04 pushgateway[45252]: ts=2024-02-13T12:36:17.022Z caller=main.go:87 level=info build_context="(go=go1.21.6, platform=linux/amd64, user=root@c05cb3...s=unknown)"
Feb 13 20:36:17 mcw04 pushgateway[45252]: ts=2024-02-13T12:36:17.085Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9091
Feb 13 20:36:17 mcw04 pushgateway[45252]: ts=2024-02-13T12:36:17.085Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9091
Hint: Some lines were ellipsized, use -l to show in full.
[root@mcw04 ~]#
向pushgateway发送指标(一个地址就是一个指标组，可以放多个指标)
发送指标之前
推送指标
[root@mcw04 ~]# echo 'batchjob1_user_counter 2' | curl --data-binary @- http://localhost:9091/metrics/job/batchjob1
[root@mcw04 ~]#
刷新一下，pushgateway页面
再次执行
[root@mcw04 ~]# echo 'batchjob1_user_counter 2' | curl --data-binary @- http://localhost:9091/metrics/job/batchjob1/instance/sidekiq_server
[root@mcw04 ~]#
新增一个，虽然job一样，但是instance不一样
修改一下instance的值,又多一个
[root@mcw04 ~]# echo 'batchjob1_user_counter 2' | curl --data-binary @- http://localhost:9091/metrics/job/batchjob1/instance/sidekiq_server2
[root@mcw04 ~]#
其它的不变，就新增一个键值对，用斜线分开，就多了一个标签
[root@mcw04 ~]# echo 'batchjob1_user_counter 2' | curl --data-binary @- http://localhost:9091/metrics/job/batchjob1/instance/sidekiq_server2/myname/machangwei
[root@mcw04 ~]#
新增标签或者修改标签，都是新增一条记录
[root@mcw04 ~]# echo 'batchjob1_user_counter 2' | curl --data-binary @- http://localhost:9091/metrics/job/batchjob1/instance/sidekiq_server2/myname/machangwei2
[root@mcw04 ~]#
即使只是修改指标的值为4，执行命令，不会新增一条记录.为指标添加标签
[root@mcw04 ~]# echo 'batchjob1_user_counter 4' | curl --data-binary @- http://localhost:9091/metrics/job/batchjob1/instance/sidekiq_server2/myname/machangwei2
[root@mcw04 ~]#
可以把其他的都删除掉
只留下一条
其它的不变，
[root@mcw04 ~]# echo 'batchjob1_user_counter 8' | curl --data-binary @- http://localhost:9091/metrics/job/batchjob1/instance/sidekiq_server2/myname/machangwei2
[root@mcw04 ~]#
刷新一下，指标值变了，并没有生成新的记录
没有job就会报错
[root@mcw04 ~]# echo 'batchjob1_user_counter 8' | curl --data-binary @- http://localhost:9091/metrics/instance/sidekiq_server2/myname/machangwei2
404 page not found
[root@mcw04 ~]# echo 'batchjob1_user_counter 8' | curl --data-binary @- http://localhost:9091/metrics/instance/sidekiq_server2/myname/machangwei2/job/xxx
404 page not found
[root@mcw04 ~]# echo 'batchjob1_user_counter 8' | curl --data-binary @- http://localhost:9091/metrics/job/xxxx/instance/sidekiq_server2/myname/machangwei2/
odd number of components in label string "/instance/sidekiq_server2/myname/machangwei2/"
[root@mcw04 ~]# echo 'batchjob1_user_counter 8' | curl --data-binary @- http://localhost:9091/metrics/job/xxxxx/instance/sidekiq_server2/myname/machangwei2
[root@mcw04 ~]#
综上： echo 'batchjob1_user_counter 8' | curl --data-binary @- http://localhost:9091/metrics/job/<jobname>/{/<label>/<label>/}
在推送中传递TYPE和HELP标明指标类型。注意EOF 40 这些后面不要加空格，不然容易没法结束输入信息
[root@mcw04 ~]# cat <<EOF | curl --data-binary @- http://localhost:9091/metrics/job/batchjob1/instance/sidekiq_server2/myname/machangwei2
> # TYPE batchjob1_user_counter counter
> # HELP batchjob1_user_coounter A metric from BatchJob1.
> batchjob1_sales_counter{job_id="123ABC"} 1
> mycpu 20
> mymem 40
> EOF
[root@mcw04 ~]#
这是新增加的指标
查看这里，目前只有这一个指标
推送地址一样，指标值改成3
[root@mcw04 ~]# echo 'batchjob1_user_counter 3' | curl --data-binary @- http://localhost:9091/metrics/job/xxxxx/instance/sidekiq_server2/myname/machangwei2
[root@mcw04 ~]#
对应的值也改了
推送地址不变，指标名称不同
[root@mcw04 ~]# echo 'xiaomazhibiao1 666' | curl --data-binary @- http://localhost:9091/metrics/job/xxxxx/instance/sidekiq_server2/myname/machangwei2
[root@mcw04 ~]#
同一个下面会多个指标。也就是说推送地址是一样的，可以推送多个指标；一个地址就是多个指标的组合，是一组指标的集合，因此后面删除按钮写着删除组，也就解释的通了。
上面是推送一个指标，下面是推送多个指标，不过推送失败了。我们也可以看到，这里标签，值都给变成下面那种形式的来显示一下了
[root@mcw04 ~]# cat <<EOF | curl --data-binary @- http://localhost:9091/metrics/job/xxxxx/instance/sidekiq_server2/myname/machangwei2
> # TYPE batchjob1_user_counter counter
> # HELP batchjob1_user_coounter A metric from BatchJob1.
> batchjob1_sales_counter{job_id="123ABC"} 1
> # TYPE mycpu counter
> # HELP mycpu A metric from BatchJob1.
> mycpu 20
> # TYPE mymem counter
> # HELP mymem A metric from BatchJob1.
> mymem 40
> EOF
pushed metrics are invalid or inconsistent with existing metrics: 2 error(s) occurred:
* collected metric "mycpu" { label:{name:"instance" value:"sidekiq_server2"} label:{name:"job" value:"batchjob1"} label:{name:"myname" value:"machangwei2"} untyped:{value:20}} is not a COUNTER
* collected metric "mymem" { label:{name:"instance" value:"sidekiq_server2"} label:{name:"job" value:"batchjob1"} label:{name:"myname" value:"machangwei2"} untyped:{value:40}} is not a COUNTER
[root@mcw04 ~]#
上面报错，不是counter类型的指标，下面页面中，也显示上次推送失败了，
多次试验，才知道，多个指标发送成功，指标是counter的话，需要用_counter结尾来命名才可以成功推送成功
[root@mcw04 ~]# cat <<EOF | curl --data-binary @- http://localhost:9091/metrics/job/xxxxx/instance/sidekiq_server2/myname/machangwei2
> # TYPE batchjob1_user_counter counter
> # HELP batchjob1_user_coounter A metric from xxxxx.
> batchjob1_sales_counter{job_id="123ABC"} 1
> # TYPE mycpu counter
> # HELP mycpu A metric from xxxxx.
> mycpu_counter{job_id="123ABC"} 20
> # TYPE mymem counter
> # HELP mymem A metric from xxxxx.
> mymem_counter{job_id="123ABC"} 40
> EOF
[root@mcw04 ~]#
在pushgateway上查看指标
[root@mcw04 ~]# curl http://localhost:9091/metrics
# TYPE batchjob1_sales_counter untyped
batchjob1_sales_counter{instance="sidekiq_server2",job="batchjob1",job_id="123ABC",myname="machangwei2"} 1
batchjob1_sales_counter{instance="sidekiq_server2",job="xxxxx",job_id="123ABC",myname="machangwei2"} 1
# TYPE batchjob1_user_counter untyped
batchjob1_user_counter{instance="sidekiq_server2",job="batchjob1",myname="machangwei2"} 8
batchjob1_user_counter{instance="sidekiq_server2",job="xxxxx",myname="machangwei2"} 3
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 2.7942e-05
go_gc_duration_seconds{quantile="0.25"} 3.4098e-05
go_gc_duration_seconds{quantile="0.5"} 4.0877e-05
go_gc_duration_seconds{quantile="0.75"} 6.1473e-05
go_gc_duration_seconds{quantile="1"} 0.000781573
go_gc_duration_seconds_sum 0.006027564
go_gc_duration_seconds_count 106
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 12
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.21.6"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 3.003712e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 1.6962048e+07
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.455808e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 142363
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 4.031136e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 3.003712e+06
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 3.047424e+06
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 4.75136e+06
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 9947
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 2.981888e+06
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 7.798784e+06
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.7078412542246678e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 152310
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 2400
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 15600
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 77448
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 81480
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 5.691992e+06
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 548976
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 589824
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 589824
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 1.4521608e+07
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 7
# TYPE mycpu untyped
mycpu{instance="sidekiq_server2",job="batchjob1",myname="machangwei2"} 20
# TYPE mycpu_counter untyped
mycpu_counter{instance="sidekiq_server2",job="xxxxx",job_id="123ABC",myname="machangwei2"} 20
# TYPE mymem untyped
mymem{instance="sidekiq_server2",job="batchjob1",myname="machangwei2"} 40
# TYPE mymem_counter untyped
mymem_counter{instance="sidekiq_server2",job="xxxxx",job_id="123ABC",myname="machangwei2"} 40
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 0.25
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 4096
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 11
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 1.7883136e+07
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.70782788203e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 1.27039488e+09
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes 1.8446744073709552e+19
# HELP push_failure_time_seconds Last Unix time when changing this group in the Pushgateway failed.
# TYPE push_failure_time_seconds gauge
push_failure_time_seconds{instance="sidekiq_server2",job="batchjob1",myname="machangwei2"} 0
push_failure_time_seconds{instance="sidekiq_server2",job="xxxxx",myname="machangwei2"} 1.707837778691478e+09
# HELP push_time_seconds Last Unix time when changing this group in the Pushgateway succeeded.
# TYPE push_time_seconds gauge
push_time_seconds{instance="sidekiq_server2",job="batchjob1",myname="machangwei2"} 1.7078360100219057e+09
push_time_seconds{instance="sidekiq_server2",job="xxxxx",myname="machangwei2"} 1.7078378257050233e+09
# HELP pushgateway_build_info A metric with a constant '1' value labeled by version, revision, branch, goversion from which pushgateway was built, and the goos and goarch for the build.
# TYPE pushgateway_build_info gauge
pushgateway_build_info{branch="HEAD",goarch="amd64",goos="linux",goversion="go1.21.6",revision="109280c17d29059623c6f5dbf1d6babab34166cf",tags="unknown",version="1.7.0"} 1
# HELP pushgateway_http_push_duration_seconds HTTP request duration for pushes to the Pushgateway.
# TYPE pushgateway_http_push_duration_seconds summary
pushgateway_http_push_duration_seconds{method="post",quantile="0.1"} NaN
pushgateway_http_push_duration_seconds{method="post",quantile="0.5"} NaN
pushgateway_http_push_duration_seconds{method="post",quantile="0.9"} NaN
pushgateway_http_push_duration_seconds_sum{method="post"} 0.012381439999999999
pushgateway_http_push_duration_seconds_count{method="post"} 20
# HELP pushgateway_http_push_size_bytes HTTP request size for pushes to the Pushgateway.
# TYPE pushgateway_http_push_size_bytes summary
pushgateway_http_push_size_bytes{method="post",quantile="0.1"} NaN
pushgateway_http_push_size_bytes{method="post",quantile="0.5"} NaN
pushgateway_http_push_size_bytes{method="post",quantile="0.9"} NaN
pushgateway_http_push_size_bytes_sum{method="post"} 6156
pushgateway_http_push_size_bytes_count{method="post"} 20
# HELP pushgateway_http_requests_total Total HTTP requests processed by the Pushgateway, excluding scrapes.
# TYPE pushgateway_http_requests_total counter
pushgateway_http_requests_total{code="200",handler="push",method="post"} 12
pushgateway_http_requests_total{code="200",handler="static",method="get"} 22
pushgateway_http_requests_total{code="200",handler="status",method="get"} 21
pushgateway_http_requests_total{code="202",handler="delete",method="delete"} 4
pushgateway_http_requests_total{code="400",handler="push",method="post"} 8
# TYPE xiaomazhibiao1 untyped
xiaomazhibiao1{instance="sidekiq_server2",job="xxxxx",myname="machangwei2"} 666
[root@mcw04 ~]#
每个指标组都有 push_time_seconds，最后一次推送时间
[root@mcw04 ~]# curl http://localhost:9091/metrics|grep push_time
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 9062 0 9062 0 0 1517k 0 --:--:-- --:--:-- --:--:-- 1769k
# HELP push_time_seconds Last Unix time when changing this group in the Pushgateway succeeded.
# TYPE push_time_seconds gauge
push_time_seconds{instance="sidekiq_server2",job="batchjob1",myname="machangwei2"} 1.7078360100219057e+09
push_time_seconds{instance="sidekiq_server2",job="xxxxx",myname="machangwei2"} 1.7078378257050233e+09
[root@mcw04 ~]#
[root@mcw04 ~]# echo 'batchjob1_user_counter 3' | curl --data-binary @- http://localhost:9091/metrics/job/xxxxx/instance/sidekiq_server2/myname/machangwei3
[root@mcw04 ~]# curl http://localhost:9091/metrics|grep push_time
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 9370 0 9370 0 0 1481k 0 --:--:-- --:--:-- --:--:-- 1525k
# HELP push_time_seconds Last Unix time when changing this group in the Pushgateway succeeded.
# TYPE push_time_seconds gauge
push_time_seconds{instance="sidekiq_server2",job="batchjob1",myname="machangwei2"} 1.7078360100219057e+09
push_time_seconds{instance="sidekiq_server2",job="xxxxx",myname="machangwei2"} 1.7078378257050233e+09
push_time_seconds{instance="sidekiq_server2",job="xxxxx",myname="machangwei3"} 1.7078416958989272e+09
[root@mcw04 ~]#
删除pushgateway中的指标
报错了
[root@mcw04 ~]# curl -X delete http://localhost:9091/metrics/job/xxxxx/instance/
Method Not Allowed
[root@mcw04 ~]#
大写才行
[root@mcw04 ~]# curl -X DELETE http://localhost:9091/metrics/job/xxxxx/instance/
[root@mcw04 ~]#
查看，缺少一个持久化保存的文件，创建重启
[root@mcw04 ~]# less /var/log/messages
[root@mcw04 ~]# systemctl status pushgateway.service
● pushgateway.service - pushgateway
Loaded: loaded (/usr/lib/systemd/system/pushgateway.service; disabled; vendor preset: disabled)
Active: active (running) since Tue 2024-02-13 20:38:02 CST; 4h 4min ago
Docs: https://prometheus.io/docs/introduction/overview
Process: 45314 ExecStop=/bin/kill -s TERM $MAINPID (code=exited, status=0/SUCCESS)
Main PID: 45317 (pushgateway)
CGroup: /system.slice/pushgateway.service
└─45317 /usr/local/pushgateway --persistence.file="/usr/local/mcwpushgateway/pushgateway_persist_file" --persistence.interval=5m --web.listen-address=:9091 Feb 13 23:16:53 mcw04 pushgateway[45317]: ts=2024-02-13T15:16:53.754Z caller=diskmetricstore.go:219 level=error msg="error persisting metrics" err="open \"/usr/local/... directory"
Feb 13 23:16:53 mcw04 pushgateway[45317]: ts=2024-02-13T15:16:53.754Z caller=push.go:141 level=error msg="pushed metrics are invalid or inconsistent with existing met...:\"job\" va
Feb 13 23:20:09 mcw04 pushgateway[45317]: ts=2024-02-13T15:20:09.321Z caller=push.go:141 level=error msg="pushed metrics are invalid or inconsistent with existing met...:\"job\" va
Feb 13 23:20:37 mcw04 pushgateway[45317]: ts=2024-02-13T15:20:37.483Z caller=push.go:141 level=error msg="pushed metrics are invalid or inconsistent with existing met...:\"job\" va
Feb 13 23:20:53 mcw04 pushgateway[45317]: ts=2024-02-13T15:20:53.527Z caller=push.go:141 level=error msg="pushed metrics are invalid or inconsistent with existing met...:\"job\" va
Feb 13 23:21:53 mcw04 pushgateway[45317]: ts=2024-02-13T15:21:53.756Z caller=diskmetricstore.go:219 level=error msg="error persisting metrics" err="open \"/usr/local/... directory"
Feb 13 23:22:58 mcw04 pushgateway[45317]: ts=2024-02-13T15:22:58.692Z caller=push.go:141 level=error msg="pushed metrics are invalid or inconsistent with existing met...:\"job\" va
Feb 13 23:26:53 mcw04 pushgateway[45317]: ts=2024-02-13T15:26:53.758Z caller=diskmetricstore.go:219 level=error msg="error persisting metrics" err="open \"/usr/local/... directory"
Feb 14 00:28:15 mcw04 pushgateway[45317]: ts=2024-02-13T16:28:15.899Z caller=diskmetricstore.go:219 level=error msg="error persisting metrics" err="open \"/usr/local/... directory"
Feb 14 00:37:45 mcw04 pushgateway[45317]: ts=2024-02-13T16:37:45.769Z caller=diskmetricstore.go:219 level=error msg="error persisting metrics" err="open \"/usr/local/... directory"
Hint: Some lines were ellipsized, use -l to show in full.
[root@mcw04 ~]# ls /usr/local/mcwpushgateway/pushgateway_persist_file
ls: cannot access /usr/local/mcwpushgateway/pushgateway_persist_file: No such file or directory
[root@mcw04 ~]# touch /usr/local/mcwpushgateway/pushgateway_persist_file
[root@mcw04 ~]# systemctl restart pushgateway
[root@mcw04 ~]# systemctl status pushgateway.service
● pushgateway.service - pushgateway
Loaded: loaded (/usr/lib/systemd/system/pushgateway.service; disabled; vendor preset: disabled)
Active: active (running) since Wed 2024-02-14 00:43:26 CST; 2s ago
Docs: https://prometheus.io/docs/introduction/overview
Process: 48123 ExecStop=/bin/kill -s TERM $MAINPID (code=exited, status=0/SUCCESS)
Main PID: 48126 (pushgateway)
CGroup: /system.slice/pushgateway.service
└─48126 /usr/local/pushgateway --persistence.file="/usr/local/mcwpushgateway/pushgateway_persist_file" --persistence.interval=5m --web.listen-address=:9091 Feb 14 00:43:26 mcw04 systemd[1]: Started pushgateway.
Feb 14 00:43:26 mcw04 systemd[1]: Starting pushgateway...
Feb 14 00:43:26 mcw04 pushgateway[48126]: ts=2024-02-13T16:43:26.980Z caller=main.go:86 level=info msg="starting pushgateway" version="(version=1.7.0, branch=HEAD, re...ab34166cf)"
Feb 14 00:43:26 mcw04 pushgateway[48126]: ts=2024-02-13T16:43:26.980Z caller=main.go:87 level=info build_context="(go=go1.21.6, platform=linux/amd64, user=root@c05cb3...s=unknown)"
Feb 14 00:43:26 mcw04 pushgateway[48126]: ts=2024-02-13T16:43:26.982Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9091
Feb 14 00:43:26 mcw04 pushgateway[48126]: ts=2024-02-13T16:43:26.982Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9091
Hint: Some lines were ellipsized, use -l to show in full.
[root@mcw04 ~]#
重启之后，因为没有持久化，都没了
创建一个，重启之后还是没有了
[root@mcw04 ~]# echo 'batchjob1_user_counter 3' | curl --data-binary @- http://localhost:9091/metrics/job/xxxxx/instance/sidekiq_server2/myname/machangwei3
[root@mcw04 ~]#
修改了持久化的间隔，但是没有持久化 成功，重启之后还是没有了。上面删除命令也没有生效
[root@mcw04 ~]# vim /usr/lib/systemd/system/pushgateway.service
[root@mcw04 ~]# cat /usr/lib/systemd/system/pushgateway.service
[Unit]
Description=pushgateway
Documentation=https://prometheus.io/docs/introduction/overview
After=network-online.target remote-fs.target nss-lookup.target
Wants=network-online.target [Service]
Type=simple
PIDFile==/var/run/pushgateway.pid
ExecStart=/usr/local/pushgateway \
--persistence.file="/usr/local/mcwpushgateway/pushgateway_persist_file" \
--persistence.interval=10s \
--web.listen-address=:9091
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s TERM $MAINPID [Install]
WantedBy=multi-user.target
[root@mcw04 ~]# systemctl daemon-reload
[root@mcw04 ~]# systemctl restart pushgateway
[root@mcw04 ~]# cat /usr/local/mcwpushgateway/pushgateway_persist_file
[root@mcw04 ~]# echo 'batchjob1_user_counter 3' | curl --data-binary @- http://localhost:9091/metrics/job/xxxxx/instance/sidekiq_server2/myname/machangwei3
[root@mcw04 ~]# cat /usr/local/mcwpushgateway/pushgateway_persist_file
[root@mcw04 ~]# cat /usr/local/mcwpushgateway/pushgateway_persist_file
[root@mcw04 ~]#
[root@mcw04 ~]#
[root@mcw04 ~]# cat /usr/local/mcwpushgateway/pushgateway_persist_file
[root@mcw04 ~]# systemctl restart pushgateway
[root@mcw04 ~]#
从客户端发送指标到pushgateway (python案例)
修改一下pushgateway所在ip，然后执行脚本
[root@mcw04 ~]# python3 pythonpush.py
[root@mcw04 ~]# cat pythonpush.py
#!/usr/bin/env /usr/bin/python3
# -*- coding:utf-8 -*- from prometheus_client import CollectorRegistry, Gauge, push_to_gateway if __name__ == '__main__':
registry = CollectorRegistry()
labels = ['req_status', 'req_method', 'req_url']
g_one = Gauge('requests_total', 'url请求次数', labels, registry=registry)
g_two = Gauge('avg_response_time_seconds', '1分钟内的URL平均响应时间', labels, registry=registry)
g_one.labels('200','GET', '/test/url').set(1) #set设定值
g_two.labels('200','GET', '/test/api/url/').set(10) #set设定值
push_to_gateway('http://10.0.0.14:9091', job='SampleURLMetrics', registry=registry)
[root@mcw04 ~]#
可以看到，多了这个job这个指标组
添加一个指标
[root@mcw04 ~]# cat pythonpush.py
#!/usr/bin/env /usr/bin/python3
# -*- coding:utf-8 -*- from prometheus_client import CollectorRegistry, Gauge, push_to_gateway if __name__ == '__main__':
registry = CollectorRegistry()
labels = ['req_status', 'req_method', 'req_url']
g_one = Gauge('requests_total', 'url请求次数', labels, registry=registry)
g_two = Gauge('avg_response_time_seconds', '1分钟内的URL平均响应时间', labels, registry=registry)
g_one.labels('200','GET', '/test/url').set(1) #set设定值
g_two.labels('200','GET', '/test/api/url/').set(10) #set设定值
push_to_gateway('http://10.0.0.14:9091', job='SampleURLMetrics', registry=registry)
[root@mcw04 ~]# vim pythonpush.py
[root@mcw04 ~]# cat pythonpush.py
#!/usr/bin/env /usr/bin/python3
# -*- coding:utf-8 -*- from prometheus_client import CollectorRegistry, Gauge, push_to_gateway if __name__ == '__main__':
registry = CollectorRegistry()
labels = ['req_status', 'req_method', 'req_url']
g_one = Gauge('requests_total', 'url请求次数', labels, registry=registry)
g_two = Gauge('avg_response_time_seconds', '1分钟内的URL平均响应时间', labels, registry=registry)
g_three=Gauge('zhibiao_name', '小马测试', ['myname','myage'], registry=registry)
g_one.labels('200','GET', '/test/url').set(1) #set设定值
g_two.labels('200','GET', '/test/api/url/').set(10) #set设定值
g_three.labels('machangwei','18', ).set(10) #set设定值
push_to_gateway('http://10.0.0.14:9091', job='SampleURLMetrics', registry=registry)
[root@mcw04 ~]#
[root@mcw04 ~]# python3 pythonpush.py
[root@mcw04 ~]#
指标名称，指标备注描述信息，指标标签。
抓取pushgateway
[root@mcw03 ~]# tail -7 /etc/prometheus.yml
# action: labeldrop
- job_name: pushgateway
honor_labels: true
file_sd_configs:
- files:
- targets/pushgateway/*.json
refresh_interval: 5m
[root@mcw03 ~]# mkdir /etc/targets/pushgateway
[root@mcw03 ~]#
[root@mcw03 ~]#
[root@mcw03 ~]#
[root@mcw03 ~]#
[root@mcw03 ~]# vim /etc/targets/pushgateway/mcw04.json
[root@mcw03 ~]# cat /etc/targets/pushgateway/mcw04.json
[{
"targets": ["10.0.0.14:9091"]
}]
[root@mcw03 ~]# curl -X POST http://localhost:9090/-/reload
[root@mcw03 ~]#
如上配置，添加基于文件的抓取目标，文件里的目标是pushgateway的访问地址
然后在浏览器表达式中，可以看到我们推送到pushgateway的指标
而下面的指标，是上面章节中用python脚本推送到pushgateway的
在这里，也能看到这个pushgateway
修改honer_labels为false
[root@mcw03 ~]# tail -7 /etc/prometheus.yml
# action: labeldrop
- job_name: pushgateway
honor_labels: false
file_sd_configs:
- files:
- targets/pushgateway/*.json
refresh_interval: 5m
[root@mcw03 ~]# curl -X POST http://localhost:9090/-/reload
然后再推送一个指标到pushgateway
[root@mcw04 ~]# echo 'myname xiaoma' | curl --data-binary @- http://localhost:9091/metrics/job/xxxxx/instance/sidekiq_server2/myname/machangwei4
text format parsing error in line 1: expected float as value, got "xiaoma"
[root@mcw04 ~]# echo 'myage 18' | curl --data-binary @- http://localhost:9091/metrics/job/xxxxx/instance/sidekiq_server2/myname/machangwei4
[root@mcw04 ~]#
指标已经存在了
重载Prometheus，让它立即抓取新指标
pushgateway上它本身带的下面两个标签，都被重写改名了，加了exported_前缀，
prometheus使用4的更多相关文章
- prometheus监控系统
关于Prometheus Prometheus是一套开源的监控系统,它将所有信息都存储为时间序列数据:因此实现一种Profiling监控方式,实时分析系统运行的状态.执行时间.调用次数等,以找到系统的 ...
- Prometheus 系统监控方案 一
最近一直在折腾时序类型的数据库,经过一段时间项目应用,觉得十分不错.而Prometheus又是刚刚推出不久的开源方案,中文资料较少,所以打算写一系列应用的实践过程分享一下. Prometheus 是什 ...
- Prometheus 系统监控方案 二 安装与配置
下载Prometheus 下载最新安装包,本文说的都是在Linux x64下面内容,其它平台没尝试过,请选择合适的下载. Prometheus 主程序,主要是负责存储.抓取.聚合.查询方面. Aler ...
- [系统集成] 部署 mesos-exporter 和 prometheus 监控 mesos task
前几天我在mesos平台上基于 cadvisor部署了 influxdb 和 grafana,用于监控 mesos 以及 docker app 运行信息,发现这套监控系统不太适合 mesos + do ...
- Docker 监控- Prometheus VS Cloud Insight
如今,越来越多的公司开始使用 Docker 了,2 / 3 的公司在尝试了 Docker 后最终使用了它.为了能够更精确的分配每个容器能使用的资源,我们想要实时获取容器运行时使用资源的情况,怎样对 D ...
- 安装prometheus+grafana监控mysql redis kubernetes等
1.prometheus安装 wget https://github.com/prometheus/prometheus/releases/download/v1.5.2/prometheus-1.5 ...
- Prometheus : 入门
Prometheus 是一个开源的监控系统.支持灵活的查询语言(PromQL),采用 http 协议的 pull 模式拉取数据等特点使 Prometheus 即简单易懂又功能强大. Prometheu ...
- Prometheus 架构 - 每天5分钟玩转 Docker 容器技术（83）
Prometheus 是一个非常优秀的监控工具.准确的说,应该是监控方案.Prometheus 提供了监控数据搜集.存储.处理.可视化和告警一套完整的解决方案. 让我们先来看看 Prometheus ...
- 剖析Prometheus的内部存储机制
Prometheus有着非常高效的时间序列数据存储方法,每个采样数据仅仅占用3.5byte左右空间,上百万条时间序列,30秒间隔,保留60天,大概花了200多G(引用官方PPT). 接下来让我们看看他 ...
- Prometheus 到底 NB 在哪里？- 每天5分钟玩转 Docker 容器技术（84）
本节讨论 Prometheus 的核心,多维数据模型.我们先来看一个例子. 比如要监控容器 webapp1 的内存使用情况,最传统和典型的方法是定义一个指标 container_memory_usag ...
随机推荐
- HarmonyOS音频开发指导：使用AudioRenderer开发音频播放功能
AudioRenderer是音频渲染器,用于播放PCM(Pulse Code Modulation)音频数据,相比AVPlayer而言,可以在输入前添加数据预处理,更适合有音频开发经验的开发者,以 ...
- HarmonyOS自动化测试框架—Hypium
原文:https://mp.weixin.qq.com/s/nb5txfDcmGn_VZJXRPEYUQ,点击链接查看更多技术内容. 应用开发过程中,要确保应用的功能和界面能满足预期,往往需要通过测试 ...
- BI 和报表有什么区别
BI 从早期提出的概念上来划分可以分为数据仓库.ETL.olap 和报表这几部分可以看到报表只是 BI 中的一个组成部分,只不过数据在 web 端展示时通常是通过报表形式,所以经常会把报表当做是 BI ...
- 重新整理.net core 计1400篇[五] (.net core 添加mvc 中间件 )
前言 前面提过.net core web实际上是一个服务和若干个中间件组成. 现在我们有一个服务了,那么可想而知我们要完成mvc这种功能,我们需要的就是一个中间件. 好的,接下来,我们只要添加中间件即 ...
- 什么是ip协议一
前言 两节结束,为网络底层系列做铺垫. 首先来看一张图: IOS有七层,但是我们可以简化层4层,ip属于传输层,可以说是非常重要,下面简单的做一个介绍. 正文 ip的介绍: 1.ip是tcp/ip 协 ...
- mmcls/mmdet模型部署至 TorchServe
mmcls/mmdet模型部署至 TorchServe 官方教程:模型部署至 TorchServe - MMClassification 0.23.2 文档 接口说明: serve/inference ...
- 调用App Store Connect Api
对iOS的证书.描述文件.账号.设备等管理,之前都去苹果开发者中心操作,官网上操作也比较繁杂,想搞一些自动化之类的,更是麻烦,有时候官网都打不开-- 其实苹果还提供里一套API接口,创建证书.创建账号 ...
- 争论不休的一个话题：金额到底是用Long还是BigDecimal？
在网上一直流传着一个争论不休的话题:金额到底是用Long还是用BigDecimal?这个话题一出在哪都会引起异常无比激烈的讨论.... 比如说这个观点:算钱用BigDecimal是常识 有支持用Lon ...
- JavaScript中如何实现函数缓存？有哪些应用场景？
一.是什么 函数缓存,就是将函数运算过的结果进行缓存 本质上就是用空间(缓存存储)换时间(计算过程) 常用于缓存数据计算结果和缓存对象 const add = (a,b) => a+b;cons ...
- 04_el和data的两种写法
总结: data与el的两种写法 1.el的两种写法 (1)new Vue时需要配置el属性: (2)先创建Vue实例,然后再通过vm.$mount('#root')指 ...