在k8s集群中,利用prometheus的jmx_exporter进行tomcat的JVM性能监控,并用grafana作前端展示
查找了很多文档,没有完全达到我要求的,
于是,作了一定的调整,成现在这样。
操作步骤如下:
一,准备好两个文件。
jmx_prometheus_javaagent-0.3.1.jar
jmx_exporter.yml
jmx_exporter.yml作最简单配置如下:
--- rules: - pattern: ".*"
二,tomcat的启动项,一直在bin下单独的setenv.sh里操作,新增如下启动选项,将jar包及yaml加入启动选项。
JAVA_OPTS="$JAVA_OPTS -javaagent:/xxx/bin/jmx_prometheus_javaagent-0.3.1.jar=1234:/xxx/bin/jmx_exporter.yml"
三,制作dockerfile时,要将这两个东东也CP到上面指定目录,并EXPOSE一个新的端口。
... COPY --chown=a:a tomcat /xxx/tomcat COPY --chown=a:a jdk /xxx/jdk COPY --chown=a:a jmx_prometheus_javaagent-0.3.1.jar /xxx/tomcat/bin/ COPY --chown=a:a jmx_exporter.yml /xxx/tomcat/bin/ WORKDIR ${CATALINA_HOME} USER a EXPOSE 8080 1234
四,更新prometheus的配置文件configmap,新增一个job。
- job_name: 'tomcat-pods' tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: endpoints relabel_configs: - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape, __meta_kubernetes_service_annotation_prometheus_io_jvm_scrape] regex: true;true action: keep - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_app_metrics_patn] action: replace target_label: __metrics_path__ regex: (.+) - source_labels: [__meta_kubernetes_pod_ip, __meta_kubernetes_service_annotation_prometheus_io_app_metrics_port] action: replace target_label: __address__ regex: (.+);(.+) - source_labels: [__meta_kubernetes_namespace] action: replace target_label: kubernetes_namespace - source_labels: [__meta_kubernetes_pod_name] action: replace target_label: kubernetes_pod_name
五,在对应的需要监控JVM的应用时,新增一个SERVICE。
apiVersion: v1 kind: Service metadata: labels: name: xxx name: xxx namespace: default annotations: prometheus.io/scrape: "true" prometheus.io/jvm-scrape: "true" prometheus.io/jvm-port: "1234" prometheus.io/jvm-path: "/metrics" spec: ports: - port: 1234 targetPort: 1234 selector: name: xxx
六,最后,导入grafana的插件文件。
{ "__inputs": [ { "name": "DS_PROM", "label": "prom", "description": "A prometheus datasource with JMX Exporter scraping", "type": "datasource", "pluginId": "prometheus", "pluginName": "Prometheus" } ], "__requires": [ { "type": "grafana", "id": "grafana", "name": "Grafana", "version": "4.5.0-pre1" }, { "type": "panel", "id": "graph", "name": "Graph", "version": "" }, { "type": "datasource", "id": "prometheus", "name": "Prometheus", "version": "1.0.0" } ], "annotations": { "list": [] }, "description": " tomcat JVM Dashboard for JVM metrics with Prometheus / JMX Exporter", "editable": true, "gnetId": 3066, "graphTooltip": 0, "id": 10, "iteration": 1544507474291, "links": [], "panels": [ { "aliasColors": {}, "bars": false, "dashLength": 10, "dashes": false, "datasource": "${DS_PROM}", "fill": 1, "gridPos": { "h": 7, "w": 24, "x": 0, "y": 0 }, "id": 2, "legend": { "alignAsTable": true, "avg": true, "current": true, "max": true, "min": true, "rightSide": true, "show": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], "nullPointMode": "null", "percentage": false, "pointradius": 5, "points": false, "renderer": "flot", "repeat": null, "seriesOverrides": [], "spaceLength": 10, "stack": false, "steppedLine": false, "targets": [ { "expr": "jvm_memory_bytes_used{kubernetes_pod_name=~\"$kubernetes_pod_name\"}", "format": "time_series", "interval": "", "intervalFactor": 5, "legendFormat": "{{area}} memory [{{instance}}]", "metric": "jvm_memory_bytes_used", "refId": "A", "step": 5 } ], "thresholds": [], "timeFrom": null, "timeShift": null, "title": "Memory used", "tooltip": { "shared": true, "sort": 0, "value_type": "individual" }, "type": "graph", "xaxis": { "buckets": null, "mode": "time", "name": null, "show": true, "values": [] }, "yaxes": [ { "format": "bytes", "label": null, "logBase": 1, "max": null, "min": null, "show": true }, { "format": "short", "label": null, "logBase": 1, "max": null, "min": null, "show": true } ], "yaxis": { "align": false, "alignLevel": null } }, { "aliasColors": {}, "bars": false, "dashLength": 10, "dashes": false, "datasource": "${DS_PROM}", "fill": 1, "gridPos": { "h": 7, "w": 12, "x": 0, "y": 7 }, "id": 3, "legend": { "avg": false, "current": false, "max": false, "min": false, "show": true, "total": false, "values": false }, "lines": true, "linewidth": 1, "links": [], "nullPointMode": "null", "percentage": false, "pointradius": 5, "points": false, "renderer": "flot", "seriesOverrides": [], "spaceLength": 10, "stack": false, "steppedLine": false, "targets": [ { "expr": "jvm_threads_current{kubernetes_pod_name=~\"$kubernetes_pod_name\"}", "format": "time_series", "intervalFactor": 5, "legendFormat": "current [{{instance}}]", "metric": "jvm_threads_current", "refId": "A", "step": 10 }, { "expr": "jvm_threads_daemon{kubernetes_pod_name=~\"$kubernetes_pod_name\"}", "format": "time_series", "intervalFactor": 5, "legendFormat": "daemon [{{instance}}]", "metric": "jvm_threads_daemon", "refId": "B", "step": 10 } ], "thresholds": [], "timeFrom": null, "timeShift": null, "title": "Threads used", "tooltip": { "shared": true, "sort": 0, "value_type": "individual" }, "type": "graph", "xaxis": { "buckets": null, "mode": "time", "name": null, "show": true, "values": [] }, "yaxes": [ { "decimals": 0, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, "show": true }, { "format": "short", "label": null, "logBase": 1, "max": null, "min": null, "show": true } ], "yaxis": { "align": false, "alignLevel": null } }, { "aliasColors": {}, "bars": false, "dashLength": 10, "dashes": false, "datasource": "${DS_PROM}", "fill": 1, "gridPos": { "h": 7, "w": 12, "x": 12, "y": 7 }, "id": 4, "legend": { "avg": false, "current": false, "max": false, "min": false, "show": true, "total": false, "values": false }, "lines": true, "linewidth": 1, "links": [], "nullPointMode": "null", "percentage": false, "pointradius": 5, "points": false, "renderer": "flot", "seriesOverrides": [], "spaceLength": 10, "stack": false, "steppedLine": false, "targets": [ { "expr": "jvm_classes_loaded{kubernetes_pod_name=~\"$kubernetes_pod_name\"}", "format": "time_series", "interval": "", "intervalFactor": 5, "legendFormat": "loaded [{{instance}}]", "metric": "jvm_classes_loaded", "refId": "A", "step": 10 } ], "thresholds": [], "timeFrom": null, "timeShift": null, "title": "Class loading", "tooltip": { "shared": true, "sort": 0, "value_type": "individual" }, "type": "graph", "xaxis": { "buckets": null, "mode": "time", "name": null, "show": true, "values": [] }, "yaxes": [ { "decimals": 0, "format": "short", "label": "", "logBase": 1, "max": null, "min": null, "show": true }, { "format": "short", "label": null, "logBase": 1, "max": null, "min": null, "show": true } ], "yaxis": { "align": false, "alignLevel": null } }, { "aliasColors": {}, "bars": false, "dashLength": 10, "dashes": false, "datasource": "${DS_PROM}", "fill": 1, "gridPos": { "h": 7, "w": 12, "x": 0, "y": 14 }, "id": 5, "legend": { "avg": false, "current": false, "max": false, "min": false, "show": true, "total": false, "values": false }, "lines": true, "linewidth": 1, "links": [], "nullPointMode": "null", "percentage": false, "pointradius": 5, "points": false, "renderer": "flot", "seriesOverrides": [], "spaceLength": 10, "stack": false, "steppedLine": false, "targets": [ { "expr": "rate(jvm_gc_collection_seconds_sum{kubernetes_pod_name=~\"$kubernetes_pod_name\"}[1m])", "format": "time_series", "interval": "", "intervalFactor": 5, "legendFormat": "{{gc}} [{{instance}}]", "metric": "jvm_gc_collection_seconds_sum", "refId": "A", "step": 10 } ], "thresholds": [], "timeFrom": null, "timeShift": null, "title": "GC time / 1 min. rate", "tooltip": { "shared": true, "sort": 0, "value_type": "individual" }, "type": "graph", "xaxis": { "buckets": null, "mode": "time", "name": null, "show": true, "values": [] }, "yaxes": [ { "format": "s", "label": null, "logBase": 1, "max": null, "min": null, "show": true }, { "format": "short", "label": null, "logBase": 1, "max": null, "min": null, "show": true } ], "yaxis": { "align": false, "alignLevel": null } }, { "aliasColors": {}, "bars": false, "dashLength": 10, "dashes": false, "datasource": "${DS_PROM}", "fill": 1, "gridPos": { "h": 7, "w": 12, "x": 12, "y": 14 }, "id": 6, "legend": { "avg": false, "current": false, "max": false, "min": false, "show": true, "total": false, "values": false }, "lines": true, "linewidth": 1, "links": [], "nullPointMode": "null", "percentage": false, "pointradius": 5, "points": false, "renderer": "flot", "seriesOverrides": [], "spaceLength": 10, "stack": false, "steppedLine": false, "targets": [ { "expr": "jvm_gc_collection_seconds_count{kubernetes_pod_name=~\"$kubernetes_pod_name\"}", "format": "time_series", "interval": "", "intervalFactor": 5, "legendFormat": "{{gc}} [{{instance}}]", "metric": "", "refId": "A", "step": 10 } ], "thresholds": [], "timeFrom": null, "timeShift": null, "title": "GC count", "tooltip": { "shared": true, "sort": 0, "value_type": "individual" }, "type": "graph", "xaxis": { "buckets": null, "mode": "time", "name": null, "show": true, "values": [] }, "yaxes": [ { "decimals": 0, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, "show": true }, { "format": "short", "label": null, "logBase": 1, "max": null, "min": null, "show": true } ], "yaxis": { "align": false, "alignLevel": null } } ], "refresh": "30s", "schemaVersion": 16, "style": "dark", "tags": [ "JVM", "prometheus", "jmx_exporter" ], "templating": { "list": [ { "allValue": "", "current": { "tags": [], "text": "All", "value": [ "$__all" ] }, "datasource": "${DS_PROM}", "hide": 0, "includeAll": true, "label": "kubernetes_pod_name", "multi": true, "name": "kubernetes_pod_name", "options": [], "query": "label_values(jvm_memory_bytes_max,kubernetes_pod_name)", "refresh": 1, "regex": "", "skipUrlSync": false, "sort": 0, "tagValuesQuery": "", "tags": [], "tagsQuery": "", "type": "query", "useTags": false } ] }, "time": { "from": "now-30m", "to": "now" }, "timepicker": { "refresh_intervals": [ "5s", "10s", "30s", "1m", "5m", "15m", "30m", "1h", "2h", "1d" ], "time_options": [ "5m", "15m", "1h", "6h", "12h", "24h", "2d", "7d", "30d" ] }, "timezone": "", "title": " tomcat JVM overview - Prometheus", "uid": "PVQFY0Pmz", "version": 2 }
在k8s集群中,利用prometheus的jmx_exporter进行tomcat的JVM性能监控,并用grafana作前端展示的更多相关文章
- k8s集群中部署prometheus server
1.概述 本文档主要介绍如何在k8s集群中部署prometheus server用来作为监控的数据采集服务器,这样做可以很方便的对k8s集群中的指标.pod的.节点的指标进行采集和监控. 2.下载镜像 ...
- 在 Nebula K8s 集群中使用 nebula-spark-connector 和 nebula-algorithm
本文首发于 Nebula Graph Community 公众号 解决思路 解决 K8s 部署 Nebula Graph 集群后连接不上集群问题最方便的方法是将 nebula-algorithm / ...
- 【K8S学习笔记】Part2:获取K8S集群中运行的所有容器镜像
本文将介绍如何使用kubectl列举K8S集群中运行的Pod内的容器镜像. 注意:本文针对K8S的版本号为v1.9,其他版本可能会有少许不同. 0x00 准备工作 需要有一个K8S集群,并且配置好了k ...
- k8s 集群中的etcd故障解决
一次在k8s集群中创建实例发现etcd集群状态出现连接失败状况,导致创建实例失败.于是排查了一下原因. 问题来源 下面是etcd集群健康状态: [root@docker01 ~]# cd /opt/k ...
- 将 master 节点服务器从 k8s 集群中移除并重新加入
背景 1 台 master 加入集群后发现忘了修改主机名,而在 k8s 集群中修改节点主机名非常麻烦,不如将 master 退出集群改名并重新加入集群(前提是用的是高可用集群). 操作步骤 ssh 登 ...
- k8s集群中遇到etcd集群故障的排查思路
一次在k8s集群中创建实例发现etcd集群状态出现连接失败状况,导致创建实例失败.于是排查了一下原因. 问题来源 下面是etcd集群健康状态: 1 2 3 4 5 6 7 8 9 10 11 [roo ...
- 实操教程丨如何在K8S集群中部署Traefik Ingress Controller
注:本文使用的Traefik为1.x的版本 在生产环境中,我们常常需要控制来自互联网的外部进入集群中,而这恰巧是Ingress的职责. Ingress的主要目的是将HTTP和HTTPS从集群外部暴露给 ...
- 终于解决 k8s 集群中部署 nodelocaldns 的问题
自从开始在 kubernetes 集群中部署 nodelocaldns 以提高 dns 解析性能以来,一直被一个问题困扰,只要一部署 nodelocaldns ,在 coredns 中添加的 rewr ...
- 在k8s集群中安装rook-ceph 1.8版本步骤
官方文档地址:https://rook.io/docs/rook/v1.8/quickstart.html Kubernetes 最小版本号 Kubernetes 最小版本号:Kubernetes v ...
随机推荐
- POJ 1182 食物链 (带权并查集)
食物链 Time Limit: 1000MS Memory Limit: 10000K Total Submissions: 78551 Accepted: 23406 Description ...
- tomcat 启用NIO
从Tomcat6.0以后, Java开发者很容易就可以是用NIO的技术来提升tomcat的并发处理能力. <Connector port="8080" protocol=&q ...
- 浅入 dancing links x(舞蹈链算法)
abastract:利用dancing links 解决精确覆盖问题,例如数独,n皇后问题:以及重复覆盖问题. 要学习dacning links 算法,首先要先了解该算法适用的问题,精确覆盖问题和重复 ...
- java生成棋盘
第一步:新建java项目,具体的命名,看下面的文件结构. 第二步:代码区 DrawChessBoard类: package com.hp.chenyanlong; import java.awt.Gr ...
- Kubernetes HPA
简介 通过手工执行 kubectl scale 命令或者通过修改deployment的replicas数量,可以实现 Pod 扩容或缩容.但如果仅止于此,显然不符合 Google 对 Kubernet ...
- Go_22: Golang 命令行 test 应用
1. 测试某一个包下的所有测试用例 cd /myGoProject/src/go-test/utils go test -v "-v" 参数 go test -v ... 表示无论 ...
- Swift学习笔记2
1.函数参数都有一个外部参数名(external parameter name)和一个局部参数名(local parameter name).外部参数名用于在函数调用时标注传递给函数的参数,局部参数名 ...
- Centos 7和 Centos 6开放查看端口 防火墙关闭打开
Centos 7 firewall 命令: 查看已经开放的端口: firewall-cmd --list-ports 开启端口 firewall-cmd --zone=public --add-por ...
- javascript构造函数模块
var Person = (function(){ var Constr; Constr = function(){ this.name = 'carl'; } Constr.prototype = ...
- asp.net分页之AJAX 分页
查询功能是开发中最重要的一个功能,大量数据的显示,我们用的最多的就是分页. 在ASP.NET 中有很多数据展现的控件,比如Repeater.GridView,用的最多的GridView,它同时也自带了 ...