在k8s集群中,利用prometheus的jmx_exporter进行tomcat的JVM性能监控,并用grafana作前端展示
查找了很多文档,没有完全达到我要求的,
于是,作了一定的调整,成现在这样。
操作步骤如下:
一,准备好两个文件。
jmx_prometheus_javaagent-0.3.1.jar
jmx_exporter.yml
jmx_exporter.yml作最简单配置如下:
--- rules: - pattern: ".*"
二,tomcat的启动项,一直在bin下单独的setenv.sh里操作,新增如下启动选项,将jar包及yaml加入启动选项。
JAVA_OPTS="$JAVA_OPTS -javaagent:/xxx/bin/jmx_prometheus_javaagent-0.3.1.jar=1234:/xxx/bin/jmx_exporter.yml"
三,制作dockerfile时,要将这两个东东也CP到上面指定目录,并EXPOSE一个新的端口。
... COPY --chown=a:a tomcat /xxx/tomcat COPY --chown=a:a jdk /xxx/jdk COPY --chown=a:a jmx_prometheus_javaagent-0.3.1.jar /xxx/tomcat/bin/ COPY --chown=a:a jmx_exporter.yml /xxx/tomcat/bin/ WORKDIR ${CATALINA_HOME} USER a EXPOSE 8080 1234
四,更新prometheus的配置文件configmap,新增一个job。
- job_name: 'tomcat-pods' tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs: - role: endpoints relabel_configs: - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape, __meta_kubernetes_service_annotation_prometheus_io_jvm_scrape] regex: true;true action: keep - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_app_metrics_patn] action: replace target_label: __metrics_path__ regex: (.+) - source_labels: [__meta_kubernetes_pod_ip, __meta_kubernetes_service_annotation_prometheus_io_app_metrics_port] action: replace target_label: __address__ regex: (.+);(.+) - source_labels: [__meta_kubernetes_namespace] action: replace target_label: kubernetes_namespace - source_labels: [__meta_kubernetes_pod_name] action: replace target_label: kubernetes_pod_name
五,在对应的需要监控JVM的应用时,新增一个SERVICE。
apiVersion: v1 kind: Service metadata: labels: name: xxx name: xxx namespace: default annotations: prometheus.io/scrape: "true" prometheus.io/jvm-scrape: "true" prometheus.io/jvm-port: "1234" prometheus.io/jvm-path: "/metrics" spec: ports: - port: 1234 targetPort: 1234 selector: name: xxx
六,最后,导入grafana的插件文件。
{ "__inputs": [ { "name": "DS_PROM", "label": "prom", "description": "A prometheus datasource with JMX Exporter scraping", "type": "datasource", "pluginId": "prometheus", "pluginName": "Prometheus" } ], "__requires": [ { "type": "grafana", "id": "grafana", "name": "Grafana", "version": "4.5.0-pre1" }, { "type": "panel", "id": "graph", "name": "Graph", "version": "" }, { "type": "datasource", "id": "prometheus", "name": "Prometheus", "version": "1.0.0" } ], "annotations": { "list": [] }, "description": " tomcat JVM Dashboard for JVM metrics with Prometheus / JMX Exporter", "editable": true, "gnetId": 3066, "graphTooltip": 0, "id": 10, "iteration": 1544507474291, "links": [], "panels": [ { "aliasColors": {}, "bars": false, "dashLength": 10, "dashes": false, "datasource": "${DS_PROM}", "fill": 1, "gridPos": { "h": 7, "w": 24, "x": 0, "y": 0 }, "id": 2, "legend": { "alignAsTable": true, "avg": true, "current": true, "max": true, "min": true, "rightSide": true, "show": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], "nullPointMode": "null", "percentage": false, "pointradius": 5, "points": false, "renderer": "flot", "repeat": null, "seriesOverrides": [], "spaceLength": 10, "stack": false, "steppedLine": false, "targets": [ { "expr": "jvm_memory_bytes_used{kubernetes_pod_name=~\"$kubernetes_pod_name\"}", "format": "time_series", "interval": "", "intervalFactor": 5, "legendFormat": "{{area}} memory [{{instance}}]", "metric": "jvm_memory_bytes_used", "refId": "A", "step": 5 } ], "thresholds": [], "timeFrom": null, "timeShift": null, "title": "Memory used", "tooltip": { "shared": true, "sort": 0, "value_type": "individual" }, "type": "graph", "xaxis": { "buckets": null, "mode": "time", "name": null, "show": true, "values": [] }, "yaxes": [ { "format": "bytes", "label": null, "logBase": 1, "max": null, "min": null, "show": true }, { "format": "short", "label": null, "logBase": 1, "max": null, "min": null, "show": true } ], "yaxis": { "align": false, "alignLevel": null } }, { "aliasColors": {}, "bars": false, "dashLength": 10, "dashes": false, "datasource": "${DS_PROM}", "fill": 1, "gridPos": { "h": 7, "w": 12, "x": 0, "y": 7 }, "id": 3, "legend": { "avg": false, "current": false, "max": false, "min": false, "show": true, "total": false, "values": false }, "lines": true, "linewidth": 1, "links": [], "nullPointMode": "null", "percentage": false, "pointradius": 5, "points": false, "renderer": "flot", "seriesOverrides": [], "spaceLength": 10, "stack": false, "steppedLine": false, "targets": [ { "expr": "jvm_threads_current{kubernetes_pod_name=~\"$kubernetes_pod_name\"}", "format": "time_series", "intervalFactor": 5, "legendFormat": "current [{{instance}}]", "metric": "jvm_threads_current", "refId": "A", "step": 10 }, { "expr": "jvm_threads_daemon{kubernetes_pod_name=~\"$kubernetes_pod_name\"}", "format": "time_series", "intervalFactor": 5, "legendFormat": "daemon [{{instance}}]", "metric": "jvm_threads_daemon", "refId": "B", "step": 10 } ], "thresholds": [], "timeFrom": null, "timeShift": null, "title": "Threads used", "tooltip": { "shared": true, "sort": 0, "value_type": "individual" }, "type": "graph", "xaxis": { "buckets": null, "mode": "time", "name": null, "show": true, "values": [] }, "yaxes": [ { "decimals": 0, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, "show": true }, { "format": "short", "label": null, "logBase": 1, "max": null, "min": null, "show": true } ], "yaxis": { "align": false, "alignLevel": null } }, { "aliasColors": {}, "bars": false, "dashLength": 10, "dashes": false, "datasource": "${DS_PROM}", "fill": 1, "gridPos": { "h": 7, "w": 12, "x": 12, "y": 7 }, "id": 4, "legend": { "avg": false, "current": false, "max": false, "min": false, "show": true, "total": false, "values": false }, "lines": true, "linewidth": 1, "links": [], "nullPointMode": "null", "percentage": false, "pointradius": 5, "points": false, "renderer": "flot", "seriesOverrides": [], "spaceLength": 10, "stack": false, "steppedLine": false, "targets": [ { "expr": "jvm_classes_loaded{kubernetes_pod_name=~\"$kubernetes_pod_name\"}", "format": "time_series", "interval": "", "intervalFactor": 5, "legendFormat": "loaded [{{instance}}]", "metric": "jvm_classes_loaded", "refId": "A", "step": 10 } ], "thresholds": [], "timeFrom": null, "timeShift": null, "title": "Class loading", "tooltip": { "shared": true, "sort": 0, "value_type": "individual" }, "type": "graph", "xaxis": { "buckets": null, "mode": "time", "name": null, "show": true, "values": [] }, "yaxes": [ { "decimals": 0, "format": "short", "label": "", "logBase": 1, "max": null, "min": null, "show": true }, { "format": "short", "label": null, "logBase": 1, "max": null, "min": null, "show": true } ], "yaxis": { "align": false, "alignLevel": null } }, { "aliasColors": {}, "bars": false, "dashLength": 10, "dashes": false, "datasource": "${DS_PROM}", "fill": 1, "gridPos": { "h": 7, "w": 12, "x": 0, "y": 14 }, "id": 5, "legend": { "avg": false, "current": false, "max": false, "min": false, "show": true, "total": false, "values": false }, "lines": true, "linewidth": 1, "links": [], "nullPointMode": "null", "percentage": false, "pointradius": 5, "points": false, "renderer": "flot", "seriesOverrides": [], "spaceLength": 10, "stack": false, "steppedLine": false, "targets": [ { "expr": "rate(jvm_gc_collection_seconds_sum{kubernetes_pod_name=~\"$kubernetes_pod_name\"}[1m])", "format": "time_series", "interval": "", "intervalFactor": 5, "legendFormat": "{{gc}} [{{instance}}]", "metric": "jvm_gc_collection_seconds_sum", "refId": "A", "step": 10 } ], "thresholds": [], "timeFrom": null, "timeShift": null, "title": "GC time / 1 min. rate", "tooltip": { "shared": true, "sort": 0, "value_type": "individual" }, "type": "graph", "xaxis": { "buckets": null, "mode": "time", "name": null, "show": true, "values": [] }, "yaxes": [ { "format": "s", "label": null, "logBase": 1, "max": null, "min": null, "show": true }, { "format": "short", "label": null, "logBase": 1, "max": null, "min": null, "show": true } ], "yaxis": { "align": false, "alignLevel": null } }, { "aliasColors": {}, "bars": false, "dashLength": 10, "dashes": false, "datasource": "${DS_PROM}", "fill": 1, "gridPos": { "h": 7, "w": 12, "x": 12, "y": 14 }, "id": 6, "legend": { "avg": false, "current": false, "max": false, "min": false, "show": true, "total": false, "values": false }, "lines": true, "linewidth": 1, "links": [], "nullPointMode": "null", "percentage": false, "pointradius": 5, "points": false, "renderer": "flot", "seriesOverrides": [], "spaceLength": 10, "stack": false, "steppedLine": false, "targets": [ { "expr": "jvm_gc_collection_seconds_count{kubernetes_pod_name=~\"$kubernetes_pod_name\"}", "format": "time_series", "interval": "", "intervalFactor": 5, "legendFormat": "{{gc}} [{{instance}}]", "metric": "", "refId": "A", "step": 10 } ], "thresholds": [], "timeFrom": null, "timeShift": null, "title": "GC count", "tooltip": { "shared": true, "sort": 0, "value_type": "individual" }, "type": "graph", "xaxis": { "buckets": null, "mode": "time", "name": null, "show": true, "values": [] }, "yaxes": [ { "decimals": 0, "format": "short", "label": null, "logBase": 1, "max": null, "min": null, "show": true }, { "format": "short", "label": null, "logBase": 1, "max": null, "min": null, "show": true } ], "yaxis": { "align": false, "alignLevel": null } } ], "refresh": "30s", "schemaVersion": 16, "style": "dark", "tags": [ "JVM", "prometheus", "jmx_exporter" ], "templating": { "list": [ { "allValue": "", "current": { "tags": [], "text": "All", "value": [ "$__all" ] }, "datasource": "${DS_PROM}", "hide": 0, "includeAll": true, "label": "kubernetes_pod_name", "multi": true, "name": "kubernetes_pod_name", "options": [], "query": "label_values(jvm_memory_bytes_max,kubernetes_pod_name)", "refresh": 1, "regex": "", "skipUrlSync": false, "sort": 0, "tagValuesQuery": "", "tags": [], "tagsQuery": "", "type": "query", "useTags": false } ] }, "time": { "from": "now-30m", "to": "now" }, "timepicker": { "refresh_intervals": [ "5s", "10s", "30s", "1m", "5m", "15m", "30m", "1h", "2h", "1d" ], "time_options": [ "5m", "15m", "1h", "6h", "12h", "24h", "2d", "7d", "30d" ] }, "timezone": "", "title": " tomcat JVM overview - Prometheus", "uid": "PVQFY0Pmz", "version": 2 }
在k8s集群中,利用prometheus的jmx_exporter进行tomcat的JVM性能监控,并用grafana作前端展示的更多相关文章
- k8s集群中部署prometheus server
1.概述 本文档主要介绍如何在k8s集群中部署prometheus server用来作为监控的数据采集服务器,这样做可以很方便的对k8s集群中的指标.pod的.节点的指标进行采集和监控. 2.下载镜像 ...
- 在 Nebula K8s 集群中使用 nebula-spark-connector 和 nebula-algorithm
本文首发于 Nebula Graph Community 公众号 解决思路 解决 K8s 部署 Nebula Graph 集群后连接不上集群问题最方便的方法是将 nebula-algorithm / ...
- 【K8S学习笔记】Part2:获取K8S集群中运行的所有容器镜像
本文将介绍如何使用kubectl列举K8S集群中运行的Pod内的容器镜像. 注意:本文针对K8S的版本号为v1.9,其他版本可能会有少许不同. 0x00 准备工作 需要有一个K8S集群,并且配置好了k ...
- k8s 集群中的etcd故障解决
一次在k8s集群中创建实例发现etcd集群状态出现连接失败状况,导致创建实例失败.于是排查了一下原因. 问题来源 下面是etcd集群健康状态: [root@docker01 ~]# cd /opt/k ...
- 将 master 节点服务器从 k8s 集群中移除并重新加入
背景 1 台 master 加入集群后发现忘了修改主机名,而在 k8s 集群中修改节点主机名非常麻烦,不如将 master 退出集群改名并重新加入集群(前提是用的是高可用集群). 操作步骤 ssh 登 ...
- k8s集群中遇到etcd集群故障的排查思路
一次在k8s集群中创建实例发现etcd集群状态出现连接失败状况,导致创建实例失败.于是排查了一下原因. 问题来源 下面是etcd集群健康状态: 1 2 3 4 5 6 7 8 9 10 11 [roo ...
- 实操教程丨如何在K8S集群中部署Traefik Ingress Controller
注:本文使用的Traefik为1.x的版本 在生产环境中,我们常常需要控制来自互联网的外部进入集群中,而这恰巧是Ingress的职责. Ingress的主要目的是将HTTP和HTTPS从集群外部暴露给 ...
- 终于解决 k8s 集群中部署 nodelocaldns 的问题
自从开始在 kubernetes 集群中部署 nodelocaldns 以提高 dns 解析性能以来,一直被一个问题困扰,只要一部署 nodelocaldns ,在 coredns 中添加的 rewr ...
- 在k8s集群中安装rook-ceph 1.8版本步骤
官方文档地址:https://rook.io/docs/rook/v1.8/quickstart.html Kubernetes 最小版本号 Kubernetes 最小版本号:Kubernetes v ...
随机推荐
- BZOJ2721 Violet5樱花(数论)
有(x+y)n!=xy.套路地提出x和y的gcd,设为d,令ad=x,bd=y.则有(a+b)n!=abd.此时d已是和a.b无关的量.由a与b互质,得a+b与ab互质,于是将a+b除过来得n!=ab ...
- Simple Cycles Edges CodeForces - 962F(点双连通分量)
题意: 求出简单环的所有边,简单环即为边在一个环内 解析: 求出点双连通分量,如果一个连通分量的点数和边数相等,则为一个简单环 点双连通分量 任意两个点都至少存在两条点不重复的路径 即任意两条边都 ...
- Valid BFS? CodeForces - 1037D(思维 bfs)
我真是一只菜狗......emm... 题意: 判断一个从1开始的队列是否可以按照bfs的顺序 进行遍历..必须从1开始...然后后边依次是bfs顺序 解析: 看代码能看懂吧...emm...就是把每 ...
- 【BZOJ1449】[JSOI2009]球队收益(网络流,费用流)
[BZOJ1449][JSOI2009]球队收益(网络流,费用流) 题面 BZOJ 洛谷 题解 首先对于一支队伍而言,总共进行多少场比赛显然是已知的,假设是\(n_i\)场,那么它的贡献是:\(C_i ...
- Mask RCNN 原理
转自:https://blog.csdn.net/ghw15221836342/article/details/80084861 https://blog.csdn.net/ghw1522183634 ...
- Java基础-SSM之mybatis一对多和多对一关系映射
Java基础-SSM之mybatis一对多和多对一关系映射 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 一.准备测试环境(创建数据库表) 1>.创建customers表: ...
- Hadoop基础-MapReduce的Combiner用法案例
Hadoop基础-MapReduce的Combiner用法案例 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 一.编写年度最高气温统计 如上图说所示:有一个temp的文件,里面存放 ...
- bzoj千题计划292:bzoj2244: [SDOI2011]拦截导弹
http://www.lydsy.com/JudgeOnline/problem.php?id=2244 每枚导弹成功拦截的概率 = 包含它的最长上升子序列个数/最长上升子序列总个数 pre_len ...
- spark RDD 常见操作
fold 操作 区别 与 co 1.mapValus 2.flatMapValues 3.comineByKey 4.foldByKey 5.reduceByKey 6.groupByKey 7.so ...
- 《Two Dozen Short Lessons in Haskell》(二十四)代数类型
这是<Two Dozen Short Lessons in Haskell>这本书的最后一章,第23章没有习题. 这一章里介绍了Haskell如果自定义一种类型,并且用一个双人博弈游戏为例 ...