Preface
 
    We have a lot of methods to diagnose problems in our system such as strace,pstack,gstack,gdb,pt-pmp,etc.But sometimes there will be some fitful performance issues which are not so easy to trace.Thus,pt-stalk may help us in diagnosing these kind of problems.
 
Introduce
 
    pt-stalk is a tool to collect detail diagnostic data base on triggers you specified such as gdb,oprofile,strace,tcpdump.The trigger is not the conception of trigger in database.They're different at all.pt-stalk provides various options to collect comprehensive data you need.It's really useful and helpful in performance diagnosing.Let's see some details of it.
 
Procedure
 
Usage
 pt-stalk [OPTIONS]
Common Parameters
 --collect //Collect diagostic data based on spedified triggers.
--collect-gdb //Collect gdb data.It prints stack traces from all threads.
--collect-oprofile //Collect oprofile data.
--collect-strace //Collect strace data.Do not specify it when "--collect-gdb" has been used.
--collect-tcpdump //Collect tcpdump data.
--cycles //Limite the times when triggering condition occurs(default "5").
--dest //Specify the position to store diagnostic data(default "/var/lib/pt-stalk").
--disk-bytes-free //Prevent the lack of least disk space.It does not collect data until it has the necessary disk space(default "100M",valid surffixes of unit is k,M,G and T).
--disk-pct-free //It's similar with "--disk-bytes-free" but specify the percentage of disk space.
--function //Specify the contents to watch for the triggers(default "status",other value is "processlist" or <yourfilename>).
--iterations //Limit the times to collect.It will run forever if not give a specific value.
--log //Specify the postion to record logs(default "/var/log/pt-stalk.log").It only generates when deamonized.
--match //The pattern to specify when using "--function processlist".
--mysql-only //Merely collect the MySQL relevant diagnostic data but disk space is the exception.
--retention-time //Specify the purge days of diagnostic data(default "30").
--run-time //Specify how many seconds to collect diagnostic data(default "30").It should not be longer than the value of "--sleep".
--sleep //Specify how many seconds to sleep after last collection.It used to prevent too many operations of collection(default "300").
--stalk //Watch and wait for trigger occur(default "yes").
--no_stalk //The option can be specify if you want to collect diagnostic data immediately without waiting for trigger to occur.
--threshold //Specify the maximum value for collecting(default "25").
--variable //Specify the variables to compared against(default "Threads_running").
Examples
 
Generate benchmark with sysbench.
 [root@zlm2 :: ~/sysbench-1.0/src/lua]
#sysbench oltp_read_write.lua --mysql-host=192.168.1.101 --mysql-port= --mysql-user=zlm --mysql-password=aaron8219 --mysql-db=sysbench --tables= --table-size= --mysql-storage-engine=innodb cleanup
sysbench 1.0. (using bundled LuaJIT 2.1.-beta2) Dropping table 'sbtest1'...
Dropping table 'sbtest2'...
Dropping table 'sbtest3'...
Dropping table 'sbtest4'...
Dropping table 'sbtest5'...
Dropping table 'sbtest6'...
Dropping table 'sbtest7'...
Dropping table 'sbtest8'...
Dropping table 'sbtest9'...
Dropping table 'sbtest10'... [root@zlm2 :: ~/sysbench-1.0/src/lua]
#sysbench oltp_read_write.lua --mysql-host=192.168.1.101 --mysql-port= --mysql-user=zlm --mysql-password=aaron8219 --mysql-db=sysbench --tables= --table-size= --mysql-storage-engine=innodb prepare
sysbench 1.0. (using bundled LuaJIT 2.1.-beta2) Creating table 'sbtest1'...
Inserting records into 'sbtest1'
Creating a secondary index on 'sbtest1'...
Creating table 'sbtest2'...
Inserting records into 'sbtest2'
Creating a secondary index on 'sbtest2'...
Creating table 'sbtest3'...
Inserting records into 'sbtest3'
Creating a secondary index on 'sbtest3'...
Creating table 'sbtest4'...
Inserting records into 'sbtest4'
Creating a secondary index on 'sbtest4'...
Creating table 'sbtest5'...
Inserting records into 'sbtest5'
Creating a secondary index on 'sbtest5'...
Creating table 'sbtest6'...
Inserting records into 'sbtest6'
Creating a secondary index on 'sbtest6'...
Creating table 'sbtest7'...
Inserting records into 'sbtest7'
Creating a secondary index on 'sbtest7'...
Creating table 'sbtest8'...
Inserting records into 'sbtest8'
Creating a secondary index on 'sbtest8'...
Creating table 'sbtest9'...
Inserting records into 'sbtest9'
Creating a secondary index on 'sbtest9'...
Creating table 'sbtest10'...
Inserting records into 'sbtest10'
Creating a secondary index on 'sbtest10'... [root@zlm2 :: ~/sysbench-1.0/src/lua]
#sysbench oltp_read_write.lua --mysql-host=192.168.1.101 --mysql-port= --mysql-user=zlm --mysql-password=aaron8219 --mysql-db=sysbench --threads= --time= --report-interval= --rand-type=uniform run
sysbench 1.0. (using bundled LuaJIT 2.1.-beta2) Running the test with following options:
Number of threads:
Report intermediate results every second(s)
Initializing random number generator from current time Initializing worker threads... Threads started! [ 10s ] thds: tps: 258.68 qps: 5176.49 (r/w/o: 3624.11/1034.82/517.56) lat (ms,%): 15.83 err/s: 0.00 reconn/s: 0.00
[ 20s ] thds: tps: 286.25 qps: 5726.15 (r/w/o: 4008.67/1144.89/572.60) lat (ms,%): 15.00 err/s: 0.00 reconn/s: 0.00
[ 30s ] thds: tps: 270.82 qps: 5416.13 (r/w/o: 3790.80/1083.69/541.64) lat (ms,%): 16.12 err/s: 0.00 reconn/s: 0.00
[ 40s ] thds: tps: 280.98 qps: 5619.75 (r/w/o: 3934.26/1123.53/561.97) lat (ms,%): 16.12 err/s: 0.00 reconn/s: 0.00
[ 50s ] thds: tps: 298.42 qps: 5968.38 (r/w/o: 4177.83/1193.70/596.85) lat (ms,%): 14.46 err/s: 0.00 reconn/s: 0.00
[ 60s ] thds: tps: 278.88 qps: 5578.20 (r/w/o: 3904.92/1115.52/557.76) lat (ms,%): 15.83 err/s: 0.00 reconn/s: 0.00
[ 70s ] thds: tps: 280.91 qps: 5617.66 (r/w/o: 3932.21/1123.63/561.82) lat (ms,%): 15.83 err/s: 0.00 reconn/s: 0.00
[ 80s ] thds: tps: 281.68 qps: 5632.83 (r/w/o: 3942.77/1126.71/563.35) lat (ms,%): 16.12 err/s: 0.00 reconn/s: 0.00
[ 90s ] thds: tps: 281.60 qps: 5631.55 (r/w/o: 3942.07/1126.39/563.10) lat (ms,%): 16.12 err/s: 0.00 reconn/s: 0.00
[ 100s ] thds: tps: 287.62 qps: 5753.86 (r/w/o: 4028.02/1150.49/575.35) lat (ms,%): 15.55 err/s: 0.00 reconn/s: 0.00
[ 110s ] thds: tps: 308.99 qps: 6180.45 (r/w/o: 4326.12/1236.35/617.97) lat (ms,%): 13.95 err/s: 0.00 reconn/s: 0.00
... //Omitted.
Collect diagnostic data using pt-stalk.
 [root@zlm2 :: /data/mysql/mysql3306]
#pt-stalk --host localhost --port --user root --password Passw0rd --collect-gdb --cycles --variable Threads_connect --threshold
mysql: [Warning] Using a password on the command line interface can be insecure.
2018_07_09_08_15_46 Starting /usr/bin/pt-stalk --function=status --variable=Threads_connect --threshold= --match= --cycles= --interval= --iterations= --run-time= --sleep= --dest=/var/lib/pt-stalk --prefix= --notify-by-email= --log=/var/log/pt-stalk.log --pid=/var/run/pt-stalk.pid --plugin=
mysqladmin: [Warning] Using a password on the command line interface can be insecure.
2018_07_09_08_15_46 Detected value is empty; something failed? Trigger exit status:
mysqladmin: [Warning] Using a password on the command line interface can be insecure.
2018_07_09_08_15_47 Detected value is empty; something failed? Trigger exit status:
mysqladmin: [Warning] Using a password on the command line interface can be insecure.
2018_07_09_08_15_48 Detected value is empty; something failed? Trigger exit status:
mysqladmin: [Warning] Using a password on the command line interface can be insecure.
2018_07_09_08_15_50 Detected value is empty; something failed? Trigger exit status:
mysqladmin: [Warning] Using a password on the command line interface can be insecure.
2018_07_09_08_15_51 Detected value is empty; something failed? Trigger exit status:
mysqladmin: [Warning] Using a password on the command line interface can be insecure.
2018_07_09_08_15_52 Detected value is empty; something failed? Trigger exit status:
mysqladmin: [Warning] Using a password on the command line interface can be insecure.
2018_07_09_08_15_53 Detected value is empty; something failed? Trigger exit status:
mysqladmin: [Warning] Using a password on the command line interface can be insecure.
2018_07_09_08_15_54 Detected value is empty; something failed? Trigger exit status:
mysqladmin: [Warning] Using a password on the command line interface can be insecure.
2018_07_09_08_15_55 Detected value is empty; something failed? Trigger exit status:
mysqladmin: [Warning] Using a password on the command line interface can be insecure.
2018_07_09_08_15_56 Detected value is empty; something failed? Trigger exit status:
^C2018_07_09_08_15_57 Caught signal, exiting
2018_07_09_08_15_57 All subprocesses have finished
2018_07_09_08_15_57 Exiting because OKTORUN is false
2018_07_09_08_15_57 /usr/bin/pt-stalk exit status //The value of "--variable" should be "Threads_connected".
//Modify the correct value and run it again. [root@zlm2 :: /data/mysql/mysql3306]
#pt-stalk --host localhost --port --user root --password Passw0rd --collect-gdb --cycles --variable Threads_connected --threshold
mysql: [Warning] Using a password on the command line interface can be insecure.
2018_07_09_08_19_39 Starting /usr/bin/pt-stalk --function=status --variable=Threads_connected --threshold= --match= --cycles= --interval= --iterations= --run-time= --sleep= --dest=/var/lib/pt-stalk --prefix= --notify-by-email= --log=/var/log/pt-stalk.log --pid=/var/run/pt-stalk.pid --plugin=
mysqladmin: [Warning] Using a password on the command line interface can be insecure.
2018_07_09_08_19_39 Check results: status(Threads_connected)=, matched=yes, cycles_true=
2018_07_09_08_19_39 Collect triggered
2018_07_09_08_19_39 Collect PID
2018_07_09_08_19_39 Collect done
2018_07_09_08_19_39 Sleeping seconds after collect
mysqladmin: [Warning] Using a password on the command line interface can be insecure.
2018_07_09_08_24_39 Check results: status(Threads_connected)=, matched=yes, cycles_true=
2018_07_09_08_24_39 Collect triggered
2018_07_09_08_24_39 Collect PID
2018_07_09_08_24_39 Collect done
2018_07_09_08_24_40 Sleeping seconds after collect
^C2018_07_09_08_25_56 Caught signal, exiting //Terminate to collect diagnostic data by "Ctrl+C"
2018_07_09_08_25_56 Waiting up to seconds for subprocesses to finish...
2018_07_09_08_25_56 Exiting because OKTORUN is false
2018_07_09_08_25_56 /usr/bin/pt-stalk exit status
Check the value of "show global status ... ".
 (root@localhost mysql3306.sock)[sysbench]>show global status like '%Threads%';
+------------------------+-------+
| Variable_name | Value |
+------------------------+-------+
| Delayed_insert_threads | |
| Slow_launch_threads | |
| Threads_cached | |
| Threads_connected | | //Threads_connected has been up to 5.
| Threads_created | |
| Threads_running | |
+------------------------+-------+
rows in set (0.04 sec) (root@localhost mysql3306.sock)[sysbench]>show global status like '%Threads%';
+------------------------+-------+
| Variable_name | Value |
+------------------------+-------+
| Delayed_insert_threads | |
| Slow_launch_threads | |
| Threads_cached | |
| Threads_connected | | //Second time,Threads_connected has been up to 6.
| Threads_created | |
| Threads_running | |
+------------------------+-------+
rows in set (0.01 sec)
Check the output files in default directory("/var/lib/pt-stalk").
 [root@zlm2 :: /var/lib/pt-stalk]
#ls -lrt
total
-rw-r--r-- root root Jul : 2018_07_09_08_19_39-trigger
-rw-r--r-- root root Jul : 2018_07_09_08_19_39-pmap
-rw-r--r-- root root Jul : 2018_07_09_08_19_39-variables
-rw-r--r-- root root Jul : 2018_07_09_08_19_39-innodbstatus1
-rw-r--r-- root root Jul : 2018_07_09_08_19_39-mutex-status1
-rw-r--r-- root root Jul : 2018_07_09_08_19_39-ps
-rw-r--r-- root root Jul : 2018_07_09_08_19_39-lsof
-rw-r--r-- root root Jul : 2018_07_09_08_19_39-opentables1
-rw-r--r-- root root Jul : 2018_07_09_08_19_39-top
-rw-r--r-- root root Jul : 2018_07_09_08_19_39-sysctl
-rw-r--r-- root root Jul : 2018_07_09_08_19_39-disk-space
-rw-r--r-- root root Jul : 2018_07_09_08_19_39-mysqladmin
-rw-r--r-- root root Jul : 2018_07_09_08_19_39-vmstat
-rw-r--r-- root root Jul : 2018_07_09_08_19_39-procstat
-rw-r--r-- root root Jul : 2018_07_09_08_19_39-diskstats
-rw-r--r-- root root Jul : 2018_07_09_08_19_39-procvmstat
-rw-r--r-- root root Jul : 2018_07_09_08_19_39-netstat_s
-rw-r--r-- root root Jul : 2018_07_09_08_19_39-slabinfo
-rw-r--r-- root root Jul : 2018_07_09_08_19_39-interrupts
-rw-r--r-- root root Jul : 2018_07_09_08_19_39-netstat
-rw-r--r-- root root Jul : 2018_07_09_08_19_39-meminfo
-rw-r--r-- root root Jul : 2018_07_09_08_19_39-df
-rw-r--r-- root root Jul : 2018_07_09_08_19_39-processlist
-rw-r--r-- root root Jul : 2018_07_09_08_19_39-innodbstatus2
-rw-r--r-- root root Jul : 2018_07_09_08_19_39-transactions
-rw-r--r-- root root Jul : 2018_07_09_08_19_39-hostname
-rw-r--r-- root root Jul : 2018_07_09_08_19_39-mutex-status2
-rw-r--r-- root root Jul : 2018_07_09_08_19_39-opentables2
-rw-r--r-- root root Jul : 2018_07_09_08_19_39-vmstat-overall
-rw-r--r-- root root Jul : 2018_07_09_08_19_39-output //There're a series of files(starts with "tigger" and ends with "output") each time we collect the diagnostic data. -rw-r--r-- root root Jul : 2018_07_09_08_24_39-trigger
-rw-r--r-- root root Jul : 2018_07_09_08_24_39-pmap
-rw-r--r-- root root Jul : 2018_07_09_08_24_39-variables
-rw-r--r-- root root Jul : 2018_07_09_08_24_39-innodbstatus1
-rw-r--r-- root root Jul : 2018_07_09_08_24_39-ps
-rw-r--r-- root root Jul : 2018_07_09_08_24_39-mutex-status1
-rw-r--r-- root root Jul : 2018_07_09_08_24_39-sysctl
-rw-r--r-- root root Jul : 2018_07_09_08_24_39-lsof
-rw-r--r-- root root Jul : 2018_07_09_08_24_39-opentables1
-rw-r--r-- root root Jul : 2018_07_09_08_24_39-top
-rw-r--r-- root root Jul : 2018_07_09_08_24_39-disk-space
-rw-r--r-- root root Jul : 2018_07_09_08_24_39-vmstat
-rw-r--r-- root root Jul : 2018_07_09_08_24_39-mysqladmin
-rw-r--r-- root root Jul : 2018_07_09_08_24_39-procstat
-rw-r--r-- root root Jul : 2018_07_09_08_24_39-netstat
-rw-r--r-- root root Jul : 2018_07_09_08_24_39-slabinfo
-rw-r--r-- root root Jul : 2018_07_09_08_24_39-netstat_s
-rw-r--r-- root root Jul : 2018_07_09_08_24_39-interrupts
-rw-r--r-- root root Jul : 2018_07_09_08_24_39-meminfo
-rw-r--r-- root root Jul : 2018_07_09_08_24_39-diskstats
-rw-r--r-- root root Jul : 2018_07_09_08_24_39-df
-rw-r--r-- root root Jul : 2018_07_09_08_24_39-procvmstat
-rw-r--r-- root root Jul : 2018_07_09_08_24_39-processlist
-rw-r--r-- root root Jul : 2018_07_09_08_24_39-innodbstatus2
-rw-r--r-- root root Jul : 2018_07_09_08_24_39-transactions
-rw-r--r-- root root Jul : 2018_07_09_08_24_39-hostname
-rw-r--r-- root root Jul : 2018_07_09_08_24_39-mutex-status2
-rw-r--r-- root root Jul : 2018_07_09_08_24_39-opentables2
-rw-r--r-- root root Jul : 2018_07_09_08_24_39-vmstat-overall
-rw-r--r-- root root Jul : 2018_07_09_08_24_39-output [root@zlm2 :: /var/lib/pt-stalk]
#cat 2018_07_09_08_24_39-trigger
2018_07_09_08_24_39 Check results: status(Threads_connected)=, matched=yes, cycles_true=
2018_07_09_08_24_39 pt-stalk ran with --function=status --variable=Threads_connected --threshold= --match= --cycles= --interval= --iterations= --run-time= --sleep= --dest=/var/lib/pt-stalk --prefix= --notify-by-email= --log=/var/log/pt-stalk.log --pid=/var/run/pt-stalk.pid --plugin= //The trigger file records the options we have used. [root@zlm2 :: /var/lib/pt-stalk]
#cat 2018_07_09_08_24_39-vmstat-overall
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st //The vmstat-overall records the vmstat output.
//each file contains the relevant diagnostic data about what their name called.I'm not going to demonstrate all of them.
Take care of you remain disk space,pt-stalk won't run anymore if you're out of space.
 [root@zlm2 :: /data/mysql/mysql3306]
#pt-stalk --host localhost --port --user root --password Passw0rd --collect --cycles --variable Threads_connected --threshold --sleep --demonize
Cannot open /tmp/pt-stalk..FSboRq/po/daemonize: No space left on device at -e line , <$fh> chunk .
No long attribute in option spec /tmp/pt-stalk..FSboRq/po/ask-pass [root@zlm2 :: /data/mysql/mysql3306]
#df -h
]Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos-root .4G .4G 20K % / //The root of linux has been out of space.
devtmpfs 488M 488M % /dev
tmpfs 497M 497M % /dev/shm
tmpfs 497M 6.6M 491M % /run
tmpfs 497M 497M % /sys/fs/cgroup
/dev/sda1 497M 118M 379M % /boot
none 87G 81G .9G % /vagrant
Summary
  • pt-stalk is another tool in diagnosing system performance and similar with pt-pmp but not the same.
  • pt-stalk will generate many statistic files which can really help you collecting information in almost every aspect.
  • pt-stalk has four main triggers:gdb,oprofile,strace,tcpdump.It's flexible to collect data with diffrent dimensionality.
 

Percona-Tookit工具包之pt-stalk的更多相关文章

  1. Linux后台开发工具箱

    https://files-cdn.cnblogs.com/files/aquester/Linux后台开发工具箱.pdf 目录 目录 1 1. 前言 3 2. 脚本类工具 3 2.1. sed命令- ...

  2. Mysql: pt-table-checksum 和 pt-table-sync 检查主从一致性,实验过程

    一.安装 percona 包 1.安装仓库的包 https://www.percona.com/doc/percona-repo-config/yum-repo.html sudo yum insta ...

  3. Linux后台开发工具箱-葵花宝典

    Linux后台开发工具箱-葵花宝典 一见 2016/11/4 目录 目录 1 1. 前言 4 2. 脚本类工具 4 2.1. 双引号和单引号 4 2.2. 取脚本完整文件路径 5 2.3. 环境变量和 ...

  4. 推荐几款MySQL相关工具

    前言: 随着互联网技术的不断发展, MySQL 相关生态也越来越完善,越来越多的工具涌现出来.一些公司或个人纷纷开源出一些不错的工具,本篇文章主要介绍几款 MySQL 相关实用工具.提醒下,这里并不介 ...

  5. [知识库分享系列] 二、.NET(ASP.NET)

    最近时间又有了新的想法,当我用新的眼光在整理一些很老的知识库时,发现很多东西都已经过时,或者是很基础很零碎的知识点.如果分享出去大家不看倒好,更担心的是会误人子弟,但为了保证此系列的完整,还是选择分享 ...

  6. SQL慢查询安装过程

    SQL慢查询 基本操作 打开防火墙 firewall-cmd --zone=public --add-port=3306/tcp --permanent firewall-cmd --reload 安 ...

  7. 快速安装Percona pt工具

    yum install perl-DBI perl-DBD-MySQL perl-Time-HiRes perl-Time-HiRes perl-IO-Socket-SSLwget http://pk ...

  8. Percona 工具包 pt-online-schema-change 简介

    mysql的在线表结构修改,因为低效和阻塞读写.一直被诟病.至于ALTER TABLE 的原理,参看我上一篇文章.MySQL在线修改大表结构.看完后,发现的问题是还是会锁的,且对于在线更新的这块也是不 ...

  9. 安装percona工具包

    1.安装percona源 sudo yum install http://www.percona.com/downloads/percona-release/redhat/0.1-4/percona- ...

  10. percona pt toolkit 总结

    ##=====================================================##pt-osc之工作流程:1.检查更改表是否有主键或唯一索引,是否有触发器2.检查修改表 ...

随机推荐

  1. Hadoop2.X分布式集群部署

    本博文集群搭建没有实现Hadoop HA,详细文档在后续给出,本次只是先给出大概逻辑思路. (一)hadoop2.x版本下载及安装 Hadoop 版本选择目前主要基于三个厂商(国外)如下所示: 基于A ...

  2. eclipse中使用自带的git提交项目

    1.自带git插件进行配置我们的用户名和密码,即是自己github注册用户.windows--perferences--Team--Git--Configuration 2.eclipse生成SSH2 ...

  3. 【密码学】RSA密钥长度、明文长度和密文长度

    本文介绍RSA加解密中必须考虑到的密钥长度.明文长度和密文长度问题,对第一次接触RSA的开发人员来说,RSA算是比较复杂的算法,天缘以后还会补充几篇RSA基础知识专题文章,用最简单最通俗的语言描述RS ...

  4. for循环笔记

    JS获取元素方法——ById和ByTagName方法的区别 1.通过id获取,前面就只能是document,不能是其他的,但是ByTagName前面可以是document,也可以跟一个别的元素 #li ...

  5. 连接字符串(web.config)

    data source=ip; initial catalog=db1; user id=sa; password=*** <connectionStrings> <add name ...

  6. poj 1987 节点距离小于等于K(树DP)

    这题和poj 1741是一模一样的 但是1741能AC的代码,在这里却是TLE,暂时没看出哪里出现了问题.. AC代码: #include <iostream> #include < ...

  7. 基于百度ai,图灵机器人,Flask 实现的网站语音智能问答

    准备以下模块中的函数 from aip import AipSpeech import time import os import requests APP_ID = '15420654' API_K ...

  8. array_map()关于回调函数的总结

    array_map()函数的第一个参数可以是匿名函数,系统函数,也可以是自己自定义的函!在全局空间下,这些函数的调用都很简单 在这里就不多说了!我们主要讨论的是该函数调用类中的方法和静态方法的区别 摘 ...

  9. Struts2_Path

    路径问题说明: struts2中的路径问题是根据action的路径而不是jsp路径来确定,所以尽量不要使用相对路径.index.jsp虽然可以用rederect方式解决,但redirect方式并非必要 ...

  10. lucene查询解析器语法

    注意:使用QueryParser查询,关键词是会被分词的,如果不需要分词,可以选择使用Lucene提供的API查询类. Lucene提供了丰富的API来组合定制你所需要的查询器,同时也可以利用Quer ...