内核调试神器SystemTap — 探測点与语法（二）

a linux trace/probe tool.

官网：https://sourceware.org/systemtap/

探測点

SystemTap脚本主要是由探測点和探測点处理函数组成的，来看下都有哪些探測点可用。

The essential idea behind a systemtap script is to name events, and to give them handlers.

Systemtap works by translating the script to C, running the system C compiler to create a kernel module from that.

When the module is loaded, it activates all the probed events by hooking into the kernel.

(1) where to probe

Built-in events (probe point syntax and semantics)

begin：The startup of the systemtap session.

end：The end of the systemtap session.

kernel.function("sys_open")：The entry to the function named sys_open in the kernel.

syscall.close.return：The return from the close system call.

module("ext3").statement(0xdeadbeef)：The addressed instruction in the ext3 filesystem driver.

timer.ms(200)：A timer that fires every 200 milliseconds.

timer.jiffies(200)：A timer that fires every 200 jiffies.

timer.profile：A timer that fires periodically on every CPU.

perf.hw.cache_misses：A particular number of CPU cache misses have occurred.

procfs("status").read：A process trying to read a synthetic file.

process("a.out").statement("*@main.c:200")：Line 200 of the a.out program.

很多其它信息，可见stapprobes mannual page：

https://sourceware.org/systemtap/man/stapprobes.3stap.html

http://linux.die.net/man/5/stapprobes

(2) what to print

Systemtap provides a variety of such contextual data, ready for formatting.

The usually appear as function calls within the handler.

tid()：The id of the current thread.

pid()：The process (task group) id of the current thread.

uid()：The id of the current user.

execname()：The name of the current process.

cpu()：The current cpu number.

gettimeofday_s()：Number of seconds since epoch.

get_cycles()：Snapshot of hardware cycle counter.

pp()：A string describing the probe point being currently handled.

probefunc()：If known, the name of the function in which this probe was placed.

$$vars：If available, a pretty-printed listing of all local variables in scope.

print_backtrace()：If possible, print a kernel backtrace.

print_ubacktrace()：If possible, print a user-space backtrace.

$$parms：表示函数參数

$$return：表示函数返回值

thread_indent()：tapset libary中一个非常实用的函数，它的输出格式：

A timestamp (number of microseconds since the initial indentation for the thread)

A process name and the thread id itself.

很多其它信息，可见stapfuncs mannual page：

https://sourceware.org/systemtap/man/stapfuncs.3stap.html

http://linux.die.net/man/5/stapfuncs

(3) Built-in probe point types (DWARF probes)

内置的探測点，安装debuginfo后可使用。

This family of probe points uses symbolic debugging information for the target kernel or module,

as may be found in executables that have not been stripped, or in the separate debuginfo packages.

眼下支持的内置探測点类型：

kernel.function(PATTERN) // 在函数的入口处放置探測点，能够获取函数參数$PARM

kernel.function(PATTERN).return // 在函数的返回处放置探測点，能够获取函数的返回值$return，以及可能被改动的函数參数$PARM

kernel.function(PATTERN).call // 取补集，取不符合条件的函数

kernel.function(PATTERN).inline // 仅仅选择符合条件的内联函数。内联函数不能使用.return

kernel.function(PATTERN).exported // 仅仅选择导出的函数

module(MPATTERN).function(PATTERN)

module(MPATTERN).function(PATTERN).return

module(MPATTERN).function(PATTERN).call

module(MPATTERN).function(PATTERN).inline

kernel.statement(PATTERN)

kernel.statement(ADDRESS).absolute

module(MPATTERN).statement(PATTERN)

演示样例：

# Refers to all kernel functions with "init" or "exit" in the name

kernel.function("*init*"), kernel.function("*exit*")

# Refers to any functions within the "kernel/time.c" file that span line 240

kernel.function("*@kernel/time.c:240")

# Refers to all functions in the ext3 module

module("ext3").function("*")

# Refers to the statement at line 296 within the kernel/time.c file

kernel.statement("*@kernel/time.c:296")

# Refers to the statement at line bio_init+3 within the fs/bio.c file

kernel.statement("bio_init@fs/bio.c+3")

部分在编译单元内可见的源代码变量，比方函数參数、局部变量或全局变量，在探測点处理函数中相同是可见的。

在脚本中使用$加上变量的名字就能够饮用了。

变量的引用有两种风格：

$varname // 引用变量varname

$var->field // 引用结构的成员变量

$var[N] // 引用数组的元素

&$var // 变量的地址

@var("varname") // 引用变量varname

@var("var@src/file.c") // 引用src/file.c在被编译时的全局变量varname

@var("varname@file.c")->field // 引用结构的成员变量

@var("var@file.c")[N] // 引用数组的元素

&@var("var@file.c") // 变量的地址

$var$ // provide a string that includes the values of basic type values

$var$$ // provide a string that includes all values of nested data types

$$vars // 一个包括全部函数參数、局部变量的字符串

$$locals // 一个包括全部局部变量的字符串

$$params // 一个包括全部函数參数的字符串

(4) DWARF-less probing

当没有安装debuginfo时，不能使用内置的探測点。

In the absence of debugging information, you can still use the kprobe family of probes to examine the

entry and exit points of kernel and module functions. You cannot lookup the arguments or local variables

of a function using these probes.

当目标内核或模块缺少调试信息时，尽管不能使用内置的探測点，但仍然能够使用kprobe来探測函数的入口点

和退出点。此时不能使用“$+变量名”来获取函数參数或局部变量的值。

SystemTap仍然提供了一种訪问參数的方法：

当函数因被探測而停滞在它的进入点时，能够使用编号来引用它的參数。

比如。如果被探測的函数声明例如以下：

ssize_t sys_read(unsigned int fd, char __user *buf, size_t count)

能够分别使用unit_arg(1)、pointer_arg(2)、ulong_arg(3)来获取fd、buf和count的值。

此种探測点尽管不支持$return。但能够通过调用returnval()来获取寄存器的值。函数的返回值一般是保存在

这一寄存器里的，也能够调用returnstr()来获取返回值的字符串形式。

在处理函数代码里面。能够调用register("regname")来获取它被调用时特定CPU寄存器的值。

使用格式(不能用通配符)：

kprobe.function(FUNCTION)

kprobe.function(FUNCTION).return

kprobe.module(NAME).function(FUNCTION)

kprobe.module(NAME).function(FUNCTION).return

kprobe.statement(ADDRESS).absolute

语法

(1) 基本格式

probe probe-point probe- handler。即probe Probe-Point { statement }

用probe指定一个探測点(probe-point)。以及在这个探測点处运行的处理函数(probe-handler)。

每条语句不用结束符，分号“。”表示空语句。函数用{}括起来。

同意多种凝视语句：

Shell-stype：#

C-style：/* */

C++-style：//

next语句用于提前退出Probe-handler。

String连接符是“.”，比較符为“==”。

比如："hello" . "world" 。连接成"helloword"

变量属于弱数据类型，不用事先声明，不用指定数据类型。

字符串类型和数字类型的转换：

s = sprint(123) # s becomes the string "123"

probe-handler中定义的变量是局部的，不能在其他探測点处理函数中使用。

global符号用于定义全局变量。

Because of possible concurrency (multiple probe handlers running on different CPUs, each global variable

used by a probe is automatically read-locked or write-locked while the handler is running.

next语句：运行到next语句时，会立即从探測点处理函数中返回。

(2) 函数

function name(param1, param2)

{

statements

return ret

}

Recursion is possible, up to a nesting depth limit.

(3) 条件语句

if (EXPR) STATEMENT [else STATEMENT]

(4) 循环语句

while (EXPR) STATEMENT

for (A; B; C) STATEMENT

break能够提前退出循环，continue能够跳过本次循环。

(5) 上下文变量

Allow access to the probe point context. To know which variables are likely to be available, you will need to

be familiar with the kernel source you are probing.

You can use stap -L PROBEPOINT to enumerate the variables available there.

使用stap -L probe-point，来查看运行到这个探測点时，哪些上下文变量是可用的。

Two functions, user_string and kernel_string, can copy char *target variables into systemtap strings.

实例：

(6) 关联数组

These arrays are implemented as hash tables with a maximum size that is fixed at startup.

Because they are too large to be created dynamically for individual probes handler runs, they must be

declared as global.

关联数组是用哈希表实现的。最大大小在一開始就设定了。

关联数组必须是全局的。不能在探測点处理函数内部定义。

数组的索引最多能够有9个，用逗号隔开。能够是数字或字符串。

比如：global array[400]

6.1 数组

能够用多个索引来定位数组元素。

元素的数据类型有三种：数值、字符串、统计类型。

假设不指定数组的大小，那么默认设为最大值MAXMAPENTRIES(2048)。

比如：

foo[4, "hello"]++

processusage[uid(), execname()]++

6.2 元素是否存在

比如：if ([4, "hello"] in foo) { }

6.3 元素删除

比如：delete

delete times[tid()] # deletion of a single element

delete times # deletion of all elements

6.4 删除变量

比如：delete var

假设var是一个数值型变量，那么它被重置为0；假设var是一个字符串型变量，那么它被重置为""，

假设var是一个统计类型变量，那么它所在的集合被清空。

6.4 遍历

使用foreachkeyword。同意使用break/continue，在遍历期间不同意改动数组。

foreach (x = [a, b] in foo) { fuss_with(x) } # simple loop in arbitrary sequence

foreach ([a, b] in foo+ limit 5) {} # loop in increasing sequence of value, stop after 5

foreach ([a-, b] in foo) {} # loop in decreasing sequence of first key

# Print the first 10 tuples and values in the array in decreasing sequence

foreach(v = [i, j] in foo- limit 10)

printf("foo [%d, %s] = %d\n", i, j, v)

三中遍历形式：

foreach (VAR in ARRAY) STMT // 按值遍历，VAR为元素值

foreach ([VAR1, VAR2, ...] in ARRAY) STMT // 按索引遍历

foreach (VAR = [VAR1, VAR2, ...] in ARRAY) STMT // 同一时候得到元素值和元素索引

6.5 覆盖

%表示当数组容量不够时，同意新的元素覆盖掉旧的元素。

global ARRAY%[<size>], ARRAY2%

(7) 统计类型

statistics aggregates是SystemTap特有的数据类型。用于统计全局变量。

操作符为“<<<”

比如：g_value <<< b # 相当于C语言的g_value += b

这样的变量仅仅能用特定函数操作。主要包含：

@count(g_value)：全部统计操作的操作次数

@sum(g_value)：全部统计操作的操作数的总和

@min(g_value)：全部统计操作的操作数的最小值

@max(g_value)：全部统计操作的操作数的最大值

@avg(g_value)：全部统计操作的操作数的平均值

(8) 语言安全性

8.1 时间限制

探測点处理函数是有运行时间限制的，不能占用太多时间，否则SystemTap在把脚本编译为C语言时会报错。

每一个探測点处理函数仅仅能运行1000条语句。这个数量是可配置的。

8.2 动态内存分配

探測点处理函数中不同意动态内存分配。

No dynamic memory allocation whatsoever takes place during the execution of probe handlers.

Arrays, function contexts, and buffers are allocated during initialization.

8.3 锁

多个探測点处理函数抢占一个全局变量锁时，某几个探測点处理函数可能会超时，被放弃运行。

訪问全局变量时会加锁，防止它被并发的改动。

If multiple probes seek conflicting locks on the same global variables, one or more of them will time out and be

aborted. Such events are tailed as skipped probes, and a count is displayed at session end.

8.4 bug

内核中少数对时间很敏感的地方(上下文切换、中断处理)。是不能设为探測点的。

Putting probes indiscriminately into unusually sensitive parts of the kernel (low level context switching, interrupt

dispatching) has reportedly caused crashes in the past. We are fixing these bugs as they are found, and

constructing a probe "blacklist", but it is not complete.

8.5 改动限制

通过-D选项能够改动默认的一些限制。

-D NM=VAL emit macro definition into generated C code.

MAXNESTING - The maximum number of recursive function call levels. The default is 10.

MAXSTRINGLEN - The maximum length of strings. The default is 256 bytes for 32 bit machines and

512 bytes for all other machines.

MAXTRYLOCK - The maximum number of iterations to wait for locks on global variables before declaring

possible deadlock and skipping the probe. The default is 1000.

MAXACTION - The maximum number of statements to execute during any single probe hit. The default is 1000.

MAXMAPENTRIES - The maximum number of rows in an array if the array size is not specified explicitly when

declared. The default is 2048.

MAXERRORS - The maximum number of soft errors before an exit is triggered. The default is 0.

MAXSKIPPED - The maximum number of skipped reentrant probes before an exit is triggered. The default is 100.

MINSTACKSPACE - The minimum number of free kernel stack bytes required in order to run a probe handler.

This number should be large enough for the probe handler's own needs, plus a safety margin. The default is 1024.

(9) 命令行參数

能够从命令行传递两种类型的參数：“字符串”和数值。

9.1 数值

$1 ... $<N> 用于在脚本中引用传入的数值參数。

9.2 字符串

@1 ... @<N> 用于在脚本中引用传入的字符串參数。

(10) 条件编译

%( CONDITION %?

TRUE-TOKENS %)

%( CONDITION %? TRUE-TOKENS %: FALSE-TOKENS %)

编译条件能够是：

@defined($var) // 目标变量是否可用

kernel_v > "2.6.37" // 比較版本

kernel_vr // 比較版本(包含后缀)

arch == "x86_64" // CPU架构

kernel CONFIG option。编译选项：

%( CONFIG_UTRACE == "y" %?

do something

%)