Redis debugging guide---官方
Redis debugging guide
Redis is developed with a great stress on stability: we do our best with every release to make sure you'll experience a very stable product and no crashes. However even with our best efforts it is impossible to avoid all the critical bugs with 100% of success.
When Redis crashes it produces a detailed report of what happened, however sometimes looking at the crash report is not enough, nor it is possible for the Redis core team to reproduce the issue independently: in this scenario we need help from the user that is able to reproduce the issue.
This little guide shows how to use GDB to provide all the informations the Redis developers will need to track the bug more easily.
What is GDB?
GDB is the Gnu Debugger: a program that is able to inspect the internal state of another program. Usually tracking and fixing a bug is an exercise in gathering more informations about the state of the program at the moment the bug happens, so GDB is an extremely useful tool.
GDB can be used in two ways:
- It can attach to a running program and inspect the state of it at runtime.
- It can inspect the state of a program that already terminated using what is called a core file, that is, the image of the memory at the time the program was running.
From the point of view of investigating Redis bugs we need to use both this GDB modes: the user able to reproduce the bug attaches GDB to his running Redis instance, and when the crash happens, he creates the core
file that the in turn the developer will use to inspect the Redis internals at the time of the crash.
This way the developer can perform all the inspections in his computer without the help of the user, and the user is free to restart Redis in the production environment.
Compiling Redis without optimizations
By default Redis is compiled with the -O2
switch, this means that compiler optimizations are enabled. This makes the Redis executable faster, but at the same time it makes Redis (like any other program) harder to inspect using GDB.
It is better to attach GDB to Redis compiled without optimizations using the make noopt
command to compile it (instead of just using the plain make
command). However if you have an already running Redis in production there is no need to recompile and restart it if this is going to create problems on your side. Even if by a lesser extent GDB still works against executables compiled with optimizations.
It is great if you make sure to recompile Redis with make noopt
after the first crash, so that the next time it will be simpler to track the issue.
You should not be concerned with the loss of performances compiling Redis without optimizations, it is very unlikely that this will cause problems in your environment since it is usually just a matter of a small percentage because Redis is not very CPU-bound (it does a lot of I/O to serve queries).
Attaching GDB to a running process
If you have an already running Redis server, you can attach GDB to it, so that if Redis will crash it will be possible to both inspect the internals and generate a core dump
file.
After you attach GDB to the Redis process it will continue running as usually without any loss of performance, so this is not a dangerous procedure.
In order to attach GDB the first thing you need is the process ID of the running Redis instance (the pid of the process). You can easily obtain it using redis-cli
:
$ redis-cli info | grep process_id
process_id:58414
In the above example the process ID is 58414.
- Login into your Redis server.
- (Optional but recommended) Start screen or tmux or any other program that will make sure that your GDB session will not be closed if your ssh connection will timeout. If you don't know what screen is do yourself a favour and Read this article
Attach GDB to the running Redis server typing:
gdb
<path-to-redis-executable>
<pid>
For example: gdb /usr/local/bin/redis-server 58414
GDB will start and will attach to the running server printing something like the following:
Reading symbols for shared libraries + done
0x00007fff8d4797e6 in epoll_wait ()
(gdb)
At this point GDB is attached but your Redis instance is blocked by GDB. In order to let the Redis instance continue the execution just type continue at the GDB prompt, and press enter.
(gdb) continue
Continuing.
Done! Now your Redis instance has GDB attached. You can wait for... the next crash :)
Now it's time to detach your screen / tmux session, if you are running GDB using it, pressing the usual Ctrl-a a key combination.
After the crash
Redis has a command to simulate a segmentation fault (in other words a bad crash) using the DEBUG SEGFAULT
command (don't use it against a real production instance of course ;). So I'll use this command to crash my instance to show what happens in the GDB side:
(gdb) continue
Continuing.
Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_INVALID_ADDRESS at address: 0xffffffffffffffff
debugCommand (c=0x7ffc32005000) at debug.c:220
220 *((char*)-1) = 'x';
As you can see GDB detected that Redis crashed, and was able to show me even the file name and line number causing the crash. This is already much better than the Redis crash report back trace (containing just function names and binary offsets).
Obtaining the stack trace
The first thing to do is to obtain a full stack trace with GDB. This is as simple as using the bt command: (that is a short for backtrace):
(gdb) bt
#0 debugCommand (c=0x7ffc32005000) at debug.c:220
#1 0x000000010d246d63 in call (c=0x7ffc32005000) at redis.c:1163
#2 0x000000010d247290 in processCommand (c=0x7ffc32005000) at redis.c:1305
#3 0x000000010d251660 in processInputBuffer (c=0x7ffc32005000) at networking.c:959
#4 0x000000010d251872 in readQueryFromClient (el=0x0, fd=5, privdata=0x7fff76f1c0b0, mask=220924512) at networking.c:1021
#5 0x000000010d243523 in aeProcessEvents (eventLoop=0x7fff6ce408d0, flags=220829559) at ae.c:352
#6 0x000000010d24373b in aeMain (eventLoop=0x10d429ef0) at ae.c:397
#7 0x000000010d2494ff in main (argc=1, argv=0x10d2b2900) at redis.c:2046
This shows the backtrace, but we also want to dump the processor registers using the info registers command:
(gdb) info registers
rax 0x0 0
rbx 0x7ffc32005000 140721147367424
rcx 0x10d2b0a60 4515891808
rdx 0x7fff76f1c0b0 140735188943024
rsi 0x10d299777 4515796855
rdi 0x0 0
rbp 0x7fff6ce40730 0x7fff6ce40730
rsp 0x7fff6ce40650 0x7fff6ce40650
r8 0x4f26b3f7 1327936503
r9 0x7fff6ce40718 140735020271384
r10 0x81 129
r11 0x10d430398 4517462936
r12 0x4b7c04f8babc0 1327936503000000
r13 0x10d3350a0 4516434080
r14 0x10d42d9f0 4517452272
r15 0x10d430398 4517462936
rip 0x10d26cfd4 0x10d26cfd4 <debugCommand+68>
eflags 0x10246 66118
cs 0x2b 43
ss 0x0 0
ds 0x0 0
es 0x0 0
fs 0x0 0
gs 0x0 0
Please make sure to include both this outputs in your bug report.
Obtaining the core file
The next step is to generate the core dump, that is the image of the memory of the running Redis process. This is performed using the gcore
command:
(gdb) gcore
Saved corefile core.58414
Now you have the core dump to send to the Redis developer, but it is important to understand that this happens to contain all the data that was inside the Redis instance at the time of the crash: Redis developers will make sure to don't share the content with any other, and will delete the file as soon as it is no longer used for debugging purposes, but you are warned that sending the core file you are sending your data.
If there are sensible stuff in the data set we suggest sending the dump directly to Salvatore Sanfilippo (that is the guy writing this doc) at the email address antirez at gmail dot com.
What to send to developers
Finally you can send everything to the Redis core team:
- The Redis executable you are using.
- The stack trace produced by the bt command, and the registers dump.
- The core file you generated with gdb.
- Informations about the operating system and GCC version, and Redis version you are using.
Thank you
Your help is extremely important! Many issues can only be tracked this way, thanks! It is also possible that helping Redis debugging you'll be among the winners of the next Redis Moka Award.
Redis debugging guide---官方的更多相关文章
- AVFoundation Programming Guide(官方文档翻译4)Editing - 编辑
新博客:完整版 - AVFoundation Programming Guide 分章节版:- 第1章:About AVFoundation - AVFoundation概述- 第2章:Using A ...
- Redis集群官方推荐方案 Redis-Cluster
Redis-Cluster redis使用中遇到的瓶颈 我们日常在对于redis的使用中,经常会遇到一些问题 1.高可用问题,如何保证redis的持续高可用性. 2.容量问题,单实例redis内存无法 ...
- 【简明翻译】Hibernate 5.4 Getting Started Guide 官方入门文档
前言 最近的精力主要集中在Hibernate上,在意识到Hibernate 5 的中文资料并不多的时候,我不得不把目光转向Hibernate的官方doc,学习之余简要翻一下入门文档. 原文地址:htt ...
- Redis 简介(官方翻译)
Redis是一个开源(基于BSD开源协议).内存型结构数据存储,可当做数据库.缓存.消息代理.它支持的数据结构有字符串.哈希表.列表.集合.可随机查询的有序集合.位图.基数统计.用于半径查询的地理位置 ...
- CentOS安装Redis详细教程
构建 Redis redis 目前没有官方 RPM 安装包,我们需要从源代码编译,而为了要编译就需要安装 Make 和 GCC. 如果没有安装过 GCC 和 Make,那么就使用 yum 安装. yu ...
- 基于Redis的开源分布式服务Codis
Redis在豌豆荚的使用历程--单实例==>多实例,业务代码中做sharding==>单个Twemproxy==>多个Twemproxy==>Codis,豌豆荚自己开发的分布式 ...
- 安全稳定实现redis cluster自动化迁移
背景 目前redis集群最火的是codis和redis cluster(官方),但官方自带工具并没有支持密码操作.那么需要密码认证使用redis cluster集群的同学要仔细看了哦. 相信大家很多人 ...
- redis集群讨论
一.生产应用场景 二.存储架构演变 三.应用最佳实践 四.运维经验总结 第1.2节:介绍redis cluster在唯品会的生产应用场景,以及存储架构的演变.第3节:redis cluster的稳定性 ...
- Redis安装及主从配置(转)
一.何为Redis redis是一个key-value存储系统.和Memcached类似,它支持存储的value类型相对更多,包括string(字符串).list(链表).set(集合)和zset(有 ...
随机推荐
- arduino远程刷新(烧录)固件
在车间部署了十几个网络版的温湿度采集器(基于arduino的),这些采集器分布在不同的地方,现在要更新一下上面的固件.最笨的方法是一个一个地取下来,插到电脑的USB接口上进行固件更新,这样做显然很麻烦 ...
- CS中窗体的基类(BaseForm)注意点
窗体基类最好新建一个窗体(BaseForm) 1.这样能够保证在VS中保证他的派生窗口也能够可视化. 2.如果基类直接是一个cs类文件,对于处理派生窗口就很复杂,比如按钮权限之类的操作; 如果直接继承 ...
- Hello World! 我的程序员入坑之旅!
先说下本文标题,各行各业都有自己的行规和一些内行人玩的梗什么的,这是我开始写技术博客的第一篇,所以它的标题毫无疑问只能是Hello World! 介绍一下我自己 我算是一个少见的科班出身的开发者了,1 ...
- 975. Odd Even Jump
You are given an integer array A. From some starting index, you can make a series of jumps. The (1 ...
- django orm 以列表作为筛选条件进行查询
在Django的orm中进行查询操作时,可以通过传入列表,列表内的元素为索引值,作为一个筛选条件来进行行查询 from .models import UserInfo user_obj = UserI ...
- 文件操作(FILE)与常用文件操作函数
文件 1.文件基本概念 C程序把文件分为ASCII文件和二进制文件,ASCII文件又称文本文件,二进制文件和文本文件(也称ASCII码文件)二进制文件中,数值型数据是以二进制形式存储的, 而在文本文件 ...
- poj3233 Matrix Power Series(矩阵快速幂)
题目要求的是 A+A2+...+Ak,而不是单个矩阵的幂. 那么可以构造一个分块的辅助矩阵 S,其中 A 为原矩阵,E 为单位矩阵,O 为0矩阵 将 S 取幂,会发现一个特性: Sk +1右上角 ...
- CentOS7下 Python2.7.5升级为Python2.7.13
参考:https://www.jianshu.com/p/fad3942fc0ed 第一步:查看Centos版本及Python版本 • CentOS版本 [root@ tools_package]# ...
- git 查看提交记录
查看提交的内容 -p 选项,同时在 - 后加数字限制一下数目 git log -p -2. commit 500eeadd71a21f1166803e12a792bfa86f4ca784 (HEAD ...
- debug 工具
git blame 查看某个文件的修改记录  二分查找确定 bug 来源 启动  输入 git bisect start,启动流程 输入 git bisect bad,标记当前是错误的 输入 gi ...