Source:http://v0ids3curity.blogspot.com/2015/04/data-structure-recovery-using-pin-and.html

--------------------------------

Data Structure Recovery using PIN and PyGraphviz

 
This is a simple POC PIN tool to recover data structures in dynamically linked stripped executables, mainly for analyzing small programs. The PIN tool keeps track of heap allocations done by executable and traces all the write operations to the allocated heap memory. The trace file having allocations and write operations will be used to generate graph using pygraphviz.

Tracing Size and Address of Allocations

Size of Allocation

Right now, we track libc functions malloc, realloc, calloc, sbrk, mmap and free. All these routines are instrumented using Rtn_InsertCall to fetch the size of requested allocation.

For example, for tracing malloc

  1. RTN_InsertCall( rtn,
  2.                     IPOINT_BEFORE,
  3.                     (AFUNPTR)AllocBefore,
  4.                     IARG_ADDRINT,
  5.                     funcname,
  6.                     IARG_G_ARG0_CALLEE,
  7.                     IARG_RETURN_IP,
  8.                     IARG_END);

We fetch the size of requested allocation using IARG_G_ARG0_CALLEE and IPOINT_BEFORE. Also, we need to identify the malloc calls that are only called from our main executable. To find this we use IARG_RETURN_IP to check if the return address of the call is part of main executable, if not we don't trace the allocation. 

Address of Allocation

IARG_RETURN_IP is valid only at function entry point, so we cannot use IPOINT_AFTER along with IARG_RETURN_IP. As a work around, we save the return address during IPOINT_BEFORE. Then in instruction trace, if instruction pointer equals return address of an allocation call, we fetch the EAX value. This gives the address of allocation.

  1. if(insaddr == retaddress){
  2.         INS_InsertCall( ins,
  3.                 IPOINT_BEFORE,
  4.                 (AFUNPTR)AllocAfter,
  5.                 #ifdef __i386__
  6.                 IARG_REG_VALUE, LEVEL_BASE::REG_EAX,
  7.                 #else
  8.                 IARG_REG_VALUE, LEVEL_BASE::REG_RAX,
  9.                 #endif
  10.                 IARG_END);
  11. }

Now we have both address and size of allocation. These details are stored as dictionary as pairs of address : size. Also we don't remove an address when free is called upon that, instead if an already existing address is returned during an allocation call ie. reallocation, we just update the size of existing allocation for the new allocation request.

  1. if(allocations.count(address)==0){
  2.         allocations.insert(std::make_pair(address, allocsize));
  3. }
  4. else{
  5.         std::map<addrint, addrint="" style="font-size: 14px;">::iterator it = allocations.find(retval);
  6.         it->second = allocsize;
  7. }

.data and .bss sections

data and bss sections are also added to dictionary. The size and address of these segments are fetched from main executable and added as part of allocations

  1. if(!strcmp(sec_name.c_str(),".bss")||!strcmp(sec_name.c_str(),".data")){
  2.                 ADDRINT addr = SEC_Address(sec);
  3.                 USIZE size = SEC_Size(sec);
  4.                 if(allocations.count(addr)==0){
  5.                     allocations.insert(std::make_pair(addr, size));
  6.                 }
  7. }

Tracing Memory Writes

We trace instructions that writes into the allocated memory. As of now only XED_ICLASS_MOV class of instructions are traced. For all XED_ICLASS_MOV instruction, we check if its a memory write instruction using INS_IsMemoryWrite and is part of main executable.

In this case, we fetch the destination address of write operation using IARG_MEMORYWRITE_EA. Then we check if the destination address is part of any allocation, on success this instruction is traced.

  1. for(it = allocations.begin(); it != allocations.end(); it++){
  2.         if((des_addr >= it->first)&&(des_addr < it->first+it->second))returntrue;
  3. }

Sample Trace

  1. .data[0x804b02c,0x8]
  2. .bss[0x804b040,0xfc4]
  3. 0x8048560@sbrk[0x420]
  4. ret[0x98de000]
  5. 0x8048565@mov dword ptr [0x804c000], eax             : WRREG MEM[0x804c000] VAL[0x98de000]
  6. 0x8048575@mov dword ptr [eax+0x8],0x0               : WRIMM MEM[0x98de008] VAL[0]
  7. 0x804857f@mov dword ptr [edx+0x4], eax               : WRREG MEM[0x98de004] VAL[0]
  8. 0x8048587@mov dword ptr [eax],0x10                  : WRIMM MEM[0x98de000] VAL[0x10]
  9. 0x80485a0@mov dword ptr [eax+0x4], edx               : WRREG MEM[0x98de004] VAL[0x98de010]
  10. 0x80485ac@mov dword ptr [eax+0x8], edx               : WRREG MEM[0x98de018] VAL[0x98de000]

Graphing

Node Create

For each allocation in trace file generated by PIN tool, a new node is created in the graph. Each node is uniquely identified using a node id which is assigned sequentially. An ordered dictionary is maintained, key being node id and value is dictionary of address and size of allocation. New allocations are added to the start of ordered dictionary. 

An edge count is associated with each of created node. This will be used for pruning away nodes without any edges.

Separate nodes are created for bss and data sections. But this is optional.

Example

Say a structure is allocated in heap using malloc, this is how a node will look like

  1. 0x80488c5@malloc[0x20]
  2. ret[0x8fcf030]
  1. -------------------
  2. |[0]0x8fcf030   |
  3. -------------------

[0] is the node id, this could signify the order of allocation. Every new allocator call gets a new id, irrespective of the return address

0x8fcf030 is the address returned by allocator call

Node Update

For each instruction, fetch the target address of write operation. If the target address is part of any allocation, update the node to which the target address belongs to. Basically we create a new port in the record node.

A new port signifies an element of an allocation, say element of a structure.

Then check if the source value is part of any allocation. If yes, we consider the source value as an address. Then update the node to which the source address belongs to. This operation could be interpreted as a pointer assignment [or link creation]

  1. 0x80488c5@malloc[0x20]
  2. ret[0x8fcf030]
  3. 0x8048957@movbyte ptr [eax+edx*1],0x0      : WRIMM MEM[0x8fcf031] VAL[0]
  4. 0x80489bb@mov dword ptr [eax+0x14], edx      : WRREG MEM[0x8fcf044] VAL[0x8fcf058]
  5. 0x8048a40@mov dword ptr [eax+0x18], edx      : WRREG MEM[0x8fcf048] VAL[0x8fcf008]
  6. 0x8048a4e@mov dword ptr [eax+0x1c], edx      : WRREG MEM[0x8fcf04c] VAL[0x8fcf008]
  1. -------------------------------------------------------------------------
  2. |[0]0x8fcf030   |0x8fcf031  |0x8fcf044  |  0x8fcf048  |  0x8fcf04c  |
  3. -------------------------------------------------------------------------

Now first field [0] 0x8fcf030 is the meta data for the chunk ie node id and return address of allocator call. The rest of 4 fields signifies 4 individual write operations into this allocation [example, 4 elements of a structure] 

Create Link

If both source and destination values are valid address and belongs to a traced allocation, we link both ports of the nodes. Whenever a link is created, edge count of source and destination are incremented.

Similarly, during memory overwrite an edge is removed and edge count is decremented. 

Example,

  1. 0x804882a@malloc[0x20]
  2. ret[0x8fcf008]
  3. …...
  4. 0x80488c5@malloc[0x20]
  5. ret[0x8fcf030]
  6. …...
  7. 0x80489bb@mov dword ptr [eax+0x14], edx             : WRREG MEM[0x8fcf044] VAL[0x8fcf058]
  8. 0x8048a40@mov dword ptr [eax+0x18], edx             : WRREG MEM[0x8fcf048] VAL[0x8fcf008]
  9. 0x8048a4e@mov dword ptr [eax+0x1c], edx             : WRREG MEM[0x8fcf04c] VAL[0x8fcf008]

Above is a series of pointer writes into memory allocated at 0x8fcf030. The address points to another allocation at 0x8fcf008. Hence we link both

Prune Node

Finally after parsing all instructions, remove nodes that doesn't have any edges. For this, check if the edge count for a node is 0. If yes, remove the node.

Other Options

By default, we consider only the first non-NULL write operation for node update and link creation. This might be good enough to reveal some of data structures. Any memory writes to an address after first write non-NULL are skipped. But one can use relink option to consider more than single write operation for graphing. This could be useful when relink operations are done, say circular linked list.

NULL writes can also be enabled as option. This might be useful along with relink.

The tool itself doesn't itself have the intelligence to say what data structure is used, but can graph the allocation and links to help someone understand a data structure from the revealed shape.

Example - Singly Linked List

Example - Binary Tree

Example - HackIM Mixme Circular Doubly Linked List

The POC code which I use for CTF is available here. To repeat again, this works on small binaries, as things get complex the graph might make less sense. There is lot of scope for improvement though.

References

AES Whitebox Unboxing: No Such Problem, this served as excellent reference for the usage of PIN tool and pygraphviz to visualize memory access

Thanks to Danny K, for help with Intel PIN Framework.

[转]Data Structure Recovery using PIN and PyGraphviz的更多相关文章

  1. [LeetCode] All O`one Data Structure 全O(1)的数据结构

    Implement a data structure supporting the following operations: Inc(Key) - Inserts a new key with va ...

  2. [LeetCode] Add and Search Word - Data structure design 添加和查找单词-数据结构设计

    Design a data structure that supports the following two operations: void addWord(word) bool search(w ...

  3. [LeetCode] Two Sum III - Data structure design 两数之和之三 - 数据结构设计

    Design and implement a TwoSum class. It should support the following operations:add and find. add - ...

  4. Finger Trees: A Simple General-purpose Data Structure

    http://staff.city.ac.uk/~ross/papers/FingerTree.html Summary We present 2-3 finger trees, a function ...

  5. Mesh Data Structure in OpenCascade

    Mesh Data Structure in OpenCascade eryar@163.com 摘要Abstract:本文对网格数据结构作简要介绍,并结合使用OpenCascade中的数据结构,将网 ...

  6. ✡ leetcode 170. Two Sum III - Data structure design 设计two sum模式 --------- java

    Design and implement a TwoSum class. It should support the following operations: add and find. add - ...

  7. leetcode Add and Search Word - Data structure design

    我要在这里装个逼啦 class WordDictionary(object): def __init__(self): """ initialize your data ...

  8. Java for LeetCode 211 Add and Search Word - Data structure design

    Design a data structure that supports the following two operations: void addWord(word)bool search(wo ...

  9. HDU5739 Fantasia(点双连通分量 + Block Forest Data Structure)

    题目 Source http://acm.hdu.edu.cn/showproblem.php?pid=5739 Description Professor Zhang has an undirect ...

随机推荐

  1. JAVA转让JS功能

    今天,在发展中使用js和Java互动.通常我们使用更多的是js转让Java方法.可以使用dwr.Ajax.jquery.突然发现Java转让js然后,我真的没见过,今天,互联网提供以下信息,顺便总结: ...

  2. DDD实践案例:引入事件驱动与中间件机制来实现后台管理功能

    DDD实践案例:引入事件驱动与中间件机制来实现后台管理功能 一.引言 在当前的电子商务平台中,用户下完订单之后,然后店家会在后台看到客户下的订单,然后店家可以对客户的订单进行发货操作.此时客户会在自己 ...

  3. C语言中符号格式说明

    scanf 语法: #include <stdio.h>int scanf( const char *format, ... ); scanf()函数根据由format(格式)指定的格式从 ...

  4. C#中的Params、ref、out的区别

    C# Params params params 关键字可以指定在参数数目可变处采用参数的方法参数. 在方法声明中的 params 关键字之后不允许任何其他参数,并且在方法声明中只允许一个 params ...

  5. C#操作Xml:XSLT语法 在.net中使用XSLT转换xml文档示例

    XSL即可扩展的样式表文件. 可以格式化xml的显示,也可以将xml转换成需要的另一种格式. 学习XSL必须熟悉XPath.XSL和XPath一样简单强大,容易学习. 1. XSL既然可以格式化xml ...

  6. javascript权威指南(2)

    JavaScript预定义了一系列全局变量和函数,在自定义变量和函数式要避免使用这些预定义的名称: arguments encodeURI  Infinity  Number  RegExp Arra ...

  7. POJ 2262 Goldbach&#39;s Conjecture(素数相关)

    POJ 2262 Goldbach's Conjecture(素数相关) http://poj.org/problem?id=2262 题意: 给你一个[6,1000000]范围内的偶数,要你将它表示 ...

  8. 让你的Windows不断重启的C语言代码

    原文:让你的Windows不断重启的C语言代码 没有写Linux的原因是因为搞不定Linux下的权限问题,而Windows下基本上使用电脑的用户都是管理员,所以钻个空了,不多说下面是代码#includ ...

  9. 关于android中sqllite对时间的操作

    sql 中有时间的类型,date,time,datetime,方便关于记录的维护,下面一个demo演示怎么在每条记录中默认增加时间 源码下载地址 http://www.codes51.com/code ...

  10. 对象javascript

    //对象 var lennon=Object(); lennon.name="John"; lennon.year=1940; lennon.living=false; conso ...