Gumshoe - Microsoft Code Coverage Test Toolset

2014-07-17

What is Gumshoe?

How to instrument a binary?

step-by-step instruction

Step 1: Install GumShoe

Step 2: Instrument binary

Step 3: Rename the binaries as follows and copy to run path

Step4: Start a Gumshoe session and add resources of .covsym and .dpk files

Step5: Do test scenarios to check if these scenarios covered the fixed code

Step6: View Gumshoe Results

Gumshoe concpets

How does Gumshoe relate to Magellan?

Reference

What is Gumshoe?

Top

Gumshoe is a toolset for integrating code coverage into the workflow between developers and testers by providing real-time feedback on the code coverage of code changed by a developer.

DownLoad (The version 2.0.1300.1000 is stable)

How to instrument a binary?

Top

Instrumentation is the process of adding hooks into a binary so that code coverage can be collected on it when the binary is run. Gumshoe uses Magellan to instrument most built binaries (i.e. .NET, C, C++ binaries). Gumshoe also supports plugins for javascript instrumentation.

Instrumentation typically produces three files:

An instrumented binary,
an instrumented pdb,
and a covsym file.

The covsym file must be added to the session for Gumshoe to be able to then collect coverage on the instrumented binary.

Attention: Instrumentation can cause binaries to run slower than usual. You probably will not want to do performance tests with instrumented binaries. Badly designed tests can sometimes fail because actions may take longer on an instrumented binary than a non-instrumented binary.

step-by-step instruction

Top

Step 1: Install GumShoe

DownLoad (recommend version 2.0.1300.1000)

Step 2: Instrument binary

Before Instrument binary, we created C# class binary 'Triangle.dll' and Console Application 'CodeCoverageTest' to call method in 'Triangle.dll':

Triangle.dll: is binary to be done code coverage test.
CodeCoverageTest.exe: call method in 'Triangle.dll' to simulate test.

Triangle.dll source code:

 namespace Triangle

 {

     public class Triangle

     {

         public bool IsTriangle(int a, int b, int c)

         {

             if (a >  && b >  && c >  && a + b > c && a + c > b && b + c > a)

                 return true;

             return false;

         }

     }

 }

CodeCoverageTest.dll source code

 using System;

 namespace CodeCoverageTest

 {

     class Program

     {

         static void Main(string[] args)

         {

             Triangle.Triangle tri = new Triangle.Triangle();

             bool isTriangle = false;

             isTriangle= tri.IsTriangle(, , );

             Console.WriteLine("Is Triangle?" + isTriangle.ToString());

             Console.Read();

             isTriangle = tri.IsTriangle(, , );

             Console.WriteLine("Is Triangle?" + isTriangle.ToString());

             Console.Read();

         }

     }

 }

Execute bbcover command to Instrument binary

//binary name include path

bbcover.exe /i <binary name> /CovSym
for example: bbcover.exe /i C:\CodeCoverageTest\Triangle\bin\Debug\Triangle.dll /CovSym

Above command will create three files:

An instrumented binary: Triangle.dll.instr
an instrumented pdb: Triangle.dll.instr.pdb
and a covsym file: Triangle.dll.covsym

Step 3: Rename the binaries as follows and copy to run path

Rename the intrument binaries 'Triangle.dll.instr' to 'Triangle.dll', and 'Triangle.dll.instr.pdb' to 'Triangle.pdb'.

Copy the renamed binaries to run path - the path which CodeCoverageTest.exe could call it.

Attention: if orginal binary install into GAC, your renamed binary should alos install into GAC, then can call it.

Step4: Start a Gumshoe session and add resources of .covsym and .dpk files

Imput command as following:

gumshoe start

gumshoe addresource covsym:C:\CodeCoverageTest\Triangle\bin\Debug\Triangle.dll.covsym

For dpk file, it is contain fixed code files and difference file compare with fix and before fix. if add such dpk as resources, gumshoe can show the covered fixed blocks and arcs. for details, please refer to gumsheo concepts DPK.

The command of add dpk as resource is as following:

gumshoe addresource dpk:<dpk file>

Step5: Do test scenarios to check if these scenarios covered the fixed code

copy CodeCoverageTest.exe to run path and start it, so exe can call method in Triangle.dll

Step6: View Gumshoe Results

input following command to see result in command prompt:

gumshoe list full

Input following command to see result in Gumshoe GUI:

gumshoe publish <folder>

open the folder and double click covproj file to open Gumshoe GUI.

Because the test data of Triangle edges is: 1,1,2, the code coverage results is as following screenshot:

Lines colored in green with a "Y" next to them means the line was fully covered.
Lines colored in red with a "N" next to them means the line was not covered.
Lines colored in teal with a "P" next to them means that the line was partially covered. Because Magellan collects code coverage at the assembly block level, multiple blocks sometimes correspond to one source line. In cases of a partially hit line, one block is hit while another is not hit. The most common case is an if statement where one part of the conditional was hit but the other part was not hit. See the Magellan Tools page for information on how to disassemble a function into blocks to see the block-level coverage.

Gumshoe Concepts

Top

Code blocks and arcs:

A block is a sequence of machine instructions with exactly one entry point and one exit point. An arc is what connects two blocks to one another

Coverage symbols (covsym):

Gumshoe needs "resources" to properly interpret data. A coverage symbols file, also known as a covsym, maps binary data from the instrumented binary executable specific to the build being tested.

DPK:

A SD(Source Depot) Package file commonly used by various groups across the company to bundle a batch of source code changes. In fact, the Office triage process is based on SD Packages. A .dpk file is used to pass around a set of diffs as a single unit and consists of base source files, your modified source files and a manifest, bundled in a zip file with the extension .dpk. SDPack provides mechanisms to create, manage, view, and apply .dpk files. See Toolbox for more information on SDPack. A DPK file can be added to Gumshoe as a resource. From that DPK Gumshoe will automatically generate a filter based on the changed\added code in addition to extracting the modified source files so that you can see an annotated view of the code without having to first locate the source files.

Filter:
A filter is used in Gumshoe to scope down a change. One of the most common scenarios for a filter is applying a source line filter (or a DPK filter) to see the coverage scoped to a particular changelist. Other filters include symbol filters, which allow you to filter on static covsym data such as BinaryName or Directory, and trace filters, which allow you to filter based on a trace name.

Sessions:
Once a Gumshoe session is started, it remains active until you specifically enter the command to end the session, even if you reboot your computer.

How does Gumshoe relate to Magellan?

Top

Magellan is a great framework for collecting code coverage and scales incredibly well to meet the needs of a large organization. But it can be cumbersome to use on a small scale such as your desktop. Gumshoe builds on top of Magellan and aims to simplify the experience of using code coverage and improving the workflow between developers and testers.

One of the key scenarios for which it is currently used in Office is during buddy testing. It enables a user to quickly scope the coverage results to those lines of code that were added\changed by the developer and then see if they have even been executed during ad-hoc testing (and it does this without using a Magellan database if you're using a local session)!

With code coverage, 100% covered != 100% tested, but 0% covered does == 100% *not* tested. With Gumshoe you can quickly identify the code that has not been tested and easily move from black box to white box testing where additional test cases may become obvious. And when those additional test cases may not be obvious, it means it is time to have a conversation with your developer if you haven’t already. In fact, this is one of the benefits that we’ve seen come out of Gumshoe during our use in Office. It has helped change the dynamic between dev and test and drive a deeper conversation around the code.

The current version of Gumshoe leverages Magellan for managed\unmanaged code. It also provides a plugin mechanism for other types of code coverage. Currently, a JSscript plugin for Magellan is available. There are beta plugins for SQL and PERL coverage also in development.

Reference

[1] https://microsoft.sharepoint.com/teams/Gumshoe/default.aspx

[2] Gumshoe and Magellan

[3] Gumshoe Walkthrough