Sunteți pe pagina 1din 76

Automated Debugging

Ansuman Banerjee
Indian Statistical Institute
Outline
• Introduction
• Program Analysis Techniques for Debugging
– Program Slicing
–Testing based Fault Localization
• Debugging Change-Induced Bugs
• Summary
What is a bug?
“A software bug (or just "bug") is an error, flaw, mistake, … in a
computer program that prevents it from behaving as intended
(e.g., producing an incorrect result). … Reports detailing bugs in a
program are commonly known as bug reports, fault reports, …
change requests, and so forth.” --- Wikipedia

“Even today, debugging remains very much of an art. Much of the


computer science community has largely ignored the debugging
problem….. over 50 percent of the problems resulted from the time
and space chasm between symptom and root cause or inadequate
debugging tools.” -- IBM Sys Jnl

 Debugging is a time consuming activity

 Need methods / tools to trace back to root cause of bug from the
manifested error 1/19/2011
3
Software Controlled Devices Everywhere
Automobiles

Medical Entertainme

Handheld
Airplanes
Military
Do we live in a Bug-free world?
• Therac 25: Four dead

• The European Ariane 5 rocket


explodes 40 s into its maiden flight

• Mars Polar Lander lost

• Pentium FDIV bug

• …Numerous in “Software Hall of


Shame”
Facts on Debugging
• Software bugs are costing a lot every
year
• Improvements could reduce cost by
30%
• Validation (including debugging) can
easily take up to 50-75% of the
development time
• When debugging, some people are
three times as efficient than others
7
Debugging: Some numbers

• ACM CrossTalk article


• Predictive Quality Control with Software Inspections
• http://www.stsc.hill.af.mil/crosstalk/1994/06/xt94d06e.asp

• Real-life case studies to find actual errors in software via very


slow code inspection

• Errors are classified as major errors or not.

• Outcome = 13 major errors per 1000 LOC.


• For Windows Vista (~50 Million LOC), approx. 650000 errors !!

• Need automation support


• Program Analysis
Software Debugging
• Typical Debugging Steps
Execution
1. Hypothesize
Error? D
the cause of
Error? C the error
Error? B
2. Try to confirm it
Observable A
error
A Sample Program
int main(int argc, char *argv[])
{
int *a;
int i;

a = (int *)malloc((argc - 1) * sizeof(int));


for (i = 0; i < argc - 1; i++)
a[i] = atoi(argv[i + 1]);

shell_sort(a, argc);

printf("Output: ");
for (i = 0; i < argc - 1; i++)
printf("%d ", a[i]);
printf("\n");

free(a);

return 0; 10
A Sample Run
$ sample 9 8 7
Output: 7 8 9

$ sample 11 14
Output: 0 11

11
How to Debug
(Sommerville 2004)

Locate Design Repair Re-test


error error repair error program

12
The Traffic Principle
T rack the problem
R eproduce
A utomate
F ind Origins
F ocus
I solate
C orrect

13
The Traffic Principle
T rack the problem
R eproduce
A utomate
F ind Origins
F ocus
I solate
C orrect

14
From Defect to Failure
1. The programmer creates a
defect – an error in the
code. Variables

2. When executed, the defect
creates an infection – an ✘
error in the state.
3. The infection propagates.
✘ ✘
4. The infection causes a
failure.
This infection chain must be
✘ t
traced back – and broken.

15
Debugging

• Every failure can be Variables


traced back to some ✘
infection, and every
infection is caused by ✘
some defect.
• Debugging means to ✘ ✘
relate a given failure to
the defect – and to
remove the defect.

16
Search in Space + Time

17
The Defect

18
A Program State

19
A Sample Program
$ sample 9 8 7
Output: 7 8 9

$ sample 11 14
Output: 0 11

20
int main(int argc, char *argv[])
{
int *a;
int i;

a = (int *)malloc((argc - 1) * sizeof(int));


for (i = 0; i < argc - 1; i++)
a[i] = atoi(argv[i + 1]);

shell_sort(a, argc);

printf("Output: ");
for (i = 0; i < argc - 1; i++)
printf("%d ", a[i]);
printf("\n");

free(a);

return 0;
}

21
Find Origins
• The 0 printed is the
value of a[0]. Where
does it come from?
• Basic idea: Track or
deduce value origins
• Separates relevant
from irrelevant values
• We can trace back a[0]
to shell_sort

22
static void shell_sort(int a[], int size)
{
int i, j;
int h = 1;
do {
h = h * 3 + 1;
} while (h <= size);
do {
h /= 3;
for (i = h; i < size; i++)
{
int v = a[i];
for (j = i; j >= h && a[j - h] > v; j -= h)
a[j] = a[j - h];
if (i != j)
a[j] = v;
}
} while (h != 1);
}

23
Search in Time

• In shell_sort, the
state must have
become infected.
• Basic idea:
Observe a
transition from
sane to infected.

24
Observing a Run

25
Specific Observation
static void shell_sort(int a[], int size)
{
fprintf(stderr, “At shell_sort”);
for (i = 0; i < size; i++)
fprintf(stderr, “a[%d] = %d\n”, i, a[i]);
fprintf(stderr, “size = %d\n”, size);
int i, j;
int h = 1; $ sample 11 14
... a[0] = 11
} a[1] = 14
a[2] = 0
size = 3
Output: 0 11

The state is infected at the call of shell_sort!


26
Fixing the Program
int main(int argc, char *argv[])
{
int *a;

// Input array
a = (int *)malloc((argc - 1) * sizeof(int));
for (int i = 0; i < argc - 1; i++)
a[i] = atoi(argv[i + 1]);

// Sort array
shell_sort(a, argc);
Should be argc - 1
// Output array
printf("Output: ");
for (int i = 0; i < argc - 1; i++)
printf("%d ", a[i]);
printf("\n");

free(a);
return 0;
}
27
Finding Causes
Infected state Sane state

The difference
causes the failure
28
Concepts
A failure comes to be in three stages:
1. The programmer creates a defect
2. The defect causes an infection
3. The infection causes a failure -- an
externally visible error.
Not every defect results in an infection,
and not every infection results in a
failure.
29
The Traffic Principle
T rack the problem
R eproduce
A utomate
F ind Origins
F ocus
I solate
C orrect

30
Automated Debugging
A variety of tools and techniques
available to automate debugging:
• Program Analysis
• Slicing
• Observing & Watching Runs
• Asserting Invariants
• Detecting Anomalies

• Isolating Cause-Effect Chains


31
Outline
• Introduction
• Program Analysis Techniques for
Debugging
• Program Slicing
• Testing based Fault Localization
• Debugging Change-Induced Bugs
• Summary
Program Analysis for Debugging
Program Analysis for Debugging

• What to analyze?
• the execution run, Dynamic Analysis
• the source code, Static Analysis

• How to analyze?
• Program Slicing
• Testing based Fault Localization
Program Slicing
1 b=1;
2 y=1;
3 If (a>1){
4 if (b>1){
5 x=2;
}
}
6 printf (“%d”, x);
Program Slicing
1 b=1;
2 y=1;
3 If (a>1){
Control
Dependence 4 if (b>1){
5 x=2; Data
Dependence
}
}
Slicing
6 printf (“%d”, x); Criterion
Program Slicing
1 b=1;
2 y=1;
Control
3 If (a>1){
Dependence 4 if (b>1){
Data
5 x=2; Dependence
}
}
Slicing
6 printf (“%d”, x); Criterion
Dependence Graph

B
Control / Data
Dependence

A Statement
Program Slicing

Control / Data
Dependence

Statement
Slicing Criterion
Program Slicing

Control / Data
Dependence

Statement

Slicing Criterion
Static vs Dynamic Slicing
• Static Slicing
• source code
• statement
• static dependence
• Dynamic Slicing (useful for debugging)
• a particular execution
• statement instance
• dynamic dependence
Use of Dynamic Slicing
Input

Instrument
Program Exec. Trace

Debugging Output
Dynamic Slice =
criterion Bug Report

OK
Unexpected,
debug it
Static vs Dynamic
Slicing
1 b=1;
2 If (a>1)
3 x=1;
4 else
5 x=2;
6 printf (“%d”, x); Slicing Criterion
Dynamic Slice - Illustration
0. scanf(“%d”, &A);

1. If (A == 0) {;
2. W = X;
3. U = A;
4. }

5. printf(“%d\n”, U);

Criterion: 5,U with input A = 0


Trace: <0,1,2,3,4,5> Slice = {0,1,3,5}
Data dependences encountered 5 -> 3, 1 -> 0
Control dependence encountered 3 -> 1
Forward and Backward
Slicing

1 a=1;
2 x=a; Slicing Criterion
3 if (x>1)
4 y=10;
Backward Slicing

1 a=1;
2 x=a;
3 if (x>1)
4 y=10;
Forward Slicing

1 a=1;
2 x=a;
3 if (x>1)
4 y=10;
Slicing: Tool Support

• WET (Whole Execution Traces)


• http://wet.cs.ucr.edu/

• BitBlaze
• http://bitblaze.cs.berkeley.edu/
Outline
• Introduction
• Program Analysis Techniques for Debugging
• Program Slicing
• Testing based Fault Localization
• Debugging Change-Induced Bugs
• Summary
Testing and Debugging
• Two very close software development
activities

Testing
Testing Stage
Methods

Debugging Stage Debugging


Methods
Testing Based Fault Localization
• Use testing to help debugging
Test Suite (test cases)

Program

Failing Runs Successful Runs


Fault Localization

Successful Run Pool Testing

Change Failing
Generate Choose Input

Failing Run Successful Run

Difference Metric
Compare Execution

Difference As bug report


Testing Based Fault
Localization

Difference:
Bug?

Failing Run Successful Run


Testing Based Fault
Localization

 What to Compare
 choice of the Execution Run

 How to Compare
Choice of the Execution
Run
• Failing run
• Select one failing run for comparison
• Different failing runs may correspond to
different error causes
• Successful Run
• A Single successful run, VS
• A Set of successful runs
Choice of a Successful
Run
• The difference can be related to:
• different inputs
• the error

• The successful run and the failing run should be as


similar as possible
• Choose a “good” successful run for comparison
Choose a Successful
Run
• Comparison of Differences Select one
failing run for comparison

diff
diff’

Failing π Successful π’ Failing π Successful π’


Choose a Successful
Run
• Comparison of Differences Select one
failing run for comparison

diff
< diff’
Testing Based Fault
Localization
 What to Compare
 choice of the Execution Run

 How to Compare
 statement / basic block
 predicates / branch statement
 variable values
 potential invariants
Fault Localization - invariants
• Collect potential invariants from successful runs
• e.g. x > 0 at the beginning of the method foo()
• Check whether these potential invariants hold in the
failing run being investigated.
• Past project in an offering of this course, in fact!

• Invariant detector --- Daikon


• http://pag.csail.mit.edu/daikon/
• The potential invariants are inferred, not specified.

• The tool, of course, needs to fix a language of potential


invariants.
Outline
• Introduction
• Program Analysis Techniques for Debugging
• Program Slicing
• Testing based Fault Localization
• Debugging Change-Induced Bugs
• Summary
Debugging Change
Induced Bugs

74
1
Change-Induced Bugs
• Changes may induce bugs in stable programs
• Changes may be due to several factors

• Cost of managing software evolution takes large percent


of the total cost of software.

• Finding root cause of regression bugs is still a major


headache in large software development projects.
75
1
Manual review isn’t enough!

76
1
A True Story
• Release 4.17 of the GNU debugger GDB brought several new features, languages,
and platforms, but for some reason, it no longer integrated properly with the
graphical front-end data display debugger DDD

• The arguments specified within DDD were not being passed to the debugged
program.

• Something changed within GDB such that it no longer worked

• Something? Between the 4.16 and 4.17 releases, no less than 178,000 lines
changed.
• How can I isolate the change that caused the bug and make GDB work again?

• The GDB example is an instance of the “worked yesterday, not today” problem: after
applying a set of changes, the program no longer works as it should.
Research Objective
Test Input t

Old Stable New Buggy


Program P Program P’

78
1
An experiment we tried
Bug-free distribution:
Linux (GNU Core-utils, net –tools)

Development
Changes

Buggy distribution: BusyBox

Busybox distribution is 121 KLOC.


Errors to be root-caused in some common
utilities: arp, top, printf 79
1/19/2011
Debugging Busybox
• The concept
• Correct program: GNU Coreutils
• Changed Version: Busybox
• De-facto distribution for embedded devices.
• Aims for low code size
• Less checks and more errors.

• The practice
• Large bug report produced using trace comparison
80
1/19/2011
A glimpse inside the
The ARP bug
ARP bug
Embedded Linux

Crash

-Ainet
Shows all computers
connected to host
with inet address
family
GNU Coreutils

Crash identified as NULL pointer access at crash site


hw_type unexpectedly set as NULL at crash site
81
1
BusyBox ARP
int arp_main(int argc, char **argv) {
....
option_mask32 = getopt32(argc, argv, "A:p:H:t:i:adnDsv“, &protocol,
&protocol, &hw_type, &hw_type, &device);
if (option_mask32 & ARP_OPT_A || option_mask32 & ARP_OPT_p){
ap = get_aftype(protocol);
}
if (option_mask32 & ARP_OPT_A || option_mask32 & ARP_OPT_p){
hw = get_hwtype(hw_type);
else hw_set = 0;

if (hw_set==0)
if ((hw = get_hwtype(DFLT_HW)) == NULL)
// Error check and exit
}
const struct hwtype *get_hwtype (const char *name) {
.... const struct hwtype * const *hwp; Slicing Criteria
hwp = hwtypes;
while (*hwp != NULL) { name == NULL
if (!strcmp((*hwp)->name, name)) Crash since
return (*hwp); name is NULL
hwp++;
} Test Input: -Ainet
return NULL; 82
1
ARP in GNU core-utils
int arp_main(int argc, char **argv) {
....
option_mask32 = getopt32(argc, argv, "A:p:H:t:i:adnDsv“, &protocol,
&protocol, &hw_type, &hw_type, &device);
if (option_mask32 & ARP_OPT_A || option_mask32 & ARP_OPT_p){
ap = get_aftype(protocol);
}
if (option_mask32 & ARP_OPT_H || option_mask32 & ARP_OPT_p)
hw = get_hwtype(hw_type);
else hw_set = 0;

if (hw_set==0)
if ((hw = get_hwtype(DFLT_HW)) == NULL)
// Error check and exit
}
…… const struct hwtype *get_hwtype (const char *name) {
const struct hwtype * const *hwp; hwp = hwtypes;
while (*hwp != NULL) {
Slicing Criteria
if (!strcmp((*hwp)->name, name))
name != NULL
return (*hwp);
hwp++;
}
Test Input: -Ainet
return NULL;
}
83
1
Methodology in action
Trace Collection with –Ainet for
buggy and stable ARP

Identify variable responsible


for crash and map it to stable
ARP

name == NULL name != NULL comparison

Slice busyBox Slice net-tools


trace trace
solver

Map back to source to Differing


terms

Generate bug report 84


1/19/2011
Acknowledgement: Thanks to Andreas
Zeller
• Slides available from Prof. Zeller’s homepage
• Yesterday, my program worked. Today, it does
not. Why? Proc. FSE 99
• Automated Debugging: Are We Close? IEEE TC
• Locating Causes of Program bugs, ICSE 2005
• Debugging Debugging, FSE 2009 Keynote
Ongoing Research Projects

 Change-driven test-suite augmentation


 Problem: Developing test cases / test plan
for changed features is a manual activity
 Objective: Automatically creating test
cases for the changes made to an old
program

 Debugging concurrent programs


86
1
Ongoing Research Projects
 Debugging optimized binaries without source
 Source code of the buggy program may not be
available
 Stack trace sent from client site

 Microsoft Office crashes

 Can we decipher this report

based on registry footprint?

87
1
Thank you !

S-ar putea să vă placă și