Advanced Troubleshooting with Debug Diagnostics on IIS 6 Draft 2.

5 5/13/06
Name Title Group Microsoft Corporation

Agenda
Introduction A look at Debug Diagnostics Dig into crash scenarios Understanding hangs Investigating memory leaks Session Summary

Introduction
Overview of Debugging: Past and Present Integral tools for resolving Debug scenarios Variables impacting Debugging

Overview of Debugging
Past and Present
Past: offered many tools, fewer resolutions
Previous tools were often not server-ready No single tool to address different debugging scenarios Required expertise in debugging

Present: more advanced tools, still timeconsuming to solve

Still lacks any analytical piece Existing tools are not integrated into single tool Do you really want to learn debugging?

Talking about tools

There are several tools designed at resolving the same problems As Windows Servers have grown, so have the debugging tools Example of debugging tools
Exception Monitor ADPlus.vbs (i.e. cdb.exe) IIS Debug Toolkit 1.1 IIS State

Variables Impacting Debugging (2)

IISs Extensibility Examples
ISAPI ASP.NET Active Server Pages (ASP) COM components PHP (ISAPI) Perl (CGI or ISAPI) Oracle\MySQL

Ever felt like this?

These extensible parts are often integrated and executing in same process

Debug Diagnostics 1.0

Core pieces of DebugDiag Understanding Rules Using DebugDiags User Interface

Core pieces of Debug Diag

3 pieces make up Debug Diag 1.0
Service Host (also called Controller) User Interface
DbgSvc.exe
DbgSvc.exe DebugDiag.exe

Core pieces of Debug Diag

The Service DbgSvc.exe

Service is multipurposed
Interacts with the Debugger Host Allows:
Service required to attach invasively to process To run from Terminal Services

DbgSvc.exe
To register the service: C:\DebugDiag> DbgSvc /service
To start the service: C:\DebugDiag> net start DbgSvc

To stop the service C:\DebugDiag> net stop DbgSvc

Unregister the service C:\DebugDiag> DbgSvc /unregserver

Core pieces of Debug Diag

The Host DbgHost.exe

Host
Where the DbgEng.dll instance is loaded Exposes 3 classes

DbgSvc.exe

Attac h

DbgHost.exe

DbgControl: Attach/Detach from processes or open/analyze a memory dump DbgObj: Collect process or memory dump information Manager: Mainly to output analysis data to the report file

Core pieces of Debug Diag

The User Interface DebugDiag.exe

User Interface designed to simplify:

Rule creation View processess Initiate analysis of memory dumps Add analysis scripts Change properties for Debug Diag

DebugDiag.exe

Control Scripts DbgHost.exe

Crash Scenarios
Background of crashes Using DebugDiag to quickly identify root cause of crashes

Background: Exceptions
What is an exception?
How do applications cause exceptions What is a handled exception? Unhandled exceptions need debugging and resolution
Access Violation: C000 0005
Breakpoint Exception: 8000 0003 Stack Overflow: C000 00FD C++ exception: E06D7363 Managed exception: E0434F4D Invalid Handle: C000 0008

Background: Heap (2)

What is a heap?
Heap is a data structure

Heap Corruption:
Certain rules must be followed when using the heap When rules are violated, heap corruption occurs Heap corruption crashes Code that corrupts heap is usually not the ones impacted

Debugging Crashes with Debug Diagnostics: Crash Rule

Name Title Group

Hang Scenarios
Investigating the theory of process hangs Using DebugDiag to debug process hangs

Theory of Process Hangs

What is a process?
Memory boundary for execution of code Allocated by Operating System done by calling CreateProcess or CreateProcessAsUser API Processes do not execute code Contains a minimum of 1 thread

What is a thread?
An independent sequence of instructions executing within a process

Theory of Process Hangs

Thread implications on applications
The level of concurrency (virtual) is dictated by the number of threads Real concurrency is dictated by the number of execution units
1 processor = one or more execution units

What are common thread pools for IIS?

ATQ thread pool ASP thread pool ASP.NET thread pool

Runaway Threads
A runaway thread is a thread stuck in a infinite (or near) loop
Performing a task that is consuming CPU Either by design or because of error conditions

Runaway thread is taking CPU resources away from other threads

Often takes away from good threads Causes performance degradation for server

Defining a Blocked Thread

A thread that isnt using any CPU resources Waiting on database calls to return Waiting on web service calls to return Waiting on a socket/network call to return Waiting on a lock

Thread 3 Thread IIS Thread 4 1

SQL

Locks and Deadlocks

What is a lock?
A synchronization mechanism to protect resources from being corrupted due to simultaneous access by multiple threads

Why do developers use locks?

Protecting data Protecting other resources that are not memory based

Impact on applications if locks are poorly implemented

Causes threads to block Can cause deadlock situations when using multiple locks in the application Process hangs when no threads are available

Debugging Deadlocks and Process Hangs with DebugDiag: Hang Rule

Name Title Group

Looking into Leaks

What is a leak? Types of Leaks Debugging a Leak with Debug Diagnostics

Leak Scenarios (2)

Why are leaks difficult to identify and resolve?
Server applications often cache memory allocations The allocations are long term These applications, to outsiders, look to have memory leaks

There are many other caches

Using Performance Monitor and seeing memory growth does not equal a memory leak

Leak Scenarios (3)

What are short-term allocations?
Memory expensive applications use spike allocations but are destroyed immediately after request (not long- sustaining)

These are not Memory Leaks

Leak Scenarios (4)

Difference between cache and memory leaks?
Memory leaks are caused by unbounded allocations Cache memory usage rises quickly, but tops out after application is stabilized

Nailing a Memory Leak in a Web Application using DebugDiag: Memory Rule

Name Title Group

Session Summary
Debugging Live Web Applications without downtime is challenging

Crashes cause processes to terminate due to unhandled exceptions Hangs are caused due to race conditions, lock contention, network blocking, database blocking, infinite loops and deadlocks Leaks are caused by the failure to release memory allocated beyond typical short term allocations and caching Debug Diagnostics has an extensibility model that allows it to successfully capture and analyze common debugging scenarios to determine the root cause Know what DebugDiag can doand also what it cant do

Debug Diagnostics 1.0 web resources

http://www.microsoft.com/windowsserver20 03/iis/diagnostictools/default.mspx

 2005 Microsoft Corporation. All rights reserved.

This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.