TROUBLESHOOTING LINUX

1 BEST PRACTICES AND INITIAL INVESTIGATION

1.1 Introduction

1.2 Getting Your System(s) Ready for Effective Problem Determination

1.3 The Four Phases of Investigation

1.3.1 Phase #1: Initial Investigation Using Your Own Skills

1.3.2 Phase #2: Searching the Internet Effectively

1.3.3 Phase #3: Begin Deeper Investigation (Good Problem Investigation Practices)

1.3.4 Phase #4: Getting Help or New Ideas

1.4 Technical Investigation

1.4.1 Symptom Versus Cause

1.5 Troubleshooting Commercial Products

2 STRACE AND SYSTEM CALL TRACING EXPLAINED

2.1 Introduction

2.2 What Is strace?

2.2.1 More Information from the Kernel Side

2.2.2 When to Use It

2.2.3 Simple Example

2.2.4 Same Program Built Statically

2.3 Important strace Options

2.3.1 Following Child Processes

2.3.2 Timing System Call Activity

2.3.3 Verbose Mode

2.3.4 Tracing a Running Process

2.4 Effects and Issues of Using strace

2.4.1 strace and EINTR

2.5 Real Debugging Examples

2.5.1 Reducing Start Up Time by Fixing LD_LIBRARY_PATH

2.5.2 The PATH Environment Variable

2.5.3 stracing inetd or xinetd (the Super Server)

2.5.4 Communication Errors

2.5.5 Investigating a Hang Using strace

2.5.6 Reverse Engineering (How the strace Tool Itself Works)

2.6 System Call Tracing Examples

2.6.1 Sample Code

2.6.2 The System Call Tracing Code Explained

3 THE /PROC FILESYSTEM

3.1 Introduction

3.2 Process Information

3.2.1 /proc/self

3.2.2 /proc/<pid> in More Detail

3.2.3 /proc/<pid>/cmdline

3.2.4 /proc/<pid>/environ

3.2.5 /proc/<pid>/mem
3.2.6 /proc/<pid>/fd

3.2.7 /proc/<pid>/mapped base

3.3 Kernel Information and Manipulation

3.3.1 /proc/cmdline

3.3.2 /proc/config.gz or /proc/sys/config.gz

3.3.3 /proc/cpufreq

3.3.4 /proc/cpuinfo

3.3.5 /proc/devices

3.3.6 /proc/kcore

3.3.7 /proc/locks

3.3.8 /proc/meminfo

3.3.9 /proc/mm

3.3.10 /proc/modules

3.3.11 /proc/net

3.3.12 /proc/partitions

3.3.13 /proc/pci

3.3.14 /proc/slabinfo

3.4 System Information and Manipulation

3.4.1 /proc/sys/fs

3.4.2 /proc/sys/kernel

3.4.3 /proc/sys/vm

3.5 Conclusion
4 COMPILING

4.1 Introduction

4.2 The GNU Compiler Collection

4.2.1 A Brief History of GCC

4.2.2 GCC Version Compatibility

4.3 Other Compilers

4.4 Compiling the Linux Kernel

4.4.1 Obtaining the Kernel Source

4.4.2 Architecture Specific Source

4.4.3 Working with Kernel Source Compile Errors

4.4.4 General Compilation Problems

4.5 Assembly Listings

4.5.1 Purpose of Assembly Listings

4.5.2 Generating Assembly Listings

4.5.3 Reading and Understanding an Assembly Listing

4.6 Compiler Optimizations

5 THE STACK

5.1 Introduction

5.2 A Real-World Analogy

5.3 Stacks in x86 and x86-64 Architectures

5.4 What Is a Stack Frame?

5.5 How Does the Stack Work?

5.5.1 The BP and SP Registers

5.5.2 Function Calling Conventions

5.6 Referencing and Modifying Data on the Stack

5.7 Viewing the Raw Stack in a Debugger

5.8 Examining the Raw Stack in Detail

5.8.1 Homegrown Stack Traceback Function

6 THE GNU DEBUGGER (GDB)

6.1 Introduction

6.2 When to Use a Debugger

6.3 Command Line Editing

6.4 Controlling a Process with GDB

6.4.1 Running a Program Off the Command Line with GDB

6.4.2 Attaching to a Running Process

6.4.3 Use a Core File

6.5 Examining Data, Memory, and Registers

6.5.1 Memory Map

6.5.2 Stack

6.5.3 Examining Memory and Variables

6.5.4 Register Dump

6.6 Execution

6.6.1 The Basic Commands

6.6.2 Settings for Execution Control Commands

6.6.3 Breakpoints

6.6.4 Watchpoints

6.6.5 Display Expression on Stop

6.6.6 Working with Shared Libraries

6.7 Source Code

6.8 Assembly Language

6.9 Tips and Tricks

6.9.1 Attaching to a Process—Revisited

6.9.2 Finding the Address of Variables and Functions

6.9.3 Viewing Structures in Executables without Debug Symbols

6.9.4 Understanding and Dealing with Endian-ness