Foreword
Prefa e.
About the Authors
A knowledgments
PART ONE Observability Methods
Chapter 1 Introdu tion to Observability Tools
1.1 Observability Tools
1.2 Drill-Down Analysis
1.3 About Part One
Chapter 2 CPUs
2.1 Tools for CPU Analysis
2.2 vmstat Tool
2.3 CPU Utilization
2.4 CPU Saturation
2.5 psrinfo Command
2.6 uptime Command
2.7 sar Command
2.8 Clock Tick Woes
2.9 mpstat Command
2.10 Who Is Using the CPU?
2.11 CPU Run Queue Latency
2.12 CPU Statisti s Internals
2.13 Using DTrace to Explain Events from Performance Tools
2.14 DTrace Versions of runq- sz, %runo
2.15 DTrace Probes for CPU States
Chapter 3 Pro esses
3.1 Tools for Pro ess Analysis
3.2 Process Statistics Summary: prstat
3.3 Process Status: ps
3.4 Tools for Listing and Controlling Pro esses
3.5 Process Introspection Commands
3.6 Examining User-Level Locks in a Process
3.7 Tracing Processes
3.8 Java Processes
Chapter 4 Disk Behavior and Analysis
4.1 Terms for Disk Analysis
4.2 Randomcvs. Sequential I/O
4.3 Storage Arrays
4.4 Sector Zoning
4.5 Max I/O Size
4.6 iostat Utility
4.7 Disk Utilization
4.8 Disk Saturation
4.9 Disk Throughput
4.10 iostat Referen e
4.11 Reading iostat
4.12 iostat Internals
4.13 sar -d
4.14 Trace Normal Form TNF) Tracing for I/O
4.15 DTrace for I/O
4.16 Disk I/O Time
4.17 DTraceToolkit Commands
4.18 DTraceTazTool
Chapter 5 File Systems
5.1 Layers of File System and I/O
5.2 Observing Physi al I/O
5.3 File System Latency
5.4 Causes of Read/Write File System Latency
5.5 Observing File System “Top End” Activity
5.6 File System Caches
5.7 NFS Statistics
Chapter 6 Memory
6.1 Tools for Memory Analysis
6.2 vmstat(1M) Command
6.3 Types of Paging
6.4 Physical Memory Allocation
6.5 Relieving Memory Pressure
6.6 Scan Rate asca Memory Health Indicator
6.7 Process Virtual and Resident Set Size
6.8 Using pmap to Inspect Process Memory Usage
6.9 Calculating Pro ess Memory Usage with ps and pmap
6.10 Displaying Page-Size Information with pmap
6.11 Using DTrace for Memory Analysis
6.12 Obtaining Memory Kstats
6.13 Using the Perl Kstat API to Look at Memory Statistics
6.14 System Memory Allo ation Kstats
6.15 Kernel Memow with kstat
6.16 System Paging Kstats
6.17 Observing MMU Performance Impact with trapstat
6.18 Swap Space
Chapter 7 Networks
7.1 Terms for Network Analysis
7.2 Packets Are Not Bytes
7.3 Network Utilization
7.4 Network Saturation
7.5 Network Errors
7.6 Misconfigurations
7.7 Systemwide Statistics
7.8 Per-Process Network Statistics
7.9 TCP Statistics
7.10 IP Statistics
7.11 ICMP Statistics
Chapter 8 Performance Counters
8.1 Introducing CPU Caches
8.2 pustat Command
8.3 putrack Command
8.4 busstat Command
Chapter 9 Kernel Monitoring
9.1 Tools for Kernel Monitoring
9.2 Profiling the Kernel and Drivers
9.3 Analyzing Kernel Locks
9.4 DTrace lockstat Provider
9.5 DTrace Kernel Profiling
9.6 Interrupt Statistics: vmstat -i
9.7 Interrupt Analysis: intrstat
PART TWO Observability Infrastructure
Chapter 10 Dynami Tracing
10.1 Introduction to DTrace
10.2 The Basics
10.3 Inspecting Java Appli ations with DTrace
10.4 DTrace Architecture
10.5 Summary
10.6 Probe Reference
10.7 MDB Reference
Chapter 11 Kernel Statistics
11.1 C-Level Kstat Interface
11.2 Command-Line Interface
11.3 Using Peri to A ess kstats
11.4 Snooping a Program's kstat Use with DTrace
11.5 Adding Statistics to the Solaris Kernel
11.6 Additional Information
PART THREE Debugging
Chapter 12 The Modular Debugger
12.1 Introduction to the Modular Debugger
12.2 MDB Concepts
Chapter 13 An MDB Tutorial
13.1 Invoking MDB
13.2 MDB Command Syntax
13.3 Working with Debugging Targets
13.4 GDB-to-MDB Referen e
13.5 dcmd and Walker Reference
Chapter 14 Debugging Kernels
14.1 Working with Kernel Cores
14.2 Examining User Process Stacks within a Kernel Image
14.3 Swit hing MDB to Debugca Specifi Process
14.4 kmdb, the Kernel Modular Debugger
14.5 Kernel Built-In MDB dcmds
APPENDICES
Appendix A Tunables and Settings
Appendix B DTrace One-Liners
Appendix C Java DTrace Scripts
Appendix D Sample Peri Kstat Utilities
Bibliography
Index