IBM NA Advanced Technical Skills (ATS) Oracle Solutions Team

Breaking the Oracle I/O Performance Bottleneck

12/16/2009

2009 IBM Corporation

IBM NA Advanced Technical Skills (ATS) - Oracle Solutions Team

L Legal li information f ti
The information in this presentation is provided by IBM on an "AS IS" basis without any warranty, guarantee or assurance of any kind. IBM also does not provide any warranty, guarantee or assurance that the information in this paper is free from errors or omissions. Information is believed to be accurate as of the date of publication. You should check with the appropriate vendor to obtain current product information. Any proposed use of claims in this presentation outside of the United States must be reviewed by local IBM country counsel prior to such use. AIX, BladeCenter, GPFS, HACMP, IBM, Micro-Partitioning, POWER, Power Systems PowerHA, Systems, PowerHA PowerVM PowerVM, pSeries and RS/6000 are trademarks or registered trademarks of the International Business Machines Corporation. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. affiliates All other products or company names are used for identification purposes only, and may be trademarks of their respective owners.

2009 IBM Corporation

IBM NA Advanced Technical Skills (ATS) - Oracle Solutions Team

Presentation Overview
In most Oracle database environments, I/O related wait events are responsible for a substantial portion of the total DB processing ti time. P Poor I/O subsystem b t performance f can unnecessarily il contribute to poor end user response time, long running batch jobs and/or system throughput capacity issues. This p presentation will discuss storage g subsystem y industry y trends that have impacted Oracle database performance historically. It will also discuss one rapidly emerging storage technology Solid State Disk (SSD). A number of p practical suggestions gg will be provided p for breaking g the Oracle I/O performance bottleneck using available tools and techniques.

2009 IBM Corporation

IBM NA Advanced Technical Skills (ATS) - Oracle Solutions Team

CPU Performance Trends (2000-2007) (2000 2007)

Per core performance increased 13x Total system performance increased 68x TPC-C High Water Mark (2000-2007)
80

~ 40% CAGR* ~ 80% CAGR* CAGR

Relative Pe R erformance e

70 60 50 40 30 20 10 0 2000

Total System "Per Core"

2001

2002

2003

2004

2005

2006

2007 CAGR* = Compound Average Growth Rate

2009 IBM Corporation

Based on 2000 2007 TPC-C benchmark results Source: http://www.tpc.org

4

IBM NA Advanced Technical Skills (ATS) - Oracle Solutions Team

Hard Disk Drive (HDD) Capacity

HDD storage capacity is growing at 40% per year, similar to CPU performance advances
Areal Density

Source: Storage Class Memory: The next storage system technology by R.F. Freitas and W.W. Wilcke, ,
IBM Journal of Research and Development, July/September 2008
5 2009 IBM Corporation

IBM NA Advanced Technical Skills (ATS) - Oracle Solutions Team

Hard Disk Drive (HDD) Performance

Sequential Bandwidth only improving ~ 15% per year largely driven by areal density increases
Sequential Bandwidth

Random I/O latency improving very slowly

Rotational speed topped out at 15,000 RPM Seek times improving ~ 5% per year at best
Source: Storage Class Memory: The next storage system technology by R.F. Freitas and W.W. Wilcke, IBM
Journal of Research and Development, July/September 2008
6 2009 IBM Corporation

IBM NA Advanced Technical Skills (ATS) - Oracle Solutions Team

The I/O Performance Challenge

Exploding data volumes are driving the need for ever larger, faster servers
Thankfully, Moores Law still applies to processor technology

Evolving business requirements are driving demand for ever faster access to data HDD performance hasnt kept pace

Plus HDD consumes a lot of energy and takes up a lot of space

2009 IBM Corporation

IBM NA Advanced Technical Skills (ATS) - Oracle Solutions Team

Meeting the Challenge

I/O avoidance
Larger server memory cache Larger I/O subsystem cache Application redesign

More Efficient I/O

Data Layout Strategies Data Compression Smarter cache management algorithms Application redesign

Getting beyond HDD

In Memory Database Selective use of Solid State Disk (SSD)

2009 IBM Corporation

IBM NA Advanced Technical Skills (ATS) - Oracle Solutions Team

Improving Oracle Database I/O Performance -- TODAY

Striping Data to avoid I/O hotspots Tuning the Database Buffer Cache Implementing Data Compression Using Solid State Disk

Using faster HDD based storage subsystems

15k RPM SAS dri drives es Ample read/write cache Enterprise Class cache management algorithms

2009 IBM Corporation

IBM NA Advanced Technical Skills (ATS) - Oracle Solutions Team

Data Striping to Avoid I/O Hotspots

Old Wisdom: Isolate files based on function and/or usage
Manually intensive effort Leads to I/O hotspots that impact throughput capacity and performance

New Wisdom: Stripe objects across as many physical disks as possible

Minimal manual intervention Evenly balanced I/O across all available physical components Good average I/O response time object throughput capacity with no hotspots

Implementation Options:
ASM and GPFS do this automatically within a given disk group or filesystem Can be implemented with conventional Volume Managers and filesystems:
http://www-1.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP100319
Volume (Disk) Group

AIX Storage
LUN 1 LUN 2 LUN 3 LUN 4

HW Striping

SW W Striping

hdisk 1

Disk Group D

hdisk 2 hdisk 3 hdisk 4

10

2009 IBM Corporation

IBM NA Advanced Technical Skills (ATS) - Oracle Solutions Team

Tuning Oracle DB Buffer Cache

Buffer Cache is the primary database I/O avoidance option Old Wisdom: If the buffer hit% is > 90% it its s good enough New Wisdom: Depending on workload, a higher hit% may be optimal
For a given workload with a buffer hit% of 98%, a 1% increase (to 99%) will reduce physical I/O requests by 50% Reducing IOPS will also improve response time for remaining I/Os In many cases, adding server memory may be cheaper than adding I/O subsystem cache memory or short-stroking disks

Evaluate impact of increasing db db_cache_size cache size on physical I/O

Oracle AWR Buffer Cache Advisory report is a good place to start Monitor for and address potential impacts:
Increased p peak CPU demand Increased logical read rates System paging (add physical memory as necessary)

11

2009 IBM Corporation

IBM NA Advanced Technical Skills (ATS) - Oracle Solutions Team

Oracle AWR Buffer Cache Statistics (Real World Example)

Instance Efficiency Percentages
Buffer Nowait %: Buffer Hit %: Library Hit %: Execute to Parse %: Parse CPU to Parse Elapsd %: 99.97 98.89 70.53 28.44 30.81 Redo NoWait %: In-memory Sort %: Soft Parse %: Latch Hit %: % Non-Parse CPU: 100.00 100.00 26.01 99.96 89.14

B ff Hit% = 98 Buffer 98.89 89

Buffer Pool Advisory Report

P D D D D D D D D D D D D D D Size for Est (M) 256 288 320 352 384 400 416 448 480 512 544 576 608 640 Size Factor 0.64 0.72 0.80 0.88 0.96 1.00 1.04 1.12 1.20 1.28 1.36 1.44 1.52 1.60 Buffers for Estimate 16,080 18,090 20,100 22,110 24,120 25,125 26,130 28,140 30,150 32,160 34,170 36,180 38,190 40,200 Est Phys Read Factor Estimated Physical Reads 1.11 1.11 1.08 1.05 1.02 1.00 0.98 0.94 0.90 0.86 0.82 0.78 0.72 0.67 97,368,882 96,868,286 94,323,210 91,776,695 89,228,794 87,480,193 85,731,549 82,232,582 78,731,330 75,225,110 71,715,825 68,209,778 63,357,042 58,494,659

Predicts 29 (of 87) million block reads could be eliminated over 30 minute i t period by adding 240 MB of buffer pool cache: 2 2,000 000 read IOs /second 16,000 blocks /second 125 MB/second A 33% savings
2009 IBM Corporation

12

IBM NA Advanced Technical Skills (ATS) - Oracle Solutions Team

Oracle Data Compression

Sequential scans typically limited by sequential I/O bandwidth of backend disk storage Data Compression increases the number of rows stored per physical data block If sufficient CPU capacity is available, the potential increase in sequential data scan rate should correlate to the Compression Ratio (CR)
Oracle Compression Options:
Standard table compression
Only supported for bulk loads/inserts

CR: up to 4x (2x typical)

Advanced Compression (11g)

Supports OLTP as well as DW workloads Separately licensed feature @ $11,500/processor license

CR: up to 4x (2x typical)

Hybrid Columnar Compression (HCC)

CR: up to 10x for Query Compression

Intended for p predominantly y read only y DW workloads Only available on Sun/Oracle proprietary 11gR2 Exadata Storage Servers

13

2009 IBM Corporation

IBM NA Advanced Technical Skills (ATS) - Oracle Solutions Team

SQL Statement Tuning

AWR SQL ordered by Reads report is a good place to look for SQL statements that may benefit from tuning
SQL ordered by Reads
Physical Reads 51,014,572 Executions Reads per Exec %Total 10.62 CPU Time Elapsed (s) Time (s) 4512.75 SQL Id SQL Module SQL Text BEGIN APPS.IKN_FS_TRAC_SR_ PKG.... SELECT CONTACT_ID FROM IKN_O... SELECT SUMMARY FROM CS INCIDEN CS_INCIDEN... UPDATE CS_INCIDENTS_ALL_B SET ... SELECT wdd.delivery_name||... y_ ||

27 1,889,428.59

20648.44 cpb5vktsrxxnp

49,905,589

1,913

26,087.61

10.39

6777.33

13408.17 9kfwnug5cbszd

22,956,796

26

882,953.69

4.78

2002.24

9528.41 52rzwx2917kd2

20,329,085 16,650,797

23

883,873.26

4.23 3.47

1817.61 1564.46

8203.42 am77sx96jujqk 9668.13 f9zpsu61dq7cy SQL*Plus

5 3,330,159.40

If desired, semi-automatic tools are available to evaluate SQL statement performance and recommend possible remedial actions
2009 IBM Corporation

14

IBM NA Advanced Technical Skills (ATS) - Oracle Solutions Team

Solid State Disk (SSD)

A rapidly emerging technology that uses solid state electronics instead of spinning disks
No moving parts High random IOPS capacity Low latency L Low energy consumption ti Low capacity (relative to HDDs) High $/GB Low $/IOPS Internal drives for Power Systems and System x servers DS5000 DS8000 SAN Volume Controller

Recently announced for several IBM products, including:

Within 5 years, may well replace HDDs for Tier 1 disk storage
According to IDC, SSD revenue is expected to have a 78% CAGR through 2012 Source: http://searchstorage.techtarget.com/news/article/0,289142,sid5_gci1357860,00.html
15 2009 IBM Corporation

IBM NA Advanced Technical Skills (ATS) - Oracle Solutions Team

Using SSD With Oracle

80:20 Rule
Data access tends to be skewed If 80% of the I/O is to 20% of the data, optimizing performance for that 20% can provide substantial returns

Therefore put the hottest or most response time critical files on SSD Best Candidates:
Read mostly files with a high random I/O Access Density
For example, moderately sized control tables, indexes Access Density = IOPS/Gigabyte

Write latency sensitive files

Online redo logs Control files Undo Tablespaces Temp Tablespaces

Files with high average (read/write) wait times are better candidates than those with low average wait times
16 2009 IBM Corporation

IBM NA Advanced Technical Skills (ATS) - Oracle Solutions Team

AWR Wait Events

Top 5 Timed Events
Event Waits 188,158 Time(s) 2,757 1,259 20,320 , 6,135 1,203 325 54 13 16 9 11

3
Avg Wait(ms) 15 % Total Call Time 61.8 28.2 7.3 1.2 .3 Commit User I/O User I/O Wait Class User I/O

1 2

db file sequential read DB CPU log g file sync y db file scattered read read by other session

1. If User User I/O I/O events (especially db db file sequential read read) ) account for a high percentage of total Oracle wait time and Average Wait time is high, selective use of SSD may improve performance 2. If log file sync event appears in top 5 list and Average Wait time is high, high the online redo logs are SSD candidates 3. The higher the Average Wait time for these events, the greater the potential SSD benefit

17

2009 IBM Corporation

IBM NA Advanced Technical Skills (ATS) - Oracle Solutions Team

AWR I/O Statistics

Tablespace IO Stats
Tablespace APPS_TS_TX_DATA APPS_TS_TX_IDX UNDO_RBS3 UNDO_RBS1 UNDO RBS2 UNDO_RBS2 APPS_TS_NOLOGGING APPS_TS_SUMMARY TEMP Reads 200,399,314 42,294,848 11,684,361 5,176,461 4 210 423 4,210,423 1,861,280 1,046,663 209,066 Av Reads/s 7,952 1,678 464 205 167 74 42 8 Av Rd(ms) 4.96 4.93 4.53 5.30 5 74 5.74 2.79 5.24 21.66 Av Writes Blks/Rd 1.84 1.00 1.00 1.00 1 00 1.00 1.05 1.68 15.00 1,489,381 825,266 144 951,372 150 454,678 520,912 268,191 59 33 0 38 0 18 21 11

3
Av Buffer Waits Writes/s 153,852,375 10,495,582 239,596 271,908 200 446 200,446 0 2 195 Av Buf Wt(ms) 0.95 0.87 1.50 1.29 2 69 2.69 0.00 0.00 1.49

1
Total Read Time (seconds) 993,981 208,514 52,930 27,435 24 168 24,168 5,193 5,485 4,528

2
1. Total Read Time = Reads x Average Read Time (ms) / 1000
Could be used to prioritize candidates if Average Read time isnt uniform

2. High Average Blocks/Read values indicate sequential scan activity, which might be a reason to leave on HDD 3. High Buffer Waits values might indicate Write Latency sensitive candidates
18 2009 IBM Corporation

IBM NA Advanced Technical Skills (ATS) - Oracle Solutions Team

AWR I/O Statistics

File IO Stats (ordered by Tablespace Name)
Tablespace Filename APPS_TS_TX _DATA APPS_TS_TX _DATA APPS_TS_TX _DATA APPS TS TX APPS_TS_TX _DATA APPS_TS_TX _DATA APPS_TS_TX _DATA APPS_TS_TX _DATA DATA APPS_TS_TX _DATA APPS_TS_TX _DATA APPS_TS_TX _DATA APPS TS TX APPS_TS_TX _DATA APPS_TS_TX _DATA APPS_TS_TX _DATA APPS_TS_TX _DATA /oradata01/EPRODR/apps_ ts_tx_data01.dbf /oradata01/EPRODR/apps_ ts_tx_data02.dbf /oradata01/EPRODR/apps_ ts_tx_data03.dbf /oradata01/EPRODR/apps /oradata01/EPRODR/apps_ ts_tx_data04.dbf /oradata01/EPRODR/apps_ ts_tx_data05.dbf /oradata01/EPRODR/apps_ ts_tx_data06.dbf /oradata01/EPRODR/apps_ ts tx data07 dbf ts_tx_data07.dbf /oradata01/EPRODR/apps_ ts_tx_data08.dbf /oradata01/EPRODR/apps_ ts_tx_data09.dbf /oradata01/EPRODR/apps_ ts_tx_data10.dbf /oradata01/EPRODR/apps /oradata01/EPRODR/apps_ ts_tx_data11.dbf /oradata01/EPRODR/apps_ ts_tx_data12.dbf . /oradata01/EPRODR/apps_ ts_tx_data72.dbf /oradata01/EPRODR/apps_ ts_tx_data73.dbf Reads Av Av Av Buffer Av Buf Av Rd(ms) Writes R d / Reads/s Blk /Rd Blks/Rd W it / Writes/s W it Waits Wt( ) Wt(ms) 2,360,196 94 7.38 2.08 17,331 1 1,375,004 1.37 2,455,028 2,451,324 2 423 122 2,423,122 2,386,389 2,399,096 2,447,058 2,474,975 2,481,775 2,460,570 2 453 171 2,453,171 2,446,896 1,722,356 1,034,959 97 97 96 95 95 97 98 98 98 97 97 68 41 7.35 7.41 7 53 7.53 7.69 7.60 7.44 7.38 7.32 7.29 7 29 7.29 7.38 3.72 3.94 2.10 2.10 2 11 2.11 2.12 2.12 2.09 2.08 2.07 2.08 2 08 2.08 2.08 1.36 1.45 17,308 16,151 16 538 16,538 17,182 16,772 16,600 16,123 15,843 15,507 15 569 15,569 14,893 26,159 24,194 1 1 1 1 1 1 1 1 1 1 1 1 1 1,387,878 1,412,479 1 285 744 1,285,744 1,272,999 1,267,411 1,324,944 1,377,729 1,406,400 1,364,264 1 364 091 1,364,091 1,324,884 2,087,133 1,324,884 1.38 1.38 1 47 1.47 1.49 1.46 1.45 1.42 1.37 1.40 1 42 1.42 1.46 0.51 0.59

19

2009 IBM Corporation

IBM NA Advanced Technical Skills (ATS) - Oracle Solutions Team

Factoring in I/O Access Density

If data files are of significantly different sizes, it may be appropriate to factor I/O access density into the candidate selection process
If 2 files are of different sizes, but are responsible for equal amounts of read I/O wait time, the smaller file will be a better SSD candidate because it has a higher I/O access density Stated in another way, since SSD has a high $/GB, the smallest files with the hi h t wait highest it ti time offer ff th the bi biggest tb bang f for th the b buck k

File size may be determined from filesystem statistics, or Oracle DB views such as V$DATAFILE

20

2009 IBM Corporation

IBM NA Advanced Technical Skills (ATS) - Oracle Solutions Team

Sample Script: File Name by Read Read Wait Density Density

select bt.tsname, bt.filename, 1024*1024*1024*(et.readtim - bt.readtim) / df.bytes rd_tm_per_GB, et.readtim - bt.readtim read_time, , et.writetim - bt.writetim write_time, et.phyrds - bt.phyrds phys_reads, et.phywrts - bt.phywrts phys_writes, et.phyblkrd t h blk d - bt.phyblkrd bt h blk d blocks_read, bl k d et.phyblkwrt - bt.phyblkwrt blocks_wrt, et.singleblkrds - bt.singleblkrds single_reads, et.singleblkrdtim g - bt.singleblkrdtim g single_read_time, g _ _ , df.bytes / 1024/1024 M_Bytes from sys.dba_hist_filestatxs bt, sys.dba_hist_filestatxs et, v$datafile df where bt.snap_id = :bid and et.snap_id t id = :eid id and d bt.dbid = :dbid and bt.dbid = et.bdid and bt.instance_number = :inst_num and bt.instance_number = et.instance_number and bt.file# = et.file# and bt.file# = df.file# order by 3 desc, 4 desc;
21 2009 IBM Corporation

Total read wait time per Gigabyte Order by priority based on:
High I/O access density High average read wait time

IBM NA Advanced Technical Skills (ATS) - Oracle Solutions Team

SSD Configuration Options

Hardware RAID-5 based SSD LUNs
Provides protection against single SSD component failure RAID-10 could be used, but at higher $/GB Suitable for most purposes, e.g. .dbf files with high I/O access density

ASM SSD/HDD hybrid y disk g group p

Create HYBRID ASM disk group with Normal Redundancy SSD failure group is composed of one or more SSD LUNs (not RAIDprotected) HDD HDD failure group is composed of one or more HDD LUNs (RAID protection optional) alter system set ASM_PREFERRED_READ_FAILURE_GROUPS = HYBRID.SSD; Suitable for read read mostly mostly files at lower $/GB (data mirrored on inexpensive HDD)

JBOD SSD LUNs

SSD LUNS with no hardware RAID-protection Suitable for multiplexed redo logs or non-mirrored temp objects at even lower $/GB
22 2009 IBM Corporation

IBM NA Advanced Technical Skills (ATS) - Oracle Solutions Team

Implementation Steps (General)

1. Evaluate existing data layout strategy
If not already striping across a minimal number of disk disk pools, pools , create a strategy to get there, e.g. ASM, GPFS, File System striping If Buffer Pool Advisory recommends increasing Buffer Pool size, acquire additional physical memory (if required) and increase db db_cache_size cache size as appropriate Re-evaluate performance and address an unanticipated side effects Repeat

2. Evaluate existing Buffer Pool allocation

3. Evaluate possible use of Oracle data compression for sequential read workloads
Implement as appropriate

4 Evaluate SQL statements accounting for a larger percentage of 4. total Read I/O wait time
Tune as appropriate

23

2009 IBM Corporation

IBM NA Advanced Technical Skills (ATS) - Oracle Solutions Team

Implementation Steps (SSD)

1. Acquire a minimum of 2 SSD drives
For RAC environments, must be shared storage (e.g. DS5000, DS8000, SVC) For single instance environments, may use either internal system or external subsystem based

2. Review AWR or other Oracle performance data to determine candidates

High random read I/O Access Density:

High read IOPS, with high average read wait times and low Blocks per read average in AWR Tablespace IO Stats or File IO Stats report Prioritize based on Read Read Wait density density: : Total Read wait time/Gigabyte

Write Latency Sensitive:

High log file sync wait time (with high average wait time) in AWR Top 5 Timed Events report High buffer busy waits (with high average read wait time) in AWR Tablespace IO S Stats or Fil File IO Stats S report

3. Implement strategies based on candidate file type

24

RAID protected SSD LUNs ASM preferred preferred read read SSD/HDD hybrid JBOD based SSD LUNs
2009 IBM Corporation

IBM NA Advanced Technical Skills (ATS) - Oracle Solutions Team

QUESTIONS ANSWERS

25

2009 IBM Corporation