Hi Khurram, This is a complex error! You may want to open an SR on metalink: http://metalink.oracle.

com It's a semaphore (mutex) issue, an exclusive lock . . . Check out V$MUTEX_SLEEP and V$MUTEX_SLEEP_HISTORY for more details . . . The Oracle docs note: cursor: pin S wait on X A session waits for this event when it is requesting a shared mutex pin and another session is holding an exclusive mutex pin on the same cursor object.

One of the most confounding problems with Oracle is the resolution of buffer busy wait events. Buffer busy waits are common in an I/O-bound Oracle system, as evidenced by any system with read (sequential/scattered) waits in the top-five waits in the Oracle STATSPACK report, like this:
Top 5 Timed Events % Total Event Waits Time (s) Ela Time --------------------------- ------------ ----------- ----------db file sequential read 2,598 7,146 48.54 db file scattered read 25,519 3,246 22.04 library cache load lock 673 1,363 9.26 CPU time 2,154 934 7.83 log file parallel write 19,157 837 5.68

The main way to reduce buffer busy waits is to reduce the total I/O on the system. This can be done by tuning the SQL to access rows with fewer block reads (i.e., by adding indexes). Even if we have a huge db_cache_size, we may still see buffer busy waits, and increasing the buffer size won't help. The resolution of a "buffer busy wait" events is one of the most confounding problems with Oracle. In an I/O-bound Oracle system, buffer busy waits are common, as evidenced by any system with read (sequential/scattered) waits in the top-five waits. Reducing buffer busy waits reduces the total I/O on the system. This can be accomplished by tuning the SQL to access rows with fewer block reads by adding indexes, adjusting the database

writer or adding freelists to tables and indexes. Even if there is a huge db_cache_size , the DBA may still see buffer busy waits and, in this case, increasing the buffer size will not help. The most common remedies for high buffer busy waits include database writer (DBWR) contention tuning, adding freelists to a table and index, implementing Automatic Segment Storage Management (ASSM, a.k.a bitmap freelists), and, of course, and adding a missing index to reduce buffer touches. In order to look at system-wide wait events, we can query the v$system_event performance view. This view, shown below, provides the name of the wait event, the total number of waits and timeouts, the total time waited, and the average wait time per event.
select * from v$system_event where event like %wait%;
EVENT TOTAL_WAITS TOTAL_TIMEOUTS TIME_WAITED AVERAGE_WAIT --------------------------- ----------- -------------- ----------- -----------buffer busy waits 636528 1557 549700 .863591232 write complete waits 1193 0 14799 12.4048617 free buffer waits 1601 0 622 .388507183

The type of buffer that causes the wait can be queried using the v$waitstat view. This view lists the waits per buffer type for buffer busy waits, where COUNT is the sum of all waits for the class of block, and TIME is the sum of all wait times for that class: select * from v$waitstat;
CLASS COUNT TIME ------------------ ---------- ---------data block 1961113 1870278 segment header 34535 159082 undo header 233632 86239 undo block 1886 1706

Buffer busy waits occur when an Oracle session needs to access a block in the buffer cache, but cannot because the buffer copy of the data block is locked. This buffer busy wait condition can happen for either of the following reasons:

The block is being read into the buffer by another session, so the waiting session must wait for the block read to complete. Another session has the buffer block locked in a mode that is incompatible with the waiting session's request.

Because buffer busy waits are due to contention between particular blocks, there's nothing you can do until you know which blocks are in conflict and why the conflicts are occurring. Tuning

therefore involves identifying and eliminating the cause of the block contention. The v$session_wait performance view, shown below, can give some insight into what is being waited for and why the wait is occurring.
SQL> desc v$session_wait Name Null? ----------------------------------------- -------SID SEQ# EVENT P1TEXT P1 P1RAW P2TEXT P2 P2RAW P3TEXT P3 P3RAW WAIT_TIME SECONDS_IN_WAIT STATE Type -----------NUMBER NUMBER VARCHAR2(64) VARCHAR2(64) NUMBER RAW(4) VARCHAR2(64) NUMBER RAW(4) VARCHAR2(64) NUMBER RAW(4) NUMBER NUMBER VARCHAR2(19)

The columns of the v$session_wait view that are of particular interest for a buffer busy wait event are:

P1The absolute file number for the data file involved in the wait. P2The block number within the data file referenced in P1 that is being waited upon. P3The reason code describing why the wait is occurring.

Here's an Oracle data dictionary query for these values:

select p1 "File #". p2 "Block #", p3 "Reason Code" from v$session_wait where event = 'buffer busy waits';

If the output from repeatedly running the above query shows that a block or range of blocks is experiencing waits, the following query should show the name and type of the segment:
select owner, segment_name, segment_type from

dba_extents where file_id = &P1 and &P2 between block_id and block_id + blocks -1;

Once the segment is identified, the v$segment_statistics performance view facilitates real-time monitoring of segment-level statistics. This enables a DBA to identify performance problems associated with individual tables or indexes, as shown below.
select object_name, statistic_name, value from V$SEGMENT_STATISTICS where object_name = 'SOURCE$';

OBJECT_NAME ----------SOURCE$ SOURCE$ SOURCE$ SOURCE$ SOURCE$ SOURCE$ SOURCE$ SOURCE$ SOURCE$

STATISTIC_NAME ------------------------logical reads buffer busy waits db block changes physical reads physical writes physical reads direct physical writes direct ITL waits row lock waits

VALUE ---------11216 210 32 10365 0 0 0 0

We can also query the dba_data_files to determine the file_name for the file involved in the wait by using the P1 value from v$session_wait for the file_id.
SQL> desc dba_data_files Name Null? ----------------------------------------- -------FILE_NAME FILE_ID TABLESPACE_NAME BYTES BLOCKS STATUS RELATIVE_FNO AUTOEXTENSIBLE MAXBYTES MAXBLOCKS INCREMENT_BY USER_BYTES USER_BLOCKS Type ----------VARCHAR2(513) NUMBER VARCHAR2(30) NUMBER NUMBER VARCHAR2(9) NUMBER VARCHAR2(3) NUMBER NUMBER NUMBER NUMBER NUMBER

Interrogating the P3 (reason code) value from v$session_wait for a buffer busy wait event will

tell us why the session is waiting. The reason codes range from 0 to 300 and can be decoded.

Code -

Reason for wait A modification is happening on a SCUR or XCUR buffer but has not yet completed. The block is being read into the buffer cache. We want to NEW the block, but the block is currently being read by another session (most likely for undo). We want the CURRENT block either shared or exclusive but the block is being read into cache by another session, so we have to wait until its read() is completed. We want to get the block in current mode, but someone else is currently reading it into the cache. Wait for the user to complete the read. This occurs during buffer lookup. Block is being read by another session, and no other suitable block image was found, so we wait until the read is completed. This may also occur after a buffer cache assumed deadlock. The kernel can't get a buffer in a certain amount of time and assumes a deadlock. Therefore it will read the CR version of the block. We want to NEW the block, but someone else is using the current copy, so we have to wait for that user to finish. The session wants the block in SCUR or XCUR mode. If this is a buffer exchange or the session is in discrete TX mode, the session waits for the first time and the second time escalates the block as a deadlock, so does not show up as waiting very long. In this case, the statistic: "exchange deadlocks" is incremented, and we yield the CPU for the "buffer deadlock" wait event. During buffer lookup for a CURRENT copy of a buffer, we have found the buffer but someone holds it in an incompatible mode, so we have to wait. Trying to get a buffer in CR/CRX mode, but a modification has started on the buffer that has not yet been completed. CR/CRX scan found the CURRENT block, but a modification has started on the buffer that has not yet been completed.

0 100

110

120

130

200

210

220

230

231

Reason codes As I mentioned at the beginning of this article, buffer busy waits are prevalent in I/O-bound systems. I/O contention, resulting in waits for data blocks, is often due to numerous sessions repeatedly reading the same blocks, as when many sessions scan the same index. In this scenario, session one scans the blocks in the buffer cache quickly, but then a block has to be read from disk. While session one awaits the disk read to complete, other sessions scanning the same index soon catch up to session one and want the same block currently being read from disk. This is where the buffer busy wait occurswaiting for the buffer blocks that are being read from disk. The following rules of thumb may be useful for resolving each of the noted contention situations:

Data block contentionIdentify and eliminate HOT blocks from the application via changing PCTFREE and or PCTUSED values to reduce the number of rows per data block. Check for repeatedly scanned indexes. Since each transaction updating a block requires a transaction entry, increase the INITRANS value. Freelist block contentionIncrease the FREELISTS value. Also, when using Parallel Server, be certain that each instance has its own FREELIST GROUPs. Segment header contentionAgain, increase the number of FREELISTs and use FREELIST GROUPs, which can make a difference even within a single instance. Undo header contentionIncrease the number of rollback segments.

Mark Bobak notes that buffer busy waits with P3=0 indicate disk I/O contention, indicating the freelists will not improve concurrency: "buffer busy wait w/ P3=0 means the buffer is locked because the contents are being read from disk by another session. (See MetaLink Doc ID 34405.1 for more details.) This is most likely caused by multiple, concurrent sessions that are reading the same table or set of tables. In my experience, it's most often due to multiple, concurrent queries doing full table scans on the same table. Since this a read concurrency problem, changing freelists will NOT help." In these cases, buffering-up the tables (e.g. KEEP pool), or using faster storage (SSD) can remove this disk enqueue wait evsnts.

The v$segment_statistics view is a goldmine for funding wait events that are associated with a specific Oracle table. I wrote the following script to show run-time details about a segment (usually a table or an index), and this powerful script interrogates the v$segment_statistics view use a CASE statement. If you examine the script below you will see that the v$segment_statistics view is grouped by object_name. For each object, we display counts of the major object wait events, as seen in the CASE expression. The most important of these objectlevel wait events will give us clues into the source of the contention. For example, buffer busy waits and ITL waits all have a clear set of causes, and knowing this information is critical to understanding the root cause of your contention. For example, the most common remedies for high buffer busy waits include database writer (DBWR) contention tuning, adding freelists (or ASSM), and adding missing indexes. When reviewing objects for possible tuning issues, it is handy to have statistics such as the number of internal transaction list (ITL) waits, buffer busy waits and row lock waits that the object has experienced. Combined with the number of logical and physical reads the object has experienced, the above statistics give a complete picture of the usage of the object in question. The v$segment_statistics provides a statistic_name and value column for each table, unfortunately this format doesnt lend itself to easy use. By utilizing the crosstab technique we can easily create a report to show us these vital tuning statistics for our system. An example of this type of cross tab report is shown below. obj.xtab.sql
-- Crosstab of object and statistic for an owner -col "Object" format a20 set numwidth 12 set lines 132 set pages 50 @title132 'Object Wait Statistics' spool rep_out\&&db\obj_stat_xtab select * from ( select DECODE (GROUPING(a.object_name), 1, 'All Objects', a.object_name) AS "Object", sum(case when a.statistic_name = 'ITL waits' then a.value else null end) "ITL Waits", sum(case when a.statistic_name = 'buffer busy waits' then a.value else null end) "Buffer Busy Waits", sum(case when a.statistic_name = 'row lock waits' then a.value else null end) "Row Lock Waits", sum(case when a.statistic_name = 'physical reads' then a.value else null end) "Physical Reads", sum(case when a.statistic_name = 'logical reads' then a.value else null end) "Logical Reads"

from v$segment_statistics a where a.owner like upper('&owner') group by rollup(a.object_name)) b where (b."ITL Waits">0 or b."Buffer Busy Waits">0) / spool off clear columns ttitle off

The cross tab report generates a listing showing the statistics of concern as headers across the page rather than listings going down the page and summarizes them by object. This allows us to easily compare total buffer busy waits to the number of ITL or row lock waits. This ability to compare the ITL and row lock waits to buffer busy waits lets us see what objects may be experiencing contention for ITL lists, which may be experiencing excessive locking activity and through comparisons, which are highly contended for without the row lock or ITL waits. AN example of the output of the report, edited for length, is shown below.

Object ITL Waits Buffer Busy Waits Row Lock Waits Physical Reads Logical Reads ------------- --------- ----------------- -------------- -------------- ------------BILLING 0 63636 38267 1316055 410219712 BILLING_INDX1 1 16510 55 151085 21776800 ... DELIVER_INDX1 1963 36096 32962 1952600 60809744 DELIVER_INDX2 88 16250 9029 18839481 342857488 DELIVER_PK 2676 99748 29293 15256214 416206384 ... All Objects 12613 20348859 1253057 1139977207 20947864752

In the above report the BILLING_INDX1 index has a large amount of buffer busy waits but we cant account for them from the ITL or Row lock waits, this indicates that the index is being constantly read and the blocks then aged out of memory forcing waits as they are re-read for the next process. On the other hand, almost all of the buffer busy waits for the DELIVER_INDX1 index can be attributed to ITL and Row Lock waits. In situations where there are large numbers of ITL waits we need to consider the increase of the INITRANS setting for the table to remove this source of contention. If the predominant wait is row lock waits then we need to determine if we are properly using locking and cursors in our application (for example, we may be over using the SELECTFOR UPDATE type code.) If, on the other hand all the waits are un-accounted for buffer busy waits, then we need to consider increasing the amount of database block buffers we have in our SGA. As you can see, this object wait cross tab report can be a powerful addition to our tuning arsenal

Introduction
In my 16 years of working with Oracle I have seen the database grow and change as new features are added and old ones removed or changed. However, even as things change, they remain the

same in many respects. We still must index properly, set memory settings correctly and specify proper hardware. Welcome to My Nightmare The Common Performance Errors in Oracle Databases

Improper Memory Configuration

If you put too-small a carburetor on a car then even though the engine may be able to do 200 MPH, you are constraining it to much less performance. Likewise if you do not give enough memory to Oracle you will prevent it from reaching its full performance potential. In this section we will discuss two major areas of memory, the database buffer area and the shared pool area. The PGA areas are discussed in a later section. The Database Buffer Area Just like the old adage you cant fly anywhere unless you go through Atlanta, you arent going to get data unless you go through the buffer. Admittedly there are some direct-read scenarios, but for the most part anything that goes to users or gets into the database must go through the database buffers. Gone are the days of a single buffer area (the default) now we have 2, 4, 8,, 16, 32 K buffer areas, keep and recycle buffer pools on top of the default area. Within these areas we have the consistent read, current read, free, exclusive current, and many other types of blocks that are used in Oracles multi-block consistency model. The V$BH view (and its parent the X$BH table) are the major tools used by the DBA to track block usage, however, you may find that the data in the V$BH view can be misleading unless you also tie in block size data. Look at Figure 10. For complete scripts, see my complete Oracle script collection at www.dbaoracle.com/oracle_scripts.htm.
rem vbh_status.sql rem rem Mike Ault -- Burleson rem col dt new_value td noprint select to_char(sysdate,'ddmmyyyyhh24miss') dt from dual; @title80 'Status of DB Block Buffers' spool rep_out\&db\vbh_status&&td select status,count(*) number_buffers from v$bh group by status; spool off ttitle off clear columns

Figure 10: Simple V$BH Report

In the report in Figure 10 we see a simple version of a V$BH query. Figure 10 assumes only one buffer is in play, the default buffer, and doesnt account for any of the multiple blocksize areas or the recycle or keep areas. By not accounting for other types of buffers that may be present the report in Figure 10 can overstate the number of free buffers available. Look at Figure 11.
Date: 12/13/05 Time: 10:38 PM database STATU NUMBER_BUFFERS ----- -------------cr 33931 free 15829 xcur 371374 Status of DB Block Buffers whoville Page: 1 PERFSTAT

Figure 11: Simple V$BH report listing

From the results in Figure 11 we would conclude we had plenty of free buffers, however we would be mistaken. Look at the report in Figure 12.
Date: 12/13/05 Time: 10:39 PM database STATUS NUM --------- ---------32k cr 2930 32k xcur 29064 8k cr 1271 8k free 3 8k read 4 8k xcur 378747 free 10371 All Buffers Status whoville Page: 1 PERFSTAT

Figure 12: Detailed V$BH Status report As you can see, while there are free buffers, only 3 of them are available to the 8k, default area and none are available to our 32K area. The free buffers are actually assigned to a keep or recycle pool area (hence the null value for the blocksize) and are not available for normal usage. The script to generate this report is shown in Figure 13. For complete scripts, see my complete Oracle script collection at www.dbaoracle.com/oracle_scripts.htm.

set pages 50 @title80 'All Buffers Status' spool rep_out\&&db\all_vbh_status select '32k '||status as status, count(*) as num from v$bh where file# in( select file_id from dba_data_files where tablespace_name in ( select tablespace_name from dba_tablespaces where block_size=32768)) group by '32k '||status union select '16k '||status as status, count(*) as num from v$bh where file# in( select file_id from dba_data_files where tablespace_name in ( select tablespace_name from dba_tablespaces where block_size=16384)) group by '16k '||status union select '8k '||status as status, count(*) as num from v$bh where file# in( select file_id from dba_data_files where tablespace_name in ( select tablespace_name from dba_tablespaces where block_size=8192)) group by '8k '||status union select '4k '||status as status, count(*) as num from v$bh where file# in( select file_id from dba_data_files where tablespace_name in (

select tablespace_name from dba_tablespaces where block_size=4096)) group by '4k '||status union select '2k '||status as status, count(*) as num from v$bh where file# in( select file_id from dba_data_files where tablespace_name in ( select tablespace_name from dba_tablespaces where block_size=2048)) group by '2k '||status union select status, count(*) as num from v$bh where status='free' group by status order by 1 / spool off ttitle off

Figure 13: Script to get all Buffer Pool Status As you can see, the script is wee bit more complex than the simple V$BH script. No doubt there is a clever way to simplify the script using X and K$ tables, but then we would have to use the SYS user to run it and I prefer to use lower powered users when I go to client sites. So, if you see buffer busy waits, db block waits and the like and you run the above report and see no free buffers it is probably a good bet you need to increase the number of available buffers for the area showing no free buffers. You should not immediately assume you need more buffers because of buffer busy waits as these can be caused by other problems such as row lock waits, itl waits and other issues. Luckily Oracle10g has made it relatively simple to determine if we have these other types of waits. Look at Figure 14. For complete scripts, see my complete Oracle script collection at www.dbaoracle.com/oracle_scripts.htm.
-- Crosstab of object and statistic for an owner -col "Object" format a20

set numwidth 12 set lines 132 set pages 50 @title132 'Object Wait Statistics' spool rep_out\&&db\obj_stat_xtab select * from( select DECODE(GROUPING(a.object_name), 1, 'All Objects', a.object_name) AS "Object", sum(case when a.statistic_name = 'ITL waits' then a.value else null end) "ITL Waits", sum(case when a.statistic_name = 'buffer busy waits' then a.value else null end) "Buffer Busy Waits", sum(case when a.statistic_name = 'row lock waits' then a.value else null end) "Row Lock Waits", sum(case when a.statistic_name = 'physical reads' then a.value else null end) "Physical Reads", sum(case when a.statistic_name = 'logical reads' then a.value else null end) "Logical Reads" from v$segment_statistics a where a.owner like upper('&owner') group by rollup(a.object_name)) b where (b."ITL Waits">0 or b."Buffer Busy Waits">0) / spool off clear columns ttitle off

Figure 14: Object Statistic Crosstab Report Figure 14 shows an object statistic cross tab report based on the V$SEGMENT_STATISTICS view. The cross tab report generates a listing showing the statistics of concern as headers across the page rather than listings going down the page and summarizes them by object. This allows us to easily compare total buffer busy waits to the number of ITL or row lock waits. This ability to compare the ITL and row lock waits to buffer busy waits lets us see what objects may be experiencing contention for ITL lists, which may be experiencing excessive locking activity and through comparisons, which are highly contended for without the row lock or ITL waits. An example of the output of the report, edited for length, is shown in Figure 15. Date: 12/09/05 Page: 1 Time: 07:17 PM Object Wait Statistics PERFSTAT whoville database ITL Buffer Busy Row Lock Physical Logical Object Waits Waits Waits Reads Reads -------------- ----- ----------- -------- ---------- ----------BILLING 0 63636 38267 1316055 410219712 BILLING_INDX1 1 16510 55 151085 21776800 ... DELIVER_INDX1 1963 36096 32962 1952600 60809744 DELIVER_INDX2 88 16250 9029 18839481 342857488 DELIVER_PK 2676 99748 29293 15256214 416206384

DELIVER_INDX3 2856 104765 31710 8505812 467240320 ... All Objects 12613 20348859 1253057 1139977207 20947864752 243 rows selected. Figure 15: Example Object Cross Tab Report

In the above report the BILLING_INDX1 index has a large number of buffer busy waits but we cant account for them from the ITL or Row lock waits, this indicates that the index is being constantly read and the blocks then aged out of memory forcing waits as they are re-read for the next process. On the other hand, almost all of the buffer busy waits for the DELIVER_INDX1 index can be attributed to ITL and Row Lock waits. In situations where there are large numbers of ITL waits we need to consider the increase of the INITRANS setting for the table to remove this source of contention. If the predominant wait is row lock waits then we need to determine if we are properly using locking and cursors in our application (for example, we may be over using the SELECTFOR UPDATE type code.) If, on the other hand all the waits are un-accounted for buffer busy waits, then we need to consider increasing the amount of database block buffers we have in our SGA. As you can see, this object wait cross tab report can be a powerful addition to our tuning arsenal.

By knowing how our buffers are being used and seeing exactly what waits are causing our buffer wait indications we can quickly determine if we need to tune objects or add buffers, making sizing buffer areas fairly easy.

But what about the Automatic Memory Manager in 10g? It is a powerful tool for DBAs with systems that have a predictable load profile, however if your system has rapid changes in user and memory loads then AMM is playing catch up and may deliver poor performance as a result. In the case of memory it may be better to hand the system too much rather than just enough, just in time (JIT). As many companies have found when trying the JIT methodology in their manufacturing environment it only works if things are easily predictable. The AMM is utilized in 10g by setting two parameters, the SGA_MAX_SIZE and the SGA_TARGET. The Oracle memory manager will size the various buffer areas as needed within the range between base settings or SGA_TARGET and SGA_MAX_SIZE using the SGA_TARGET setting as an optimal and the SGA_MAX_SIZE as a maximum with the manual settings used in some cases as a minimum size for the specific memory component.