Oracle Press Case Study - Diagnosing Another Buffer Busy Waits Issue

Case Study: Diagnosing Another Buffer Busy Waits Issue
Authors: Stephan Haisley, Consulting Technical Advisor, Oracle Corporation

Editor: Vickie Carbonneau, Principal Support Engineer, Oracle Corporation
Skill Level Rating for this Case Study: Intermediate
About Oracle Case Studies

Oracle Case Studies are intended as learning tools and for sharing information or
knowledge related to a complex event, process, procedure, or to a series of related
events. Each case study is written based upon the experience that the writer/s
encountered.
Each Case Study contains a skill level rating. The rating provides an indication of what
skill level the reader should have as it relates to the information in the case study.
Ratings are:
• Expert: significant experience with the subject matter

• Intermediate: some experience with the subject matter
• Beginner: little experience with the subject matter
Case Study Abstract

This case study details how to diagnose a buffer busy wait issue on a table undergoing
large amounts of concurrent inserts. This is typical of a table being used for web-based
OLTP transactions where sales data is being recorded. The recommended solution for
such problems involves increasing the number of process freelists for the table to spread
out the use of available datablocks. The problem presented in this case study persisted
when this solution was implemented.
When diagnosing the problem several diagnostic events were set to highlight which part
of the freelist search algorithm was causing the buffer contention. These events should
not be set unless advised by Oracle Support Services.
Case History
The issue reported by customer in the case study was:
We have a big 1.4TB table and every day the application inserts around 6 to 9 million
rows.
The row contains a long raw column and is around 3KB. During the time of the inserts
we see lots of sessions waiting on buffer busy waits on the same datablocks.
The customer reported they were carrying out the inserts from a number of concurrent
sessions, and had set the number of process freelists on the table to 23. The table involved
had the following structure:
Name Null? Type

---------------------------------------- -------- ---------
ID NOT NULL NUMBER(38)
X_SIZE NOT NULL NUMBER(4)
Y_SIZE NOT NULL NUMBER(4)
BEGIN_DATE NOT NULL DATE
FINISH_DATE NOT NULL DATE
PICTURE NOT NULL LONG RAW
PICTURE_LEN NOT NULL NUMBER(38)
Three indexes existed on the table:
1. Unique index on ID column

2. Non unique index on BEGIN_DATE column
3. Non unique index on FINISH_DATE column
It was reported that these indexes showed no signs of contention, and they could always
see one session waiting on a db file sequential read and all others are waiting on a buffer
busy wait with reason code 120, which indicates the session is waiting for the block to be
read into the buffer cache.
This table had been created some time ago so the history of why the buffer busy waits
issue had suddenly become a problem was simply explained as an increase in data
volume being inserted.
The puzzling thing was that processes are supposed to map to different process freelists
based on the following algorithm:
Process free list entry = (P % NFL) + 1

where P : Oracle PID of the process (index in V$PROCESS),
and NFL : Process free lists number as defined by FREELISTS
If no free blocks can be found on the assigned process freelist (PFL) then the process will
search the Master Freelist (MFL) for space. If no space is found, committed Transaction
Freelists are merged into the MFL and those blocks are scanned for usefulness. If still no
Page 2
space is found the High Water Mark is incremented and the new blocks can be moved to
the PFL and used. The customer was reporting all processes waiting on the same
datablocks whilst carrying out the insert statements, which indicates some problem with
the freelist search mechanism since the multiple process freelists should have mitigated
this kind of contention. This is where the data gathering and analysis begins.
The customer's database version used in this case study is 8.1.7.4 on Solaris. However,
this issue could occur on any platform with database versions 8.0 or higher.
Analysis
Summary
A number of different data items were collected to help determine a cause. These
included:
• Statspack reports to show what the largest contention points were for the
database
• I/O statistics from the operating system to see if we were running into bad I/O
times slowing down the freelist search
• Datablock dumps for the segment header and other blocks being waited on to
see how the freelist lists were changing
• Several diagnostic events that dump information to a trace file and show how
the freelist search mechanism was working
Using the collected data it was possible to build a test case that reproduced the
same problem on a multiple CPU box using concurrent sessions inserting into the
same table. The test case did in fact show a problem with how the freelist search
mechanism is a single point of contention when there are few rows per block and
a high concurrency level of DML activity. An in-depth review of the collected
data is provided in the next section.
Detailed Analysis
The following sections describe how each piece of data was collected and what it shows.
Once all the data has been described a final cause determination will be presented, tying
everything together.
Page 3
The statspack report clearly showed massive I/O contention issues:
Top 5 Wait Events

~~~~~~~~~~~~~~~~~ Wait % Total
Event Waits Time (cs) Wt Time
-------------------------------------------- ------------ ------------ -------
db file sequential read 3,045,656 5,483,001 82.74
buffer busy waits 1,253,698 1,013,605 15.30
latch free 100,955 95,862 1.45
log file sync 30,198 13,384 .20
db file parallel write 19,057 8,318 .13
Looking further down the statspack report it is clear which tablespace is having all the
waits against it:
Tablespace IO Summary for DB:
Avg Read Total Avg Wait

Tablespace Reads (ms) Writes Waits (ms)
------------------------- ----------- -------- ----------- ---------- --------
DATA01 2,670,571 19.2 58,417 1,244,996 8.1
INDX01 378,959 18.6 9,345 2,513 9.4
RBS02 14 ###### 6,648 89 0.9
USERS01 248 113.1 248 0 0.0
SYSTEM 39 181.3 7 0 0.0
TEMP02 3 ###### 2 0 0.0
Ideally the average read times should be at a maximum of 10-20ms, so although the
DATA01 tablespace is in the upper band, it is still not a cause for why the customer is
reporting large waits for buffer busy waits. The datafile I/O statistics section of the
statspack reports showed many of the datafiles with waits against them, and four of the
fourteen disks being used for the datafiles showed average read times of 20-25ms.
I/O statistics from the operating system using vxstat also showed the same four disks with
a higher average read time than the rest. The other disks also showed relatively high read
times:
OPERATIONS BLOCKS AVG TIME(ms)

TYP NAME READ WRITE READ WRITE READ WRITE
Tue Jun 07 11:01:00 2005

vol data01 10620 445 169920 7120 24.7 5.5
vol data02 10148 231 162368 3696 25.7 5.3
vol data03 11559 221 184944 3536 24.6 5.3
vol data04 12018 202 192288 3232 26.7 4.5
vol data05 13061 199 208976 3184 23.0 4.8
vol data06 12254 215 196064 3440 23.8 6.0
vol data07 13153 187 210448 2992 23.9 4.4
vol data08 18047 275 288752 4400 23.9 4.7
vol data09 16159 474 258544 7584 23.6 5.2
vol data10 16492 355 263872 5680 29.6 6.5
vol data11 15840 275 253440 4400 26.0 3.8
vol data12 29629 190 474064 3040 44.2 19.2
vol data13 35560 174 568960 2784 41.8 12.8
vol data14 28231 127 451696 2032 36.3 14.0
Page 4
The I/O statistics certainly indicate a performance problem but do not explain the earlier
reports of buffer busy waits on the same blocks, contradicting with the expected way in
which process freelists spread out the use of datablocks amongst processes.
It was decided by the customer that we would assume the I/O was certainly not helping
performance issues but they wanted to concentrate on why the process freelists did not
appear to be used correctly. The investigation moved in this direction.
The buffer busy wait statistics for the class of datablock block being waited on is shown
in the statspack:
Buffer wait Statistics

Tot Wait Avg
Class Waits Time (cs) Time (cs)
------------------ ----------- ---------- ---------
data block 1,224,478 1,011,605 1
segment header 19,946 753 0
undo header 79 6 0
undo block 10 2 0
This indicates we have some hot block issue, so we needed to find out what these hot
blocks are and why they appear so hot.
In order to find the hot block(s) v$session_wait was queried to show which blocks were
being contended for. The results proved interesting as the particular block being sought
was always changing:
select b.sid,b.username,event,wait_time,p1,p2,p3,b.sql_hash_value,b.status
from v$session_wait a,v$session b
where event not like 'SQL*Net message%'
and event not like 'rdbms%' and a.sid=b.sid
and b.sql_hash_value=4290940428
and b.sid>8
order by sql_hash_value;
SID USERNAME EVENT WAIT_TIME P1 P2 P3

---------- ---------- ------------------ --------- ------ ---------- ----------
16 GALLERY buffer busy waits 0 42 249961 120
226 GALLERY db file sequential read 0 42 249961 1
SQL> /
1 In v$session_wait, the P1, P2, and P3 columns identify the file number, block number, and buffer busy reason
codes, respectively.
Page 5
SID USERNAME EVENT WAIT_TIME P1 P2 P3
---------- ---------- ------------------ --------- ------ ---------- ----------
111 GALLERY db file sequential read 0 257 101465 1
Note: In v$session_wait, the P1, P2, and P3 columns identify the file number, block
number, and buffer busy reason codes, respectively
We then needed to find out which object these blocks belonged to so we could dump the
segment header over a period of time to see if the freelists were changing:
SELECT owner, segment_name, segment_type

FROM dba_extents
WHERE file_id=42 AND 249961 BETWEEN block_id AND block_id+blocks-1;
OWNER SEGMENT_NAME SEGMENT_TYPE

------------ ------------ ------------
GALLERY ITEMS TABLE
Note: Each time we ran this query with the blocks being waited on it was always this
same table.
Dumping the segment header showed that the process freelists were being used as
expected, because all the freelist structures were changing over time:
SQL> SELECT header_file, header_block

FROM dba_segments WHERE owner=’GALLERY’ AND segment_name = ‘ITEMS’;
HEADER_FILE HEADER_BLOCK
------------ ------------
14 2
SQL> ALTER SYSTEM DUMP DATAFILE 14 BLOCK 2;
-- WAIT a few second
SQL> ALTER SYSTEM DUMP DATAFILE 14 BLOCK 2;
The dump shows (only selected information shown):
nfl = 23, nfb = 1 typ = 1 nxf = 177 ccnt = 235781012

SEG LST:: flg: USED lhd: 0xd9c2e918 ltl: 0xdf80c500 <- Master Freelist
SEG LST:: flg: USED lhd: 0x09c28b4b ltl: 0x00825c2d <- Process Freelist #1
SEG LST:: flg: USED lhd: 0xe583a1ea ltl: 0xac02742c
SEG LST:: flg: USED lhd: 0x66c37711 ltl: 0x468034f9
SEG LST:: flg: USED lhd: 0x5502bfed ltl: 0x42c2c252
SEG LST:: flg: USED lhd: 0x00000000 ltl: 0x00000000
SEG LST:: flg: USED lhd: 0x1d8081ce ltl: 0x1503c119
Page 6
SEG LST:: flg: USED lhd: 0x09c29f25 ltl: 0x09c29f25
SEG LST:: flg: USED lhd: 0xa5822a00 ltl: 0xa5822a00
SEG LST:: flg: USED lhd: 0xa0036033 ltl: 0x33038d14
SEG LST:: flg: USED lhd: 0xa1c0d61d ltl: 0x8702d4dc
SEG LST:: flg: USED lhd: 0x9483ab8e ltl: 0x0f83baec
SEG LST:: flg: USED lhd: 0x6ac071bb ltl: 0x61c23e99
SEG LST:: flg: USED lhd: 0xd401cd2c ltl: 0xaa02eabb
SEG LST:: flg: USED lhd: 0x97432a9a ltl: 0x84c3cced
SEG LST:: flg: USED lhd: 0xa88062fb ltl: 0x9483911c
SEG LST:: flg: USED lhd: 0x79436ef5 ltl: 0x790102af
SEG LST:: flg: USED lhd: 0x8000b9d1 ltl: 0x1a42881d
SEG LST:: flg: USED lhd: 0x1cc20fc7 ltl: 0x19005a17 <- Process Freelist #23
XCT LST:: flg: UNUSED lhd: 0x00000000 ltl: 0x00000000 xid:0x0000.000.00000000
... <- Transaction freelists populated when a transaction frees more blocks
XCT LST:: flg: USED lhd: 0x0d03d289 ltl: 0x0d03d289 xid:0x0001.00e.00870924
XCT LST:: flg: USED lhd: 0x2900d4fd ltl: 0x2900d4fd xid:0x0004.05e.006b4288
XCT LST:: flg: USED lhd: 0x1d42e271 ltl: 0x1d42e271 xid:0x0002.052.0086f6c3
XCT LST:: flg: USED lhd: 0xda41e80c ltl: 0xda425b83 xid:0x0004.018.006b42f4
End dump data blocks tsn: 4 file#: 14 minblk 2 maxblk 2
The second dump showed:
nfl = 23, nfb = 1 typ = 1 nxf = 177 ccnt = 235781120

SEG LST:: flg: USED lhd: 0xd60242c5 ltl: 0xdf80c500 <- MFL header changed
SEG LST:: flg: USED lhd: 0xde413a87 ltl: 0xd103aa92 <- PFL #1 changed
SEG LST:: flg: USED lhd: 0xe583a1ea ltl: 0xac02742c
SEG LST:: flg: USED lhd: 0x66c37711 ltl: 0x468034f9
SEG LST:: flg: USED lhd: 0x5502bfed ltl: 0x42c2c252
SEG LST:: flg: USED lhd: 0xdac13e6b ltl: 0xdac13e6b <- PFL #5 changed
SEG LST:: flg: USED lhd: 0x00000000 ltl: 0x00000000 <- PFL #6 changed
SEG LST:: flg: USED lhd: 0x3603b1a1 ltl: 0x3603b1a1 <- PFL #9 changed
SEG LST:: flg: USED lhd: 0x29c010c8 ltl: 0x2543d499 <- PFL #13 changed
SEG LST:: flg: USED lhd: 0xb3c27294 ltl: 0xb3c27294 <- PFL #14 changed
SEG LST:: flg: USED lhd: 0xd401cd2c ltl: 0xaa02eabb
SEG LST:: flg: USED lhd: 0x94825278 ltl: 0x4103444f <- PFL #17 changed
SEG LST:: flg: USED lhd: 0x2242077d ltl: 0xd8000f08 <- PFL #19 changed
SEG LST:: flg: USED lhd: 0xdbc351d6 ltl: 0x1a42881d <- PFL #21 changed
SEG LST:: flg: USED lhd: 0x9b00c830 ltl: 0x8d009d0e <- PFL #22 changed
SEG LST:: flg: USED lhd: 0x1cc20fc7 ltl: 0x19005a17
XCT LST:: flg: UNUSED lhd: 0x00000000 ltl: 0x00000000 xid:0x0000.000.00000000
. . .
XCT LST:: flg: USED lhd: 0x0d03d289 ltl: 0x0d03d289 xid:0x0001.00e.00870924
XCT LST:: flg: USED lhd: 0x2900d4fd ltl: 0x2900d4fd xid:0x0004.05e.006b4288
XCT LST:: flg: USED lhd: 0x1d42e271 ltl: 0x1d42e271 xid:0x0002.052.0086f6c3
XCT LST:: flg: USED lhd: 0xda41e80c ltl: 0xda425b83 xid:0x0004.018.006b42f4
End dump data blocks tsn: 4 file#: 14 minblk 2 maxblk 2
Page 7
The transaction freelists (TFL) are emptied when a process cannot find any free blocks on
the MFL (as described earlier). The fact that the TFLs have not been emptied implies the
MFL has always managed to supply enough blocks for the requesting processes or all
transactions are still uncommitted (which seems unlikely).
The second segment header dump shows 15 out of the 23 process freelists have changed,
including the master freelist. The tail of the master freelist has not changed, but the
header has, which indicates the master freelist has enough free blocks on it to satisfy all
searches within the monitored time period. The fact that many of the freelists are
changing indicates the process freelist assignment is working correctly. What this data
doesn’t prove is if the searching mechanism is working correctly and it certainly doesn’t
highlight any cause to the buffer busy waits issue.
In order to find out more about what was happening with several of the waiting sessions
during the free space search, we set a few diagnostic events to gather tracing information.
The events that were used:
Event Level Reason

10320 1 Trace the getting of a block to be used after free space search
10022 1 Trace which process freelist and block is found during search
10085 1 Trace when blocks moved from TFL to MFL
10082 1 Trace part of free space search
10080 1 Trace changing of freelist (removing blocks from list)
10046 12 Trace SQL statements with wait and bind data
Note: It is only recommended to set the freelist tracing events under advice from
Oracle Support. These events can produce a large amount of trace data so setting
them should be done only for short periods of time.
The customer was instructed to set the events for three waiting sessions when they saw a
high number of buffer busy waits with the 120 reason code. After tracing for 5 minutes,
all the events were turned off. The PL/SQL code used to enable and disable the events is
listed below.
create table tracing(sid number, serial# number, event number) tablespace

users; -- change tablespace if different
create or replace procedure trace_freelists(what IN NUMBER, onoff IN number)

as
cursor c1 is -- retrieve top 3 sessions ordered by time_waited for buffer
busy waits
SELECT s.sid , s.serial#
FROM v$session s,v$session_event se
WHERE
s.sid = se.sid
and se.event = 'buffer busy waits'
and s.sid>8
and s.server = 'DEDICATED'
and rownum<4
ORDER BY se.time_waited desc;
Page 8
cursor c2 (wevent NUMBER) is
select sid,serial# from tracing where event=wevent;
BEGIN
if what=10320 -- Freelist tracing
then
if onoff=1 -- Turn freelist tracing ON
then
for rec in c1 loop
dbms_system.set_ev(rec.sid,rec.serial#,10320,1,'');
insert into tracing values (rec.sid, rec.serial#, 10320);
commit;
end loop;
elsif onoff=0 -- Turn freelist tracing OFF
then
for rec in c2(10320) loop
delete from tracing where sid=rec.sid and event=10320;
commit;
end loop;
end if;
elsif what=10022 -- Freelist 10022 tracing
then
then
commit;
end loop;
then
commit;
end loop;
end if;
then
then
commit;
end loop;
then
commit;
end loop;
end if;
then
then
Page 9
commit;
end loop;
then
commit;
end loop;
end if;
then
then
commit;
end loop;
then
commit;
end loop;
end if;
elsif what=10046 -- SQL Trace - SHOULD BE TURNED ON AFTER 10320 TRACE
then
if onoff=1 -- Turn SQL trace ON
then
commit;
end loop;
elsif onoff=0 -- Turn SQL trace OFF
then
commit;
end loop;
end if;
end if;
end trace_freelists;
/
To enable the events:
exec trace_freelists(10320,1);
exec dbms_lock.sleep(10); -- need to wait 10secs for next event to work
exec dbms_lock.sleep(10);
Wait 5 minutes then turn off each event in the following order:
Page 10
The trace files generated confirmed the sessions were assigned different process freelists
and use different blocks for some of the inserts:
Session #1:
*** 2005-07-13 05:56:24.594
KTSGSP: flag = 0xa7, seg free list = 7, tsn = 4 block = 0xb1c3721d
*** 2005-07-13 05:56:24.655
KTSGSP: flag = 0x24, seg free list = 7, tsn = 4 block = 0xca030308
Session #2:
*** 2005-07-13 05:56:11.672
KTSGSP: flag = 0xa7, seg free list = 9, tsn = 4 block = 0xbf02e415
*** 2005-07-13 05:56:11.730
KTSGSP: flag = 0xa7, seg free list = 9, tsn = 4 block = 0xb4c265e7
Session #3:
*** 2005-07-13 05:56:05.362
KTSGSP: flag = 0xa7, seg free list = 20, tsn = 4 block = 0xabc191fa
*** 2005-07-13 05:56:05.369
KTSGSP: flag = 0xa7, seg free list = 20, tsn = 4 block = 0xabc17d9b
But it also showed times when the sessions would be checking the same blocks for use:
Session #1:
KDTGSP: seg:0x1c000002 wlk:0 rls:0 options:KTS_EXCHANGE KTS_UNLINK
pdba:0xbf40185d
WAIT #3: nam='buffer busy waits' ela= 2 p1=438 p2=91992 p3=120
... Waits on many different blocks with an occasional db file seq read
WAIT #3: nam='buffer busy waits' ela= 1 p1=839 p2=53567 p3=120 <- Starts
waiting on same blocks here
WAIT #3: nam='buffer busy waits' ela= 1 p1=832 p2=187781 p3=120 <- waiting on
session #2 to read the block
Page 11
...
This continues for at least another 30-40 blocks
Session #2:
KDTGSP: seg:0x1c000002 wlk:0 rls:0 options:KTS_EXCHANGE KTS_UNLINK
pdba:0x35429ced
WAIT #3: nam='buffer busy waits' ela= 1 p1=839 p2=53567 p3=120 <- Goes
straight to the MFL here as looking for common blocks
WAIT #3: nam='db file sequential read' ela= 1 p1=832 p2=187781 p3=1 <- first
process to need block so reads in
...
This continues for at least another 30-40 blocks
The second trace output shows that session #1 first starts to traverse a set of blocks that
another session (not known which session) is currently reading from disk. We know this
because it is still waiting on buffer busy waits with a p3 (reason code) value of 120. Then
it starts to traverse blocks that session #2 is also trying to read. This indicates that both
sessions are reading from the master freelist at the same time. We know this because
earlier in the trace file we saw they are assigned to different process freelists. Because all
processes will move to searching the master freelist if no suitable blocks are found on
their process freelist this highlights a possible limitation that can occur when several
process freelists are empty resulting in a number of processes trying to search the same
master freelist. The severity of this seemed unexpected so a bug (4523986) was opened to
get some input from Oracle Development. Development came back with the following
thoughts:
We attempt to move a section of the MFL in one go - in this case 5 blocks

and obtain the necessary info with shared locks. We then attempt the move,
taking an exclusive lock. If the sublist we've identified to move has been
changed, we start over.
The crux of the problem is that all sessions are reading the same five blocks at the same
time (and only one of them will eventually succeed in moving them to their PFL which
means the problem repeats on another five block list for the rest of them).
Your I/O issue is probably making this worse as presumably the cache is running slowly -
the five block lists are read in current shared mode.
With this input from development and the diagnostic data we have analyzed we can state
the cause determination as being due to allowing concurrent processes to traverse and
Page 12
manipulate the master freelist when looking for space. If the assigned process freelists are
empty of suitable blocks, they move to searching the master freelist. By default only one
master freelist is created and controlled by the segment header, which becomes the new
serialized point of contention.
Conclusion and Learnings

Now that we had confirmed a problem with serialization of searching and moving blocks
from the master freelist to a process freelist by concurrent sessions, we needed to find a
solution to relieve the buffer busy waits.
In the bug, development had first suggested increasing the

_bump_highwater_mark_count initialization parameter. By default this parameter is set
to 5, and works by bumping the HWM by 5 multiplied by the number of PFLs+1 (24 in
this case). When a process does not find any available blocks on the assigned PFL it
attempts to move 5 blocks from the MFL to the PFL. By increasing this parameter it only
increases the jump in HWM movement and not the amount of blocks being transferred to
a PFL. This still defaults to a maximum of 5 blocks, or less than 5 if an extent boundary
is reached.
I tested an increase of the _bump_highwater_mark_count parameter in a test system

using the customer test case and it didn’t decrease the buffer busy waits, but it did
increase the amount of waits on free buffer waits. This is due to bringing more blocks
into the buffer cache to be formatted when increasing the HWM by larger amounts.
Development had made some suggestions for solutions to this issue:
The three possible resolutions we had to this problem were:

1. Introducing a local enqueue to serialize the walk of the MFL and acquiring an
enqueue whenever the blocks move from MFL to PFL.
The prone problem in this resolution is:
(a) Acquiring an enqueue at this point can be very costly, and can make the
system slow.
(b) The customer would need to rebuild the whole system.
(c) Possible deadlocks.
(2) Pinning the Segment Header when walking the MFL slightly costly but could be
worked out.
(3) Aquiring block in CR mode, so that the other processes could do some work rather
than waiting.
Page 13
Possible workaround we looked at for this this problem was:
Use FREELIST GROUPs (even in single instance this can make a difference), I came
across some note stating possible resolutions for high buffer busy waits.
Freelist groups are mapped as (in a non-OPS environment or OPS and Single-Instance
environment):
Free list group is: (Process Id % Number of Free group ) + 1
Making RDBMS kernel code changes for 8.1.7.4 would provide a more comprehensive
solution for all tables that would have this high concurrency issue but also would have
further implications. For example, it may simply move contention from buffer busy waits
to enqueue waits which might cause a worse bottleneck.
In this customer’s case, the suggested workaround of using freelist groups makes perfect
sense. When an object is created it defaults to freelist groups of 1 where all the freelist
information is maintained in the segment header, as in this case. When using a value
greater than 1 additional datablocks are created after the segment header that will store a
master freelist, a number of process freelists and transaction freelists. The segment header
will only contain a single master freelist, the master of all freelists if you like. Freelist
groups were originally designed for use with OPS (Oracle Parallel Server) so that each
instance will be assigned a different freelist group and all processes connecting to that
instance will not interfere with free space searching on another instance causing blocks to
ping between them. Within a single instance environment, using freelist groups can still
provide some benefit because each process will be assigned to a different master and set
of process freelists. By increasing the freelist groups from 1 it allows us to reduce the
contention on the single master freelist, as now processes will search the master freelist in
their assigned freelist group block. A process will only search the MFL in the segment
header if no space can be found in their freelist group block.
I decided to test using freelist groups on my test environment and the results are listed
below:
FREELIST GROUPS = 1
Avg time for each session to complete inserts: 2:57mins
Top 5 Wait Events

~~~~~~~~~~~~~~~~~ Wait % Total
-------------------------------------------- ------------ ------------ -------
buffer busy waits 113,465 60,129 52.25
free buffer waits 599 40,244 34.97
db file sequential read 21,781 4,455 3.87
rdbms ipc reply 106 3,543 3.08
latch free 1,659 3,038 2.64
FREELIST GROUPS = 7
Wait % Total
-------------------------------------------- ------------ ------------ -------
Page 14
free buffer waits 1,988 39,364 45.20
buffer busy waits 5,491 9,534 10.95
latch free 793 5,793 6.65
FREELIST GROUPS = 17
Wait % Total
-------------------------------------------- ------------ ------------ -------
free buffer waits 462 41,452 46.90
latch free 2,045 9,825 11.12
log file parallel write 341 2,937 3.32
The test involved 12 concurrent session inserting 10,000 rows into a table of the same
structure provided at the beginning of this case study.
It is clear from my testing results that using multiple freelist groups can increase
performance significantly by decreasing the number of waits on buffer busy waits. When
the buffer busy waits decreased and freelist contention no longer became an issue,
especially when using 17 freelist groups, it is apparent that my test system has an I/O
issue due to the increase in free buffer waits and db file sequential read waits. This could
also be due to an incorrectly sized buffer cache so further investigation would be required
to remove this new area of contention.
NOTE: Freelist groups cannot be dynamically added to objects. The object must be
recreated with a new FREELIST GROUPS setting and then repopulated with data.
In conclusion, when seeing a high number of sessions waiting on buffer busy waits for the
same datablock, that are continually changing, on a table that has a number of process
freelists defined, it is possible you may be running into the serialization problem with
searching and moving blocks from the master freelist to the assigned process freelist. The
diagnostic steps outlined in this case study have described an approach to determining the
cause to such waits. A workaround is also provided of rebuilding the table with multiple
freelist groups, which was demonstrated to relieve the buffer busy waits. It is important to
note that removing one area of contention often highlights a different area that needs
further optimization.
References
Note 157250.1 Freelist Management with Oracle 8i, Stephan Haisley, Center of Expertise
Page 15

Oracle Press Case Study - Diagnosing Another Buffer Busy Waits Issue

Încărcat de

Informații document

Descriere originală:

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

Oracle Press Case Study - Diagnosing Another Buffer Busy Waits Issue

Încărcat de

Drepturi de autor:

Formate disponibile

Case Study: Diagnosing Another Buffer Busy Waits Issue

Authors: Stephan Haisley, Consulting Technical Advisor, Oracle Corporation

Skill Level Rating for this Case Study: Intermediate

About Oracle Case Studies

• Expert: significant experience with the subject matter

Case Study Abstract

Name Null? Type

Three indexes existed on the table:

1. Unique index on ID column

Process free list entry = (P % NFL) + 1

Top 5 Wait Events

Tablespace IO Summary for DB:

Avg Read Total Avg Wait

OPERATIONS BLOCKS AVG TIME(ms)

Tue Jun 07 11:01:00 2005

Buffer wait Statistics

SID USERNAME EVENT WAIT_TIME P1 P2 P3

SELECT owner, segment_name, segment_type

OWNER SEGMENT_NAME SEGMENT_TYPE

SQL> SELECT header_file, header_block

SQL> ALTER SYSTEM DUMP DATAFILE 14 BLOCK 2;

-- WAIT a few second

SQL> ALTER SYSTEM DUMP DATAFILE 14 BLOCK 2;

The dump shows (only selected information shown):

nfl = 23, nfb = 1 typ = 1 nxf = 177 ccnt = 235781012

The second dump showed:

nfl = 23, nfb = 1 typ = 1 nxf = 177 ccnt = 235781120

Event Level Reason

create table tracing(sid number, serial# number, event number) tablespace

create or replace procedure trace_freelists(what IN NUMBER, onoff IN number)

To enable the events:

We attempt to move a section of the MFL in one go - in this case 5 blocks

Conclusion and Learnings

In the bug, development had first suggested increasing the

I tested an increase of the _bump_highwater_mark_count parameter in a test system

Development had made some suggestions for solutions to this issue:

The three possible resolutions we had to this problem were:

Free list group is: (Process Id % Number of Free group ) + 1

Top 5 Wait Events

S-ar putea să vă placă și