Siebel Oracle Database Monitoring

Siebel Oracle Database Monitoring and
Tuning – Detailed Techniques

by
Eyal Markovich (eyal@totalapm.com)

Total APM Corporation
Abstract
This document discusses performance monitoring and optimization of the Oracle database
tier in Siebel eBusiness application. The document provides detailed information on key
items that can influence the performance of a Siebel application database and solutions to
some common problems.
November 2008
Siebel Oracle Database Monitoring and Tuning
- Detailed Techniques
Acknowledgements
I would like to acknowledge the contribution of Girma Watumo a Senior Principal
Consultant at Oracle Corporation. Girma and I worked on several projects and I
truly value his friendship. Besides being a friend, Girma is a highly skilled Siebel
consultant with in-depth technical knowledge of the Siebel Application. He
provided me with important insights on some key Siebel Applications items
related to performance.
About Total APM Corporation

Total APM is a leader in application performance management, developing
customized methodologies and processes for implementing practical
performance management solutions for our customers. Our expertise is in tuning
enterprise applications, and, in particular, Siebel applications. Utilizing Precise i³
to implement, monitor and tune applications, we provide mentoring and training
services that help you improve and optimize performance management. Total
APM, a partner of Precise Software Solutions, is a reseller of the popular Precise
i³ suite. We consider ourselves to be a genuine VAR (Value Added Reseller), as
we provide our customers with a total package: licensing, services and the
support required to make your i³ investment a successful one.
Total APM also manages and operates the leading Precise i³ user group at
www.AskEyal.com
About the Author

Eyal Markovich is an industry leader in performance tuning of large enterprise
systems, and the Siebel Application, in particular. With more than 15 years
experience working for Precise/Veritas/Symantec, Eyal is one of the industry's
foremost experts in Precise i³. His past experience in developing Precise i³ and
working with Precise/Symantec customers gives him a unique expertise not only
in i³, but also in application technology (such as, Siebel, Oracle, MSSQL, .Net,
and Java) as well as in customers' applications and requirements.
Copyright © 2008 Total APM Corporation. 1580 Sherman #801 Evanston, IL 60201 U.S.A.
All rights reserved.
DOCUMENTATION IS PROVIDED "AS IS" AND ALL EXPRESS OR IMPLIED CONDITIONS,

REPRESENTATIONS AND WARRANTIES, INCLUDING ANY IMPLIED WARRANTY OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NON-INFRINGEMENT, ARE
DISCLAIMED, EXCEPT TO THE EXTENT THAT SUCH DISCLAIMERS ARE HELD TO BE LEGALLY
INVALID.
2
2
Contents
Introduction .................................................................................................. 4
Missing indexes ............................................................................................. 5
Siebel session hints .......................................................................................11
Siebel CBO hint-related problems ....................................................................11
Bind variable vs. literals—a large result set vs. a small result set .........................17
Predefined Queries (PDQ)—default, and public vs. private PDQs ..........................21
Siebel Out-of–the-Box (OOTB) Business Components (BC) Level Performance

Optimization.................................................................................................22
Slow code and Interface .................................................................................27
Connection pooling ........................................................................................28
Workflow scheduling ......................................................................................29
Case-sensitive Config problems .......................................................................30
Siebel timeout ..............................................................................................31
Export + count(*) .........................................................................................33
Stats ...........................................................................................................33
Stored Outline ..............................................................................................34
Appendix B—Templates for SQLPlus and Stored outlines .....................................35

SQLPlus Template ......................................................................................35
Stored Outline Template .............................................................................36
3
3
Introduction
Oracle's Siebel Customer Relationship Management (CRM) ™ application offers
market-leading CRM solutions based on the latest Web technologies. It is a
complex enterprise application consisting of multiple architectural tiers (N-tiers).
Siebel performance monitoring and tuning have become a critical aspect of any
Siebel deployment with the need to satisfy the demands of partners, suppliers,
and customers.
Siebel application performance is defined as the Siebel application's ability to
function, and generally measured in response time or throughput. Some of the
factors affecting a Siebel application’s performance include inadequate hardware,
poor network infrastructure, heavy customization, excessive network round
trips, extensive data volume, poor tuning of database objects, and improper
database configuration. The best way to tackle Siebel application’s performance
issues is to anticipate, plan, and consequently avoid the problems. However, this
is not always possible. For example, it is often difficult to anticipate and account
for all the performance issues that can arise during a major Siebel version
upgrade, with an accompanying database platform upgrade. Furthermore, while
pre-deployment testing is the most cost-effective and risk-averse approach
toward ensuring that Siebel eBusiness enterprise applications operate at
acceptable levels of performance, many performance problems arise only in
production. Consequently, regardless of the time you spend testing, you will still
need to have a good solution for monitoring and tuning Siebel production.
Tuning Siebel appears complicated, as Siebel is a multi-tier environment.
However, and perhaps not surprisingly, we find that 90% of all slow response
events can be traced back to slow database queries.
This document focuses on the performance of the database tier in the Siebel
application. It examines the performance issues encountered, and the remedies
used during several Siebel upgrades, from Siebel 6 to 7, in several Siebel
implementations. In some cases, a Siebel upgrade was accompanied by an
Oracle version upgrade, for example, from 9.x to 10g, which added additional
complications and compounded performance issues.
The most common problem Siebel customers face when trying to tune Siebel
queries is that the SQL that is generated through Siebel cannot be easily tuned
and fixed with database-provided hints. Further complicating the tuning effort is
the fact that while the DBA may discover slow, top-heavy SQL queries in the
database, they will have a hard time relating these SQL queries to a user-
specific navigation event in the browser interface. This type of analysis is time-
consuming and requires an advanced level of expertise.
4
4
Precise i³ (also known as Symantec/i³) is an application performance monitoring

and management tool that has been utilized by Total APM as an integral part in
the process of monitoring, troubleshooting, and solving Siebel performance
problems.
There are several good Oracle Siebel product documents (especially Siebel Book
Shelf 7.8/8.x), Tech Notes, Alerts, Support Articles, and FAQs that address
Siebel Performance and testing. This document does not repeat what is already
discussed in these documents. Also, please refer to other papers for a discussion
of additional topics, such as, hardware selection, configuration, and testing.
Examples of specific Oracle Siebel documentation that can be used include the
Book Shelf > Performance Tuning Guide (7.8 or 8.x), and Technical Note 582:
Oracle CBO and Siebel Business Applications.
This document discusses some key problems and issues that contribute to the
poor performance experienced by many Siebel implementations. You can use
these guidelines section to gain a better understanding of what can cause slow
performance in your implementation. We discuss the symptoms (for example,
timeouts), and cause (such as missing indexes) of the problems, as well as some
possible solutions.
This whitepaper is technical and focuses only on monitoring and tuning the
Siebel database. It is recommended that the reader be familiar with Total APM’s
best practice recommendations for managing performance in the Siebel
application, which can be found in the whitepaper “Siebel Tuning and Monitoring
- an Organizational Approach”. This paper discusses the methodology and
processes required to address performance problems in Siebel. It also describes
the Precise i³ tool-set and lists the key features of i³ that are important for
Siebel monitoring. The paper can be downloaded at
www.totalapm.com/whitepapers.html.
Missing indexes
In our experience, a typical customized Siebel implementation requires an
extensive amount of additional indexes. The two primary reasons for missing
indexes are the existence of Siebel customizations and because Siebel uses the
new Oracle CBO in many implementation of Siebel 7.7 and later. Navigation to a
view or execution of a specific ad hoc query or PDQ in a certain view can be very
slow and time-consuming. Some views may simply not respond, resulting in a
Siebel timeout. One of the first issues that need to be addressed when
troubleshooting a problem is that of 'Missing Indexes', since this solution is the
easiest solution to implement.
In heavily customized Siebel implementations, there is an extensive use of
extension columns on Siebel base tables. Performance problems surface when
these extension columns are not indexed appropriately. In addition, performance
deteriorates when these extension columns are used to store foreign keys and
are not indexed.
5
5
When upgrading, one typically assumes that if the Siebel 6 implementation was
performing well (probably after spending time tuning the system by adding
many customized indexes), the post-upgrade will perform equally well. However,
several internal changes within Siebel (for example, the addition of Intersection
tables), and database changes (for example, changes to the CBO) may cause
the previously-customized indexes to not be used, and create a need for new
indexes.
Index manipulation results in best performance improvements when carried out
correctly. Consider the following guidelines when analyzing slow running queries:
Create new indexes - mainly to support existing custom configurations.
Modify existing indexes to add/remove columns, or change the order of
columns.
Remove unused custom indexes that were created in previous versions of
Siebel.
Note that even some out-of-the-box indexes may need to be modified or
removed.
6
6
The following is an example of the results received after tuning a Workflow query
by creating a new index. The autotrace output comparing the results before and
after indexing, exemplifies the magnitude of the change:
Statistics before the index Statistics after the index

1 recursive calls 0 recursive calls
0 db block gets 0 db block gets
89346 consistent gets 955 consistent gets
86402 physical reads 0 physical reads
0 redo size 0 redo size
9117 bytes sent via SQL*Net to client 9117 bytes sent via SQL*Net to client
734 bytes received via SQL*Net from cli 734 bytes received via SQL*Net from cli
38 SQL*Net roundtrips to/from client 38 SQL*Net roundtrips to/from client
0 sorts (memory) 0 sorts (memory)
0 sorts (disk) 0 sorts (disk)
554 rows processed 554 rows processed
Q: Should I index each customized column?

A: No. In most implementations, a high number of indexes already exist on
some of the tables (such as, S_OPTY) and you must be careful not to add too
many indexes. A new index can fix one query but may slow down another.
Moreover, each new index adds some overhead to the system and slows down
DML statements (such as, Insert, Delete, and Update). It is critical to pick the
correct set of columns (both the columns and their order are important) so that
the CBO can properly use the index, and the index can subsequently improve
performance.
A recommended solution is to index the columns that improve the queries
reported by i³ as top queries. For example, if a user is complaining about a slow
response time in production, or if a LoadRunner transaction showed a higher
average than the SLA, you can use i³ to identify the activity for that user (or
transaction) and identify the top SQL contributing to response time. There is no
point in indexing a column that speeds up a query, even if the improvement is
significant, if the query is executed only once a week. You should consider
adding new indexes that improve queries that consume high database resources,
as well as queries that cause end-users to wait a lengthy amount of time for a
response. With i³, identifying the SQLs that need to be tuned by indexing is
almost effortless.
7
7
Q: How can I determine if an index is missing for a particular

statement?
A: i³ provides you with all the information you need to determine if an index is
missing. The following is a brief description of Total APM's best practice for
analyzing a statement's performance. For additional information, please refer to
Total APM’s whitepaper, "Methodology for Analyzing a SQL Statement using
Precise i³", at www.totalapm.com/whitepapers.html. Obviously, the analysis of
each statement is unique and requires special skills. Nevertheless, the steps
below can serve as a basis for understanding statement performance:
First use i³ to identify the statement that needs to be investigated (typically, top
queries for a user or view), then examine the resource breakdown of the query
in the database. It is also beneficial to take a look at the total number of
executions and the average response time for the query.
Figure 1: A statement with high I/O Wait, high Memory wait, and high CPU consumption
As a rule of thumb, a statement that has high I/O Wait or a high Using CPU
value is a candidate for a missing index. An I/O wait can be reduced with the
introduction of a new index, if the index filters out many rows. Take, for
example, the following query:
SELECT * FROM T1.S_OPTY WHERE T1.X_ACCT_NUM = :1
8
8
Let’s assume that S_OPTY has 10 millions rows and X_ACCT_NUM has 1 million
distinct values. On an average, 10 rows would be returned from the above
query. If an index on X_ACCT_NUM is missing, Oracle CBO would need to read
the entire S_OPTY table in order to find the 10 rows. This would result in
thousands of I/O requests. If an index exists, the optimizer would be able to
return the rows in 10-15 I/O requests.
SELECT >1000 I/Os SELECT
10 rows 10 rows
S_OPTY >1000 I/Os S_OPTY 10-15 I/Os

FULL Table Scan
10 rows
Index
4-5 I/Os
X_ACCT_NUM
For the same reason, CPU Usage can be reduced by introducing a new index. If
the table in question is relatively small then the table data would be stored in
the Oracle buffer. In this case, the unnecessary thousands of I/O requests will
be Logical I/Os, reported by i³ as CPU utilization.
If you determine that the main resource consumption in the database is I/O
wait, your next step would be to identify the list of objects that I/O performs on.
This information would prove to be invaluable when investigating and analyzing
the Explain plan of the query.
9
9
Figure 2: In this example the statement is mainly waiting for S_ACT_EMP
With this information, we can start looking at the Execution plan. Most Siebel
queries are complicated and involve many tables and joins. Precise i³ visually
displays the plan and provides you with dictionary information on the objects
involved in the statements—mainly tables, indexes and columns. Together with
the information obtained before, it is possible to focus on the step in the plan
that refers to the hottest object in the query.
Figure 3: Focus on S_ACT_EMP steps in the plan and verify that the operator is optimal
Now that we have identified the problematic object and its access path, we can
investigate the columns that appear in the Where or Join clause and verify that
they are indexed correctly.
10
10
Siebel session hints

When creating the database session, the Siebel database connector makes the
following session level changes—it sets the optimization mode to
First_ROWS_10, disables hash join, disables sort-merge, and, using two
columns, enables a sanity check for the join operation.
It is important to always remember these session hints when analyzing or tuning
the performance of Siebel queries. For example, if you are executing a
problematic SQL from SQLPlus, or reviewing an execution plan, and you did not
set the above session hints, the plan that Oracle will use will most likely differ
from the plan the optimizer uses for Siebel.
When analyzing Siebel-generated SQL outside of Siebel, it is therefore
recommended to set the following session identifiers when opening the
connection:
alter session set optimizer_mode = first_rows_10

/
alter session set "_HASH_JOIN_ENABLED" = false
/
alter session set "_OPTIMIZER_SORTMERGE_JOIN_ENABLED" = FALSE
/
alter session set "_OPTIMIZER_JOIN_SEL_SANITY_CHECK" = TRUE
See Appendix B for a general template for executing SQL from SQLPLUS.
Siebel CBO hint-related problems

Siebel sets a session level hint so that the optimization level is set to first rows.
The logic is simple: Typically, client queries fetch a screen full of rows. The
cursor is then kept open to fetch more rows if the user presses the next record
set, or next record on the last row. This makes it important to optimize the
query plan to return the first few rows quickly. In other words, Siebel is saying:
“let’s retrieve the first few rows as soon as possible, and display the Client Web
page. Once the user scrolls to the next record set, fetch more rows.” In most
cases, this strategy works well, but we have come across cases where this
process brought Siebel to a halt.
The problems usually start when users need to sort information. In our
experience, the user community’s need for ad hoc and PDQ-based sorts is high.
We have experienced many cases, post upgrade to Siebel 7.8 and Oracle 10g,
where serious performance issues that weighed heavily on DB system resources
were due to queries that included sorting.
11
11
Let's take a look at the performance problem behind the usage of the
FIRST_ROWS optimization mode. Consider the following simple query (this can
be part of a complex Siebel query):
SELECT
T2.EMP_ID,
…..
FROM SIEBEL.S_EVT_ACT T1,
SIEBEL_S_ACT_EMP T2
…..
WHERE T2.EMP_ID=:1 AND
T1.ROW_ID = T2.ACTIVITY_ID
….
ORDER BY T1.OWNER_LOGIN
This is a simple query that returns activities for a single employee and sorts the
result by the OWNER_LOGIN column. The main idea is that the predicate for
filtering is done on S_ACT_EMP while the sort is requested on a column of
S_EVT_ACT. The optimizer needs to select the optimal plan for this query. There
are two main options to pick from:
(1) start with an index for the predicate EMP_ID=:1 or,
(2) start with the index on OWNER_LOGIN that will avoid the sort.
In complex implementations, we have seen cases where the optimizer preferred
avoiding the sort order at all cause.
Let’s try and understand the cost associated with each option. The calculation
here is simplified but the basic idea is the same. We will assume there are 10
Million rows in each table (S_ACT_EMP and S_EVT_ACT) and an average of 100
activities for each employee.
Option 1: The optimizer starts with an Index on S_ACT_EMP (EMP_ID) to check
the EMP_ID=:1 predicate and eventually sorts the result set.
Using i³ we can view the full execution plan. Below is a screenshot from i³
showing a subset of the execution plan. In this case the plan was simplified to
display only these two tables.
Figure 4: Sort is performed and the driving table is S_ACT_EMP
12
12
The plan consists of a Nested loop join and a sort. The optimizer chooses
S_ACT_EMP as the outer table in the join and S_EVT_ACT as the inner table.
S_ACT_EMP is the driving table because the EMP_ID=:1 predicate has a good
level of selectivity. The plan starts with the Index S_ACT_EMP_M6 which
includes the EMP_ID column. This index finds the 100 rows that have
EMP_ID=:1. The cost associated with this step is around 4–5 I/O requests
required to traverse the index. For each row found, a table ROWID lookup (1
I/O) is performed on S_ACT_EMP and the ACTIVITY_ID column is obtained.
Since this table lookup is performed on each of the 100 rows, the cost associated
with the outer table of the nested loop is 100–104 I/O requests.
For each row in the outer table, Oracle needs to find all rows in the inner table
that satisfy the join condition. Since there are no additional local predicates on
S_ACT_EMP, the join will be performed 100 times—once for each row.
For each inner table execution, an index lookup is performed using the
S_EVT_ACT_P1 (ROWID) and then the S_EVT_ACT table is accessed. The cost
associated with the inner table part of the join is 3–4 I/O requests per execution.
Since there are no additional local predicates on S_EVT_ACT, the 100 rows that
the outer table identified constitute the complete result set.
The last step in the plan is to sort these 100 rows and return them to the client.
Sort 100 rows

SORT
400-500 I/Os JOIN

Performed 100 times
once for each row in
S_ACT_EMP
100 I/Os S_ACT_EMP S_EVT_ACT 1 I/O
100 rows 1 row
Index Index
3-4 I/Os EMP_ID ROW_ID 4-5 I/Os
13
13
The total cost associated with this plan is around 400–500 logical I/Os and a sort
of 100 rows.
Now let's take a look at the second option? In this case, the sort is avoided. The
optimizer starts with S_EVT_ACT as the leading table and for each row it checks
the EMP_ID predicate in S_ACT_EMP. Once the requested number of rows is
found, the result can be sent to the client and there is no need to continue
reading the entire result set. The CBO may prefer this option in a heavily
customized implementation.
The following figure, showing how this scenario is rendered in i³, displays a
subset of the execution plan when a sort is avoided:
Figure 5: No sort is performed and the driving table is S_EVT_ACT

Observe that no sort has been carried out and that the driving table is
S_EVT_ACT rather than S_ACT_EMP. The optimizer is saying “I would rather do
anything than sort a big result table”. Why?
Recall that the optimizer is trying to retrieve the first few rows as quickly as
possible. If a sort needs to be done, the optimizer needs to read the entire result
set and sort it based on the sort criteria (OWNERE_LOGIN in this case). That is,
although the SQL specified the FIRST_ROWS hint to be optimal for the first few
rows—the entire result set MUST be processed and sorted based on the sort
criterion.
The plan consists of a Nested loop join but without a sort. The optimizer chooses
S_EVT_ACT as the outer table in the join and S_ACT_EMP as the inner table.
S_EVT_ACT is the driving table because the optimizer can use the index on
OWNER_LOGIN (S_EVT_ACT_M17) and process the rows in S_EVT_ACT, which
are already sorted. For each row in the index, it will perform a table ROWID
lookup to obtain the ROW_ID (1 I/O). This is the outer table part of the nested
loop. The inner table part of the join consists of using index S_ACT_EMP_U1 to
locate the ACTIVITY_ID (2–3 I/Os), and a table ROWID lookup (1 I/O) to check
the EMP_ID=:1 predicate.
This process continues until Oracle manages to build the result set (usually 10
rows). At this stage, the process is completed and the result can be sent to the
client. There is no need to sort the result set since the rows have already been
sorted. As expected, since there are only 100 rows in S_ACT_EMP having
EMP_ID=:1, most of the time the inner table step fails and another row in
S_EVT_ACT needs to be processed.
14
14
The total cost associated with this plan depends on how quick we can find the
first rows having EMP_ID=:1. For this calculation, we assume that these 10 rows
were found after processing half of the rows in S_EVT_ACT.
25 M I/Os JOIN
Performed 5 M times
once for each row in
5M rows S_EVT_ACT
5M I/Os S_EVT_ACT S_ACT_EMP 1 I/O
5M rows 1 row
500 I/Os Index Index

2–3
FULL INDEX SCAN OWNER_LOGIN ACTIVITY_ID
I/Os
The cost of a Full Index scan on S_EVT_ACT_M17 is high. The real problem is
that the S_EVT_ACT ROWID lookup and the inner join are performed 5 million
times (half of the total number of rows), but the total cost of this plan is around
25 million logical I/Os—just to avoid carrying out a sort.
One may ask why this problem arose in the first place. Why didn't the CBO pick
the correct plan? The answer can be found in the customization columns in the
tables, the skew of data between employees, and the usage of bind variables.
This problem can be fixed by changing the Business Component (BC) default
optimization level hint. The question you should ask yourself is how would you
know on which BC the optimization hint should be changed?
15
15
Q: How will you know which BC to change to ALL_ROWS?

A: Use i³ to gain visibility into the profile of your application. The best practice to
finding the answer to this question is to:
Focus on the top queries for each BC or View.
Check the real execution plan and verify that the optimizer mode is First
Rows.
Check the real execution plan and verify that sort is avoided.
Verify that the index that should have been used exists.
Execute the query (see Appendix B for instructions on how to execute a
Siebel query from SQL Plus) and verify that the result set is not too large.
Execute the same query by adding a /*+ ALL_ROWS */ hint and compare
the two executions (using SQLPlus autotrace).
Note that even if you found a BC based on the above, that suits ALL_ROWS, you
still need to use caution before deciding to change the session hint. Recall that
changing the session hint will change it for all queries from this BC. It is also
good practice to use i³ to:
Verify that the queries you analyzed above are being used by many users
and that this is a real system performance problem.
Check other statements that run from the same BC (queries that run
quickly now) and verify that they will continue to run quickly after the
change.
The following procedure describes the changes to make to change the BC
ALL_ROWS session hint:
Figure 6: Changing the BC OracleCBOHint parameter
16
16
Identify the BC and check out the object or Project under which the BC is
contained.
In Object Explorer navigate to BC then to the BC User Property child object.
Create a new record with the following attributes:
Property: Name; Value: OracleCBOHint
Property: Value; Value: ALL_ROWS
Property: Comment; Value: Your comment
Compile SRFs and deploy the changes in PTE.
Test for performance improvement in the targeted queries.
Deploy the changes in production.
After performing this change, perform regression tests on other areas where this
BC is in use, and make sure that they have not been negatively impacted.
An alternative solution for changing the optimizer hint to ALL_ROWS can be
tested by setting the init parameter _SORT_ELIMINATION_COST_RATIO.
Theoretically, setting this parameter—which influences the cost ratio for sort
elimination under first_rows mode—allows you to force the optimizer to perform
the sort and use the correct index. However, you are advised to thoroughly test
this setting throughout the entire application, prior to implementing it in a
Production environment as it affects the entire Siebel Application and not just
the problematic BC.
Bind variable vs. literals—a large result set vs. a small

result set
Siebel uses statements with bind variables throughout the application. Bind
variables are good for performance, as Oracle does not need to hard-parse the
query every time it is executed. However, in some cases, bind variables lead to
major performance problems. One of the more significant problems we faced
was the issue of bind variables in SQL LIKE predicates. By default Siebel uses
bind variables in passing query criteria values entered by users. In many cases,
where users used trailing wildcards (asterisk), Oracle 10g Optimizer’s calculated
query paths that were inefficient, resulting in queries that took from several
minutes to an hour. There are four query circumstances in which this happens:
(1) A user uses a partial query string and a trailing wildcard. Example: “ABC*”.
(2) A user opens a pick applet and queries by entering any value in the Starting
With box of the pick list, unless the query string is preceded by = sign.
Example: “ABC”. (=ABC will prevent wildcard).
(3) The implementation is using “AutomaticTrailingWildcards = TRUE” in the SWE
section of the application's CFG file (such as uagent.cfg) or is not present. (It
is recommended that implementations have the AutomaticTrailingWildcards
parameter set to FALSE in order to prevent unnecessary trailing wildcards
that impact performance).
17
17
(4) Automatic Trailing Wildcard is not disabled in the Siebel Search Center. This
parameter is also typically disabled. This is done through a configuration
change in OOTB business service Search Execution Service. A new Business
Service User Prop would be created with the following parameters: Name =
AppendWildcard; Value = FALSE to prevent unnecessary trailing wildcards
that impact performance. It is assumed that this is done.
In most cases, (1) and (2) above are unavoidable, based on how users use
Siebel. In both cases the solution was to use a little known Field User Property
configuration solution that instructs Siebel to pass the literal rather than the bind
variable. With this change, a query predicate T23.NAME LIKE ‘ABC%’ is sent to
the database instead of T23.NAME LIKE :5. Performance, in these cases,
changes from minutes to sub-seconds, thereby avoiding Siebel timeout, in some
cases.
Let's take a look at why performance has improved. Consider the following
statement:
SELECT ...
FROM ...
SIEBEL.S_PARTY T9,
SIEBEL.S_CONTACT T25
WHERE
T25.CON_CD != .... AND -- some criteria on S_CONTACT
(T9.ROW_ID LIKE :2)
i³ identifies that the hottest object in this query is S_CONTACT with 99% of the
waits.
Figure 7: S_CONTACT with 99.6% of the waits
18
18
If you take a look at the plan that i³ captured, you can see that the criteria on
S_PARTY.ROW_ID LIKE is not the first predicate that the optimizer checks. For
some reason the optimizer decided to ignore the search criteria and focus on the
S_CONTACT criteria. The question is, why is that?
Figure 8: S_CONTACT is the first table in the plan
The optimizer does not know the value of the :2 parameter used in the Like
predicate. Obviously, bind picking does not work here. If the value of :2 is
somewhere around ‘1-2345%’, then only a small number of rows will return
from this predicate and the plan should first check on this criteria. However, if
the parameter contains something around ‘1%’ or even ‘%1%’, then millions of
rows will be returned and this criteria should not be used as a leading predicate.
We found that the optimizer makes the wrong decision, in this case, and chooses
to ignore the predicate in cases where it should have used it instead. In these
cases, switching to literals would provide the optimizer with the missing
information it needs to pick the correct plan:
If the statement is (T9.ROW_ID LIKE ‘1-2345%’) then the index on
ROW_ID will be used.
If the statement is (T9.ROW_ID LIKE ‘%1%’) then the same plan that
was used before, will be used. It will still be slow and we will need to train
the user how to properly perform a query.
Here is the new plan when a literal is used:
Figure 9: The like predicate is the first to be evaluated
19
19
Q: Should we change the Field User Property configuration parameter in

all fields?
A: Of course not. The best practice is to use i³ to determine which fields will
stand to gain from this change. One should focus on long-running queries in i³
that have Like criteria.
To qualify as a candidate for the change, the field has to have the following:
Many distinct values in the database. Query the appropriate table or use
i³ to report the number of distinct values.
The parameters that are sent to the database filter 1-2 rows, most of the
time. Use i³ to obtain an example set of bind variables and check how
many rows were returned, using SQLPlus.
The query is executed from SQLPlus (using the suggested template for
statement execution) after replacing the Like :2 with the actual literal
value (rather than host variable) and verify that the plan changed and
the execution time resulted in major improvements.
The change is carried out as follows (refer to the figure below):
(1) Identify the BC and field in question.
(2) Check out the object or project under which the BC is contained.
(3) In Object Explorer, navigate to BC and then navigate to the field and Field
User Property child object.
(4) Create a new record with the following attributes:
a. Property: Name; Value: Use Literals For Like
b. Property: Value; Value: TRUE
c. Property: Comment; Value: Your comment
(5) Compile SRFs and deploy the changes in PTE.
(6) Test for performance improvement in the targeted query.
(7) Deploy the changes in production.
20
20
Figure 10: Changing Field Use Literal for Like property

Caveats: Searches of large tables on criteria with wildcards in leading positions,
such as "*ABC", typically impact performance. However, the use of Literals for
Like does not improve performance for such searches because it does not
significantly reduce the number of rows scanned. A widespread use of this user
property can drive up CPU use because more SQL parsing and prepares are
required, and can result in package cache overflows. Use this user property
sparingly, and only on fields for which you have confirmed the benefit by testing.
Predefined Queries (PDQ)—default, and public vs.

private PDQs
Untuned PDQs result in slow navigation to the driving list view of a screen.
Furthermore, untuned default PDQs result in performance problems that affect
every user that navigated to the view. Therefore, special care is required for
tuning default PDQs. In addition, in some cases, the navigation executed has an
open query PDQ that adds nothing to the Where clause. The number of records
returned would be large depending on the visibility type of the view (My, All,
Organization, My Team’s) and the number of records in the base table of the
primary BC.
i³ allows you to identify the cases where the default PDQ poses a performance
problem that affects many users.
There are implementations where the users navigating to the view almost always
need to run a different PDQs (or ad-hoc query) than the default PDQ. In this
case, the default PDQ does not serve any useful purpose and it is a waste of
resources to execute the default PDQ.
21
21
Therefore, if acceptable to the business application, consider one of the

following:
(1) Create a default PDQ that returns no records for list views. For example,
[Id] = NULL will always return no records. When users navigate to the view
the result will be an empty screen displayed very fast. The user can then
select the preferred PDQ or perform a targeted ad hoc query.
(2) If recommendation (1) is not acceptable to business, special attention should
be given to simplify and tune the default PDQ.
(3) Avoid default open queries. Use some default PDQs with data filtering
criteria.
Another factor that affects performance is the existence of private PDQs. A
private PDQ that is not tuned will not only affect the user executing it, but can
affect the entire application, since database resources are used up when
executing a bad query. We use i³ to monitor and tune private PDQs in the same
way we monitor public PDQs and ad hoc queries. However, the existence of
private PDQs should be taken into consideration during a Siebel upgrade.
The main lesson learned is as follows: During a Siebel upgrade, migration of
public queries to upgraded production may be justified, but the migration of
private PDQs is never justified. In order to avoid the many potential problems—
including performance problems—it is best to train and ask users to recreate
their PDQs after upgrading to the new version.
The following is an example of potential problems that are not related to
performance: If the SQL generated by the PDQ is incorrect in the new version, it
may return an incorrect set of records and the user may be unaware of the
discrepancy. This is true for public PDQs as well. Alternatively, if the user
community insists on keeping their private queries (possibly hundreds, even
thousands of them), you need to allocate the time and effort, which may require
several weeks of effort in PTE, to tune and test these private PDQs, before
migrating them to production. It is important that the team avoids migrating
untested or untuned private or public PDQs, since they may find themselves in
trouble on the first day of going live. Our recommendation is to only tune and
migrate public PDQs.
Siebel Out-of–the-Box (OOTB) Business Components

(BC) Level Performance Optimization
Up until now, we have discussed common problems in Siebel applications. The
following section discusses an unusual problem that can result in severe
performance issues. The most serious and challenging performance issues that
we faced in an Action Business Object (BO) were primarily in views contained
within the Activities Screen. While part of the problem was due to high data
volume and customization, the primary cause was due to a conflict between out-
of-the-box (OOTB) performance optimization on the primary Business
Components (BC) and Oracle 10g’s CBO. The primary BCs involved were Action
and Action See Other. In both cases our challenge was as follows: Many of the
frequently used fields in the ad hoc or PDQ-based queries were custom fields
22
22
that are not denormalized into the intersection table S_ACT_EMP used by the
Siebel BC class as query-driving tables for these views. While S_EVT_ACT is the
base table for these BCs, many of the OOTB's frequently used columns in
S_EVT_ACT are denormalized into S_ACT_EMP. For example,
S_EVT_ACT.EVT_STAT_CD, which holds the activity status, has a corresponding
denormalized column S_ACT_EMP.ACT_EVT_STAT_CD. During any data
manipulation operation, the data in these tables is kept in sync by the BC class.
Figure 11: An out-of-the-box denormalized field
When a user queries on the BC field that uses S_EVT_ACT.EVT_STAT_CD, the

BC class converts the query to use S_ACT_EMP.ACT_EVT_STAT_CD.
Siebel/Oracle designed all major BCs, such as Account, Contact, and
Opportunity, to work this way because in each case record visibility is checked
using intersection tables. In Activities, S_ACT_EMP stores the relationships
between activities and employees. Any query executed in the My Activities, My
Team’s Activities, and Other’s Activities views has to include S_ACT_EMP in the
query to check record visibility for the user before further filtering out other
criteria. Therefore, to optimize performance, Siebel/Oracle has denormalized
important columns into S_ACT_EMP. Our challenge was to decide what to do
about some frequently used custom columns and some OOTB columns that are
not denormalized, but nevertheless frequently used in this specific
implementation. It's important to understand this problem from the database
point of view.
23
23
Consider the following template of a query for the My Activities view:
SELECT
T1.EMP_ID,
…..
FROM SIEBEL.S_EVT_ACT T1,
SIEBEL_S_ACT_EMP T2
…..
WHERE T2.EMP_ID=:1 AND
T1.TODO_CD=:2
T1.ROW_ID = T2.ACTIVITY_ID
We will assume that this employee has a total of 2000 activities in the tables,
and 5 rows were returned by this query (TODO_CD=:2 filters out 1995 rows).
Below is the Explain plan that i³ captured (simplified to only display the two
tables):
Figure 12: 86% of the waits are done on S_EVT_ACT
In this plan, the optimizer correctly starts with an index on S_ACT_EMP to check
the EMP_ID predicate. For the index, the matching rows in S_ACT_EMP are
located to obtain the ACTIVITY_ID. With each activity ID, the index on
S_EVT_ACT (ROW_ID) is used to locate the corresponding row. Finally, the
check on TODO_CD is performed and the row is returned to the client if the
criterion on TODO_CD holds.
We will now try to understand the cost associated with this plan. The calculations
here are simplified but the basic idea remains.
To identify the 2000 rows in S_ACT_EMP that belong to this employee, a total of
~2000 I/O requests are required, 4–5 I/Os are required to traverse the index,
and an additional one I/O per row is required for the ROWID lookup. For each
row in S_ACT_EMP Oracle needs to identify the matching row in S_EVT_ACT (2–
3 I/Os for the index S_EVT_ACT_P1 and one I/O for the table ROWID lookup).
24
24
Eventually, the check is performed on TODO_CD, and 5 rows are returned in the
result set.
Total cost: approximately 10000 logical I/Os to retrieve 5 rows.
This query will perform considerably slower (can take up to 2 minutes).
Now imagine the same query for the My_Staff view. If the manager has a large
number of employees reporting to him, this query can result in a timeout (1 to 2
minutes per employee).
There are several issues with this plan, all related to the fact that the TODO_CD
predicate that filtered out 5 rows out of 2000 is the last element that has been
checked. As a quick solution, we created an index on S_EVT_ACT
(ROW_ID,TODO_CD). We found that the CBO will not use this index and will
continue to use the S_EVT_ACT_P1 index. We forced the use of this index with a
stored outline; as a result that performance improved dramatically. However, we
had to have a solution that was not based on stored outlines. (The limitations of
stored outlines are discussed below).
We have managed to permanently fix this issue by denormalizing several
columns in S_ACT_EMP. After denormalization the query text changed similar to
the following example:
Text before Denormalization Text after Denormalization

SELECT SELECT
T1.EMP_ID, T1.EMP_ID,
….. …..
FROM SIEBEL.S_EVT_ACT T1, FROM SIEBEL.S_EVT_ACT T1,
SIEBEL_S_ACT_EMP T2 SIEBEL_S_ACT_EMP T2
….. …..
WHERE T2.EMP_ID=:1 AND WHERE T2.EMP_ID=:1 AND
T1.TODO_CD=:2 T2.ACT_TODO_CD=:2
T1.ROW_ID = T2.ACTIVITY_ID T1.ROW_ID = T2.ACTIVITY_ID
Denormalization, in and of itself, improves performance considerably. Now,

Oracle can identify the 5 rows that satisfy the condition in the outer table and
perform the inner table join only 5 times. Cost was reduced to ~2000 I/O
requests. We further improved the situation by creating some customized
indexes on EMP_ID and creating additional columns. For example, for the above
statement, creating an index on EMP_ID,TODO_CD reduced the number of I/O
requests to less than 50.
Denormalization is not customizable; therefore it was not the first solution we
considered. We attempted many other solutions to no avail before we submitted
a Non-Standard Change Request (NSCR) to Oracle’s Siebel Expert Service for
custom denormalization of several columns (mostly extension columns). After
the request was approved, our POC test showed excellent promise.
25
25
Consequently, we implemented the denormalization solution in the PTE

environment for extensive performance functional testing and eventually
deployed it to a production environment.
The general tasks involved are as follows:
Benchmark your application’s performance (problem queries, PDQs, Load
tests).
Perform schema changes to create custom denormalized fields in
S_ACT_EMP table by creating new columns of type ‘Denormalized’. Each
denormalized column has the following unique properties that are not
editable in Tools. SES needs to make these changes.
Property: Type; Value: Denormalized
Property: Denormalization Path; Value: [ACTIVITY_ID].[<column name in
s_evt_act>] Example: [ACTIVITY_ID].[X_ASSIGNED_DT]
Property: LOV Type; Value: <LOV Type associated to the corresponding
column in s_evt_act if any>
Convert existing data by copying data from S_EVT_ACT columns to the
corresponding denormalized columns in the intersection table. Since
running a direct SQL is not recommended in Siebel implementations, an
EIM job is the best technique for this.
Compile SRFs and deploy the changes to PTE.
Test for performance (problem queries, PDQs, Load tests). This test
needs to be performed in a PTE (production-like) environment.
Test for functionalities surrounding activities such as interfaces, manual
and automatic data manipulation, Workflows that affect activities,
Assignment Manager-based and manual assignment of activities. This is
essentially a regression test of functionalities surrounding Business Object
activities to make sure that nothing malfunctions as a result of
denormalization changes.
Deploy the changes in production when improvement reaches a
satisfactory level.
Some implementations use a virtual user or queue concept to assign their
activities and Service Request records. This may help customers manage
assignments manually, when a Siebel Assignment Manager is not used to assign
employees to these records. Resource managers or queue managers often run
into performance problems when too many records are assigned to these virtual
users. For many customers, the Activity views are slow, and the problems
worsen when used to display virtual user activities or queues.
Consider the query explained above. The main problem is the fact that many
rows satisfy the condition on S_ACT_EMP. At least two I/O requests are
performed for each row in order to identify the corresponding row in
S_EVT_ACT. However, when querying on virtual users, such as queues, the
number of rows is much greater; resulting in a considerably higher number of
I/Os.
26
26
Slow code and Interface

It is common for customers to run into performance problems that arise from
not following Siebel's best scripting practices. The use of eScripts or VB scripts
for a certain function or validation, while the declarative alternative is available
out of the box, is a typical unnecessary overhead. Elaborating on this and other
scripting issues would be repeating what is contained in Siebel documentation
and is beyond the scope of this paper. Please refer to the Siebel Performance
Tuning Guide for more detailed information on best scripting practices.
As stated before, typically 90% of the performance problems are in the
database. The remaining 10% may be in the code. Consequently, i³’s ability to
identify that the problem lies in the code and not in the database is very
valuable. You would not want to spend a lot of time on database improvements,
when the eScript code is the bottleneck and most of the time is spent outside of
the database.
It is common to see Siebel Application implementations that contain dozens or
hundreds of interfaces to other systems and services. When a transaction or
single interface performance problem is reported, the challenge always lies in
identifying the bottleneck tier causing the slow response time. It is common to
first analyze the Siebel database. However, you do not want to waste important
time and resources investigating unrelated SQLs when the real problem is in a
.Net method, often outside the boundary of Siebel domain. i³ can help identify
the problematic tier and pinpoint where the bottleneck is, using modules of its
supported technology stack, such as, .Net, J2EE, MQ, and Tuxedo.
Figure 13: i³ identifies the .NET and IIS interfaces as the bottleneck
27
27
Connection pooling
Connection pooling in Siebel is very important for performance. Siebel Bookshelf
contains a section on “Database Connection Pooling Usage Guidelines” that
provides general recommendations with regards to connection pooling. Several
factors should be considered when determining whether to use connection
pooling in an implementation. For example, number of users, type of
connections etc.
An implementation with connection pooling is characterized by the following
behavior:
Opening a connection is relatively fast. The shared connection eliminates
the need to creating a physical database connection, which takes a long
time.
For the same reason, database resource utilization may be reduced.
On the other hand, as users need to share the connection, additional
overhead is added to obtain the connection for every request.
Long-running queries utilize a connection for a long time and may result
in users waiting a long time for a connection.
Therefore, when considering connection pooling it is important to analyze the
behavior of the implementation with regards to database connections. Are the
connections mostly end-users? What is the average think time? What is the
maximum number of connections?
In addition, one should use i³ to examine database resource utilization.
Connection pooling may reduce database resources. However, in order to
understand the implications of this reduction, one first needs to understand the
database behavior.
Consider the following example. Assume a site that made an executive decision
not to use connection pooling. The site has only 2,000 users and therefore
decided to create a dedicated database connection when users log on to the
Siebel Application. The database connection created upon login was kept alive
until the user logged out (or timed-out). With only 2,000 users and enough
database resources, this decision appears to make sense. Only 2,000 physical
database connections will be created and users will have no delays related to
connection management, except when first logging into the application.
However, If this site also uses a lot of EAI Web calls, this may be a mistake. The
nature of these EAI requests is as follows: there are many short requests that
use an anonymous browser user. As a result, each Web service call results in
creating a physical database connection, and then dropping it, which slows
performance. Moreover, on the database side, this behavior can lead to a high
level of overhead by the Oracle listener process. This site should therefore
enable connection pooling only on EAI connections.
It is recommended to use i³ to monitor CPU utilization of the listener process. In
addition, you should use i³ to examine the number of new physical connections
created over time, and identify if too many logons are performed.
The following figure shows how this information is displayed in i³.
28
28
Figure 14: Number of new connections (Logons) over time
Workflow scheduling
Performance issues may stem from setting an inadequate Batch Workflow
Process schedule. While each Batch Workflow Process may be tuned and
efficient, frequent executions may result in high overhead on the entire system.
It is therefore important to examine the schedule of the Workflow Processes and
make sure the scheduling is optimized. i³ can also help spot runaway Batch
Workflow processes that are running hundreds of times per day, when there is
no business need for that level of frequency.
i³ will identify batch processes that consume a high amount of resources. For
each process, you can display the list of top queries executed. You can then
examine the number of executions for the top queries over time and verify that
the number of executions meets expectations.
Figure 15: Number of executions of a statement
29
29
Case-sensitive Config problems

Another common performance problem is caused by setting the BC field level
property, 'Use Default Sensitivity', in Siebel Tools, to FALSE when the
environment is case sensitive. The translation of this configuration change is as
follows: If, at the Business Component (BC) or Application level, case
insensitivity is set to TRUE, then this field and the columns based on this field,
will behave the opposite. Therefore, if case insensitivity is set to FALSE, which is
the recommended setting for performance reasons, this column will be case-
insensitive. Given this setting, the Siebel application generates case-insensitive
queries for that particular column based on that field, thereby hurting
performance. Very often this field property was set accidentally and there is no
valid business requirement for the change.
You can identify this situation by looking at the query text i³ captures. In this
case, queries will have a 'Where' clause built with the following structure:
X_ACCT_NAME like ‘AG%’

Or X_ACCT_NAME like ‘Ag%’
Or X_ACCT_NAME like ‘aG%’
Or X_ACCT_NAME like ‘ag%’
It is not recommended to set Siebel application queries to be case-insensitive, at

the application level. Whenever you see a case-insensitive query at the SQL
level, through i³ or spooling, be suspicious of its business value. It may be an
accidental configuration. If you discover that a particular field’s 'Use Default
Sensitivity' property was accidentally set to FALSE, set the property value back
to TRUE, using Siebel Tools, as shown below.
30
30
Figure 16: Verify the 'Use Default Sensitivity' option
Siebel timeout
It is common for users to complain that their Siebel session has been timed out
when the query is very slow. In this section we describe how Siebel and Oracle
DB handle the timeout issue in terms of user experience, Siebel Application
Object Manager (AOM), and Oracle DB levels.
Figure 17: Timeout error after approximately 15 minutes
31
31
When a user runs a query, Siebel displays the cancel query dialog box, if the
query runs longer than the value specified in the CancelQueryTimeOut
parameter in the [SWE] section of the application configuration file (CFG). Out of
the box, a value of '3' is used, meaning that the dialog appears whenever a
query takes 3 seconds or longer. Based on our observations, we have seen that
this dialog box stays open for 10 minutes. If a query takes more than 10
minutes, the dialog box is replaced by a more familiar hourglass for the
additional amount of time. The browser times out according to whichever is the
earlier—the browser's timeout setting (10 minutes) or Siebel's SessionTimeout
parameter (default is 300) set in seconds, in eApps.cfg. In either case, if the
Oracle 10g DB has not completed the request when the browser client times out,
it will continue processing even after the browser times out. We have seen cases
where a query ran for more than 24 hours, while the end-user was timed out
after 15 minutes.
Figure 18: The Cancel query is displayed after 3 seconds
In order to solve this problem, customers should use i³’s monitoring and alerting
features. Once i³ identifies these extremely slow queries, the DBA should
implement DB session kill scripts to free DB resources. This process would allow
you to analyze the source of the performance problem and solve it.
Figure 19: Error displayed when a session is killed by the DBA
32
32
Note, when killing the session from the database, Siebel 7.8 launches a Retry
(up to two times) for each DB kill. The Siebel Retry feature is useful in
minimizing disruptions due to NW issues, but the Kill scripts needs to take this
into consideration and kill the queries each time they are restarted.
Export + count(*)
The usage of data export from list views in Siebel applications to spreadsheets
varies across different customer applications. Some customers depend on data
export for their data analysis, whereby they export data into spreadsheet and
manipulate it to obtain the information they need. It is common for such
customers to run into performance problems when attempting to export Siebel
data. When exporting data, Siebel first runs a COUNT(*) SQL query to count the
number of records that need to be exported. This is followed by launching a
second SQL query to obtain the data for the export. Here, the SQL structure is
the same as the one used to populate the list applet in the view, with one
exception: the ALL_ROWS optimizer hint is used. It is common for one or more
of the following conditions to hold, when Siebel export is slow:
(1) Users attempt to export a large number of records from the All or My Team’s
view.
(2) The underlying ad hoc or PDQ query is complex and/or not tuned. The impact
is magnified at export since three different SQLs are being executed—one to
populate the list view (this SQL returns only a small set of rows), and one to
count the records in the actual export.
Stats
Oracle optimizer statistics on tables, columns and indexes are the single most
important factor that drives the generation of optimal query plans. Therefore, it
is important for customers to have a good process for regenerating statistics.
Oracle Corporation suggests that accurate statistics are collected and made
available to the query optimizer. The suggested method for collecting statistics is
the DBMS_STATS package.
In a large implementation, it is a challenge to follow this suggestion. In some
cases, calculating the statistics for a single table can take several hours. Since
statistics are calculated when the system is down to avoid locks, one must have
a good strategy for the timing and methods of creating statistics. Some
customers run statistics every 2 to 3 weeks, while others calculate them every
few months. It is critical that you understand the risk associated with applying
incorrect statistics and have the ability to identify that a performance problem is
a result of this.
This task is fairly easy with i³. Simply identify the performance problem and
drill-down to the SQL level. i³ identifies whether the plan was changed, and
identifies the problematic object in the plan.
33
33
If the plan was not changed, examine the statistics on the object in comparison
with the one used by the optimizer. The optimizer may not have access to
correct information, which it needs in order to choose the best plan. In this case,
you will need to recreate the statistics.
If the plan was changed, i³ will report for which objects, used in the query,
statistics were recently collected.
Stored Outline
When creating a stored outline for an SQL, you instruct the CBO as to which
execution plan to use when the SQL is running. In Siebel Applications, stored
outlines become a very important tool during the emergency tuning process.
Tuning queries is challenging and time-consuming as you have little control over
the query’s text. By creating a stored outline for an SQL performance problem,
you ensure that the SQL performs well, at least on a temporary basis. This gives
you the time to investigate potential permanent solutions. Several init parameter
changes are required (such as Cursor_sharing=FORCE and
Query_rewrite_enabled=TRUE) and a trigger is required in order for stored
outlines to work. Even if you do not use stored outlines at this time, it is a good
idea to make these changes in preparation for the future. Refer to Oracle
documents for a list of required init.ora parameters.
Stored outlines, however, should not be treated as a permanent solution. The
reasons for this are:
The outline is defined to match a specific query text, exactly. The outline would
not be used if any modification to the text is made. For example, assume you
created a stored outline for a query to navigate to a problematic view which
timed out. When the users navigate to the view, performance is good. If,
however, users decide to sort the results, the generated query is different (since
it has a different sort order), the outline will not be used, and performance
would slow down again. Another example can be users who decide to add
additional criteria. In this case, the query will again be different and the outline
will not be used.
Stored outlines require maintenance and pose a potential risk. Since you did not
actually find the cause of the problem and fix it, you used stored outlines to
force the optimizer to use the “correct” plan. As a result, you always need to
know that the stored outlines are still valid and that the same queries are still
executed. For example, you don’t want to find out that a new build, installed for
production on Sunday night, caused some of the problematic queries to change,
because the stored outlines are not used anymore. Also, if stored outlines are
not being used, you do not want to discover this only when users start working
with the system. The best solution is to find the root cause of the performance
problem (why the optimizer is using a poor plan) and fix it.
See Appendix B for a general template for creating a stored outline.
34
34
Appendix B—Templates for SQLPlus and Stored

outlines
SQLPlus Template
The following is a template for running Siebel queries from SQLPlus or other SQL tools.
-- Set SIEBEL session hints

alter session set optimizer_mode = first_rows_10
/
alter session set "_HASH_JOIN_ENABLED" = false
/
alter session set "_OPTIMIZER_SORTMERGE_JOIN_ENABLED" = FALSE
/
alter session set "_OPTIMIZER_JOIN_SEL_SANITY_CHECK" = TRUE
/
-- Declare the bind variables that are used in the query
var v1 varchar2(32);
-- Assign values to the bind variables. Obtain the list of values from i3
begin
select '????' into :v1 from dual;
end;
/
-- Set autotrace and other sqlplus settings
set autotrace traceonly
set lines 1000
-- Execute the query as reported by i3. Don’t replace the bind variables with the literal in the text
SELECT …..
/
spool off
35
35
Stored Outline Template

The following is a template that can be used to create a stored outline. It is
recommended to create the outline in PTE, test it and then export/import the
newly created outline to production.
It is best to copy and paste the text of the query as shown in i3. Remember that
text modification can result in an outline not being used for the problematic
query.
-- Set Siebel session hints

Alter session set optimizer_mode = first_rows_10;
alter session set "_HASH_JOIN_ENABLED" = false;
alter session set "_OPTIMIZER_SORTMERGE_JOIN_ENABLED" = FALSE;
alter session set "_OPTIMIZER_JOIN_SEL_SANITY_CHECK" = TRUE;
-- Create bad outline

CREATE OUTLINE OUTLINE_NAME_BAD ON
SELECT ….
-- Create good outline by using index hint or literal, etc.

CREATE OUTLINE OUTLINE_NAME_GOOD ON
SELECT ….
-Switch the outline between the good and the bad
update outln.ol$hints set ol_name ='TMP1' where ol_name ='OUTLINE_NAME_BAD';

commit;
update outln.ol$hints set ol_name =' OUTLINE_NAME_BAD' where ol_name ='
OUTLINE_NAME_GOOD';
commit;
update outln.ol$hints set ol_name =' OUTLINE_NAME_GOOD' where ol_name ='TMP1';
commit;
/
36
36

Siebel Oracle Database Monitoring

Încărcat de

Informații document

Descriere originală:

Titlu original

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

Siebel Oracle Database Monitoring

Încărcat de

Drepturi de autor:

Formate disponibile

Siebel Oracle Database Monitoring and

Tuning – Detailed Techniques

Eyal Markovich (eyal@totalapm.com)

About Total APM Corporation

About the Author

DOCUMENTATION IS PROVIDED "AS IS" AND ALL EXPRESS OR IMPLIED CONDITIONS,

Missing indexes ............................................................................................. 5

Siebel session hints .......................................................................................11

Siebel CBO hint-related problems ....................................................................11

Predefined Queries (PDQ)—default, and public vs. private PDQs ..........................21

Siebel Out-of–the-Box (OOTB) Business Components (BC) Level Performance

Slow code and Interface .................................................................................27

Connection pooling ........................................................................................28

Workflow scheduling ......................................................................................29

Case-sensitive Config problems .......................................................................30

Siebel timeout ..............................................................................................31

Export + count(*) .........................................................................................33

Stored Outline ..............................................................................................34

Appendix B—Templates for SQLPlus and Stored outlines .....................................35

Precise i³ (also known as Symantec/i³) is an application performance monitoring

Statistics before the index Statistics after the index

Q: Should I index each customized column?

Q: How can I determine if an index is missing for a particular

SELECT * FROM T1.S_OPTY WHERE T1.X_ACCT_NUM = :1

SELECT >1000 I/Os SELECT

S_OPTY >1000 I/Os S_OPTY 10-15 I/Os

Figure 2: In this example the statement is mainly waiting for S_ACT_EMP

Siebel session hints

alter session set optimizer_mode = first_rows_10

Siebel CBO hint-related problems

Figure 4: Sort is performed and the driving table is S_ACT_EMP

Sort 100 rows

400-500 I/Os JOIN

100 I/Os S_ACT_EMP S_EVT_ACT 1 I/O

100 rows 1 row

Figure 5: No sort is performed and the driving table is S_EVT_ACT

5M I/Os S_EVT_ACT S_ACT_EMP 1 I/O

500 I/Os Index Index

Q: How will you know which BC to change to ALL_ROWS?

Figure 6: Changing the BC OracleCBOHint parameter

Bind variable vs. literals—a large result set vs. a small

Figure 7: S_CONTACT with 99.6% of the waits

Figure 8: S_CONTACT is the first table in the plan

Figure 9: The like predicate is the first to be evaluated

Q: Should we change the Field User Property configuration parameter in

Figure 10: Changing Field Use Literal for Like property

Predefined Queries (PDQ)—default, and public vs.

Therefore, if acceptable to the business application, consider one of the

Siebel Out-of–the-Box (OOTB) Business Components

Figure 11: An out-of-the-box denormalized field

When a user queries on the BC field that uses S_EVT_ACT.EVT_STAT_CD, the

Consider the following template of a query for the My Activities view:

Figure 12: 86% of the waits are done on S_EVT_ACT

Text before Denormalization Text after Denormalization

Denormalization, in and of itself, improves performance considerably. Now,

Consequently, we implemented the denormalization solution in the PTE

Slow code and Interface

Figure 14: Number of new connections (Logons) over time

Figure 15: Number of executions of a statement

Case-sensitive Config problems

X_ACCT_NAME like ‘AG%’

It is not recommended to set Siebel application queries to be case-insensitive, at