5/9/2014 Document 1057802.

Concurrent Processing - Best Practices for Performance for Concurrent Managers in E-

Business Suite (Doc ID 1057802.1)
Modified: 19-Dec-2013 Type: BULLETIN

APPLIES TO:

Oracle Concurrent Processing - Version 11.5.0 to 12.2.2 [Release 11.5 to 12.2]

Information in this document applies to any platform.

PURPOSE

Provide the best practices to achieve better performance for concurrent manager in Oracle E-Business Suite.

Please also visit the Concurrent Processing Product Information Center (Note 1304305.1) for the latest in CP
recommendations and solutions.

SCOPE

Applications DBAs, System Administrators involved in configuration and administration of Oracle E-Business Suite.

DETAILS

Best Practices for Performance for Concurrent Managers in E-Business Suite

This Document contains 5 topics

1. Generic Tips

2. Transaction Manager (TM).

3. Parallel Concurrent Processing (PCP) Environment.

4. Tuning Output Post Processor (OPP).

5. Concurrent Processing Server Tuning.

Generic Tips

1) Sleep Seconds - is the number of seconds your Concurrent manager waits between checking the list of pending
concurrent requests (concurrent requests waiting to be started). A manager only sleeps if there are no runnable jobs
in the queue.

Tip: During peak time, when the number of requests submitted is expected to be high, Set the sleep time to a
reasonable wait time(e.g. 30 seconds) dependent on the average run time and to prevent backlog. Otherwise set the
sleep time to a high number (e.g. 2 minutes). This avoids constant polls to check for new requests.

2) Increase the cache size (number of requests cached) to at least twice the number of target processes.

For example, if a manager's work shift has 1 target process and a cache value of 3, it will read three requests, and try
to run those three requests before reading any new requests

Tip: Enter a value of 1 when defining a manager that runs long, time-consuming jobs, and a value of 3 or 4 for

https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=1blhsz9c0g_254&id=1057802.1 1/6
5/9/2014 Document 1057802.1

managers that run small, quick jobs.

This is only guidance and a balance needs to struck in tuning the cache, so with fast jobs you need to cache to get
enough work for a few minutes. With slow jobs, a small queue helps should you need to reprioritize requests.

4) To maximize throughput consider reducing the sleep time of the Conflict Resolution Manager (CRM). The default
value is 60 seconds. You can consider setting to 5 or 10 seconds.

5) Avoid enabling an excessive number of standard or specialized managers. It can degrade the performance due
polling on queue tables (FND_CONCURRENT_REQUESTS...). You need to create specialized managers only if there is a
real need.

6) Set the system profile option "Concurrent: Force Local Output File Mode" to "Yes" if required . You need to apply
patch 7530490 for R12 (or) 7834670 for 11i to get this profile.

Refer Note.822368.1: Purge Concurrent Request FNDCPPUR Does Not Delete Files From File System or Slow
performance

Note:- The profile option "Concurrent: Force Local Output File Mode" is set to "No" by default. After applying the patch,
set the profile option to YES will cause FNDCPPUR to always access files on the local file system, hence FNDCPPUR will
remove the OS files faster.To enable this feature, All Concurrent Manager nodes must be able to access the output file
location via the local filesystem

7) Truncate the reports.log file in log directory. Refer Note.844976.1 for more details

Truncation of file "reports.log" is a regular maintenance work of Application DBA. Make sure that reports log file size
should not increase to its maximum limit of 2 GB. There is no purge program to truncate file "reports.log". This
maintenance needs to be done manually and regularly depending on number of concurrent program which uses
"reports.log". You can safely truncate "reports.log".

The "reports.log" file can be located under $APPLCSF/$APPLLOG.

8) Ensure "Purge Concurrent Request and/or Manager Data, FNDCPPUR," is run at regular intervals with "Entity"
parameter as "ALL". A high number of records in FND_CONCURRENT tables can degrade the performance.

Additionally, the following are very good methods to follow for optimizing the process:

Run the job in hours with low workload. Doing this after hours will lessen the contention on the tables from
running against your daily processing.
To get the requests under control, run the FNDCPPUR program with Age=20 or Age=18 would be a good
method. That means, all requests older than 18 or 20 days will be purged.
Once the requests are under control, run the FNDCPPUR program with Age=7 to maintain an efficient process.
This would solely depend on the level of processing that is performed at your site

9) Ensure that the log/out files are removed from the locations shown below as you run "Purge Concurrent Request
and/or Manager Data program".
$APPLCSF/$APPLLOG
$APPLCSF/$APPLOUT

In the event that it does not remove the log/out files, over a period of time it will slow down the performance. Please
refer to the following note which suggests the patch which fixes it.

Note.822368.1: Purge Concurrent Request FNDCPPUR Does Not Delete Files From File System or Slow performance

10) Defragment the tables periodically to reclaim unused space / improve performance

FND_CONCURRENT_REQUESTS

https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=1blhsz9c0g_254&id=1057802.1 2/6
5/9/2014 Document 1057802.1

FND_CONCURRENT_PROCESSES
FND_CRM_HISTORY
FND_ENV_CONTEXT
FND_TEMP_FILES

HOW TO DEFRAGMENT

10.1) alter table <owner>.<table_name> move;

10.2)Note that, some indexes might become unusable after table is moved, check the index
status from dba_indexes for the table moved and rebuild them too as explained in next
bullet.
select owner, index_name, status from dba_indexes
where table_owner = upper('&OWNER') and
table_name = upper('&SEGMENT_NAME');

10.3) alter index <owner>.<index_name> rebuild online;

Note: Ensure the tablespace in which the object currently exists has got sufficient space
before you move/defragment . Always take backup of the tables before moving the data. It is
recommended to perform ths action on Test instance initially then testing thoroughly before
performing it on Production instance.

10.4) You will need to collect the statistics for the tables

For example:
exec fnd_stats.gather_table_stats ('APPLSYS','FND_CONCURRENT_REQUESTS',PERCENT=>99);

11) Ensure you are in the latest code to avoid below known performance & Deadlock issues issues.

Note 1492893.1 R12: Performance Issue When Standard Managers Waiting for "enq: TX - row lock contention" Held By
ICM
Note 1060736.1 Deadlock Error During Concurrent Request Termination
Note 866298.1 Concurrent Processing - ORA-00060: Deadlock Detected - UPDATE FND_CONCURRENT_QUEUES
Note 1360118.1 Performance: Concurrent Requests Hang in Pending Status For Long Time
Note 1075684.1 Concurrent Managers are consuming high CPU and memory
Note 1541526.1 Performance: Concurrent Requests Hang in Pending Standby Status For Long Time
Note 1542216.1 Concurrent Requests fail Due to FNDSM Log File Size Grows 2 GB ; SQL*Loader-101: Invalid Argument
for username/password

Note 1577982.1 Concurrent Manager FNDSM intermittently Crashes / Shutting Down Abnormally When a Concurrent
Request is Cancelled

Note 1604300.1 Concurrent Manager FNDCRM Down / Crashed All Scheduled Concurrent Programs Are Stuck
Pending/Scheduled Pending/Standby

Transaction Manager(TM)

12 ) Profile Concurrent:Wait for Available TM - Total time to wait for a TM before switchover to next available TM.
Consider setting this to 1 (second).

https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=1blhsz9c0g_254&id=1057802.1 3/6
5/9/2014 Document 1057802.1

13) Ensure enough TMs exist to service the incoming request load.

14) When the load is high, set the following profile to optimum values to achieve better results.

PO: Approval Timeout Value - Total time for workflow call (When initiated from Forms) to time out.

15) Set the sleep time on the Transaction Manager to a high number (e.g. 10 minutes), this avoids constant polls to
check for shutdown requests.

Parallel Concurrent Processing (PCP) Environment

16) If the failover of managers is taking too long refer to Note:551895.1: Failover Of Concurrent Manager Processes
Takes More than 30 Minutes

17) To avoid known issues, apply Patch 15900099 (11i), Patch 15981173 (12.0 ), Patch 15981176 (12.1.3) along with
pre-requisites , when you are in the process of implementing PCP. Also refer NOTE:1389261.1

18) Set profile option 'Concurrent: PCP Instance Check' to 'OFF' if instance-sensitive failover is not required. Setting it
to 'ON' means that concurrent managers will fail over to a secondary application tier node if the database instance to
which it is connected goes down.

19)Transaction Manager uses DBMS_PIPE to communicate with application session prior to 11i.ATG_PF.H RUP3.
DBMS_PIPE in turn uses OS Pipe.We might use Advance Queue(AQ) with 11i.ATG_PF.H RUP3 by setting System Profile
Concurrent: TM Transport Type to QUEUE.

Note Pipes are more efficient but require a Transaction Manager to be running on each DB Instance (RAC). So you
might want to use "Queue" for easy maintenance.

20) Add these parameters depends on your Database version

+ _lm_global_posts=TRUE
+ _immediate_commit_propagation=TRUE (11g RAC)
+ max_commit_propagation_delay=0 (9i RAC)

21) To speed up the PCP Failover ,Tune the below parameters.

Kernel parameters (Find the analogous parameter for your platform)

tcp_keepalive_intvl
tcp_keepalive_probes
tcp_keepalive_time ( Do not set this value to low; since it will then use up your network resources with
unnecessary traffic)

DCD (Dead connection detection) setup; sqlnet.ora from the Database Tier

sqlnet.expire_time

Environment Variable at Concurrent Manager Tier.

FDCPTRW

PMON Cycle & Sleep Intervals for ICM (internal Concurrent Manager) setup.

Navigation OAM -> SiteMap -> Monitoring -> Internal Concurrent Manager Link(Under Availability) -> "View
Status" -> "Edit ICM Runtime Parameters"

Enable Reviver.

What is FNDREVIVER and How Is It Set? Note 466752.1

https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=1blhsz9c0g_254&id=1057802.1 4/6
5/9/2014 Document 1057802.1

PCP references:
NOTE: Multiple active managers are capable of processing the same jobs and can be defined to run at the
same time.
How To Run a Concurrent Program Against a Specific RAC Instance with PCP/RAC Setup? (Document
1129203.1)
How To Setup Parallel Concurrent Processing In R12 (Document 1525233.1)
How to Activate Parallel Concurrent Processing - Background Facts and Setup Steps (Document 602899.1>)

Tuning Output Post Processor (OPP)

In order to tune the OPP to improve performance refer the below Note. It discusses How to monitor the workload of
OPP and Suggests you how to tune Output Post Processor (OPP) to improve performance and avoid
java.lang.OutOfMemoryError exception.

NOTE:1399454.1 Tuning Output Post Processor (OPP) to Improve Performance

Concurrent Processing Server Tuning

1. Any Concurrent Processing (CP) server tuning or load balancing needs are to be addressed by Oracle Consulting.
There are way too many site specific factors that needs to be considered for optimum CP throughput: from machine
hardware, to user request volume, to required Work Shifts, to programs run time characteristics (long / short
running)--not to mention also testing and benchmarking. Such a tasks, is beyond the scope of ATG Support.

ATG support would be glad to investigate a failing manager or program issue; however, CP performance issues due to
increased concurrent request volume or due to a new installation needs to be addressed by Oracle Consulting.

2. The "Tuning Concurrent Processing" chapter of the white paper "A Holistic Approach To Performance Tuning Oracle
Applications Systems Release 11 and 11i" Note 69565.1 may provide some basic insight. Also reference the "Defining
Concurrent Managers" and the "Setting Up and Starting Concurrent Managers" chapters of the "Oracle Applications
System Administrator's Guide - Configuration".

3. As per Note 69565.1 "A Holistic Approach to Performance Tuning Oracle Applications Systems", "50% of concurrent
processing performance tuning is in the business!"

4. Visit the Concurrent Processing Product Information Center (PIC) Note 1304305.1 for additional performance and
setup documentation.

Information Center, Diagnostics, & Community

E-Business Concurrent Processing Information Center Document 1304305.1

Please reference this document regularly to review current offerings for Concurrent Processing needs.

Diagnostics
For additional help, please refer to one of the following documents on diagnostics to address current needs.
Providing diagnostic output on an issue for support when logging a service request is very helpful.

Document 179661.1 for 11i or Document 421245.1 for Rel 12.x

Core Concurrent Processing Community

https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=1blhsz9c0g_254&id=1057802.1 5/6
5/9/2014 Document 1057802.1

Visit the Core Concurrent Processing community for help from industry experts or to share knowledge.

REFERENCES

NOTE:1389261.1 - PCP Concurrent Manager Failover/Failback Does Not Work When Application Listener is Down On
Primary Node
NOTE:1060736.1 - Deadlock Error During Concurrent Request Termination
NOTE:466752.1 - Concurrent Processing - What is FNDREVIVER and How Is It Set?
NOTE:551895.1 - Concurrent Processing - Failover Of Concurrent Manager Processes Takes More than 30 Minutes
NOTE:1360118.1 - Performance: Concurrent Requests Hang in Pending Status For Long Time
NOTE:866298.1 - Concurrent Processing - ORA-00060: Deadlock Detected - UPDATE FND_CONCURRENT_QUEUES
NOTE:1075684.1 - Concurrent Managers are consuming high CPU and memory
NOTE:1399454.1 - Tuning Output Post Processor (OPP) to Improve Performance
NOTE:844976.1 - Concurrent Processing - Concurrent Reports Failing With Errors REP-0004,REP-0082 and REP-0104
NOTE:1492893.1 - R12: Performance Issue When Standard Managers Waiting for "enq: TX - row lock contention" Held
By ICM
NOTE:822368.1 - Concurrent Processing - How To Run the Purge Concurrent Request FNDCPPUR, Which Tables Are
Purged, And Known Issues Like Files Are Not Deleted From File System or Slow Performance
NOTE:104452.1 - Concurrent Processing - Troubleshooting Concurrent Manager Issues (Unix specific)
NOTE:1060707.1 - Purge Concurrent Requests/Manager Data, FNDCPPUR, Not Removing Files From Filesystem

https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=1blhsz9c0g_254&id=1057802.1 6/6