Documente Academic
Documente Profesional
Documente Cultură
Purpose
This document is intended for Customers and Service Professionals that have encountered a problem with the
Oracle BPM Engine and would like to perform some troubleshooting before contacting Technical Support.
Additionally, the document describes the information that Customer Support will generally require and how to obtain
it. While the document has been written based on BPM 5.7 it has also been revised for versions 5.5, 6.0 and 10.3.
Differences between versions are pointed out when they exist.
Introduction
This document provides general information to perform a basic or medium level of Engine troubleshooting. Among
the provided information you will find: description and details of Engine logs, tools for gathering troubleshooting
data, several sources of information depending on the nature of the problem, troubleshooting configurations,
debugging settings, sets of information that should be analyzed or collected in case of an incident, and more.
The type of issue within the BPM Engine can vary widely. Depending on each case, different sets of information
might be required. That is why this document aims at the most common required pieces, which are enumerated
below:
This document describes these different sets of information and provides instructions on how to obtain them.
Additionally, it contains a brief section on how to tune some engine settings to address typical performance issues
and a section on debugging flags.
Table of Contents
The most accurate way of obtaining the Engine version is finding the engine log entry (of severity level Info) that is
generated at startup and contains several pieces of environment information. This entry looks like this:
Another way of obtaining the Engine version is opening the Admin Center About page which is shown below. Note
that while this is the version of the BPM installation, the Engine version could be different due to incorrectly installed
upgrades or fixes. So, when in doubt, it is a good idea to check if both build numbers match.
To find out the version of your Database you should probably contact your DB administrator. There are a few
exceptions where the full version is required, thus knowing the major version number will be enough (for instance:
Oracle 10g, IBM DB2 version 8.1, MS SQL Server 2000, etc).
The JDBC database driver version is generally of greater importance due to the different defects or improvements
that might be present in each release.
The general way of obtaining the drivers version is from the manifest file included in the drivers jar or zip. The
driver might be found at different locations depending on your environment (Standalone, J2EE, etc.) and it is
important to obtain the manifest from the driver that the Engine is using. For instance, in standalone the driver is
found at $BPM_HOME/ext/<db-driver jar or zip>. Please, refer to the BPM Administrators Guide for details on the
location of the driver in J2EE environments:
http://download.oracle.com/docs/cd/E13165_01/albsi/docs57/adminguide/index.html.
Below is an example of a Manifest.mf file (this one was obtained from the file C:\bea\albpm5.7\j2eewl\ext\
ojdbc14.jar, which corresponds to the driver that the Admin Center uses to connect to FDI):
Manifest-Version: 1.0
Specification-Title: Oracle JDBC driver classes for use with JDK14
Sealed: true
Created-By: 1.4.2_08 (Sun Microsystems Inc.)
Implementation-Title: ojdbc14.jar
Specification-Vendor: Oracle Corporation
Specification-Version: Oracle JDBC Driver version - "10.2.0.2.0"
Implementation-Version: Oracle JDBC Driver version - "10.2.0.2.0"
Implementation-Vendor: Oracle Corporation
Implementation-Time: Tue Jan 24 08:55:21 2006
If the BPM Directory (or FDI) is supported by an LDAP, Active Directory or a Database version other that the one
used for the Engine Database you should also provide this Directory provider version information.
If the Engine is deployed in a J2EE environment the Application Server version and the deployment topology are
required to troubleshoot any Engine issue.
The Application Server full version can be obtained in different ways depending on the Vendor. In the next sub-
sections we present the most common examples:
In the WebLogic Server, the version (and the applied patches, if any) can be found in the start log:
<Jan 18, 2008 4:38:39 PM CST> <Info> <WebLogicServer> <BEA-000377> <Starting WebLogic Server
with Java HotSpot(TM) Client VM Version 1.5.0_04-b05 from Sun Microsystems Inc.>
<Jan 18, 2008 4:38:40 PM CST> <Info> <Management> <BEA-141107> <Version: WebLogic Server 9.2
Fri Jun 23 20:47:26 EDT 2006 783464 >
<Jan 18, 2008 4:38:42 PM CST> <Info> <WebLogicServer> <BEA-000215> <Loaded License :
C:\bea\license.bea>
<Jan 18, 2008 4:38:42 PM CST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to
STARTING>
<Jan 18, 2008 4:38:42 PM CST> <Info> <WorkManager> <BEA-002900> <Initializing self-tuning
thread pool>
The Engine can be deployed in a cluster configuration in a Web Application server or it can have a Backup location
in the Standalone version.
When running in a Web Application, the Support analyst will be interested in knowing:
The number of nodes in the cluster where the Engine has been deployed.
The physical locations of the nodes
The number of nodes running at the time of the incident
When running Standalone, the Support analyst will like to know if there is a Backup location and if it was active at
the time of the incident.
The Operating System version is also relevant for troubleshooting the Engine. Minor version details or build
numbers might not be required, though. As general rule, just include in the report Vendor and mayor version (for
instance: Windows XP Sp2, Red Hat Linux ES 4 update 3, Solaris 10 (SPARC), HP-UX 11i V2, etc.)
As you can see, the list of Basic Info generally required is quite long. Thus, we will like you to consider having this
gathered in a document to reduce the time spent at the moment you face an issue.
Have in mind that each support case can be handled by a different analyst who might not be familiar with your
environment. And even in the cases where youve been in contact with that same analyst in the past, he might not
be aware of upgrades or all the different environment configurations that you might have.
Therefore, if you can provide an up-to-date document detailing this Basic Info for the environment where you
detected the problem you will accelerate the case resolution and your Support analyst will appreciate it.
2. Engine Logs
The Engine logs are composed of a fixed number of files with a given maximum size. In order to maintain the log
size within this constrains, when all log files have been written the new log entries will be written to the oldest file in
a round-robin fashion. This limit on the amount of log stored translates into a limited time frame of log info. We will
address how to change this amount in section 2.2.1. Configuring the Log settings in the Process
Administrator.
While the content of the files is text, they are not easily readable without the help of the LogViewer or LogReader.
We will briefly cover them in sections 2.3. Using LogViewer and 2.4. Using LogReader.
Severity Indicates the level of severity of the message. The possible values, in decreasing order of severity,
are: FATAL, SEVERE, WARNING, INFO, DEBUG and TRACE. Note that TRACE is an internal
value used in special occasions to dump stack traces of an Engine thread (this value is not
available to the PBL logMessage method).
Message Text of the entry.
Time Time when the entry was generated. Note that the definition of the value is in seconds. Also note
that the date is stored in GMT-0 format and by default it is converted to the local time zone of the
computer where the LogViewer or LogReader is running. The LogViewer time zone can be
configured in the Preferences dialog.
Date Date when the entry was generated.
Application The application that sent the message. The possible values are:
* Default: corresponds to internal entries generated by the Engine and
* CIL: corresponds to entries generated by the project code (see section 3. Project
Debugging for more info).
Module Module that sent the message. Possible values are:
* Main: it is the most frequent value.
* MailDispat: corresponds to entries generated by the Mail Dispatcher module.
Thread Name of the Thread that generated the entry. Note that some thread names might be truncated
(for instance: <8>...tenerThreadPool : 2), but no information is lost because of that.
This particular kind of messages generated by the Engine is very useful when troubleshooting. These messages
have always a severity level of Info, their application is Default, and their module is Main. The following is an
example in text format for version 5.5 and 5.7:
<I>, "Oct 1, 2009 5:13:35 AM", Default, Main, <47...spatcher (63372763), "Executing item:
RUN_AUTOMATIC Inst (214,19093433,0) Act [ApplyConfig] Proc [/ResolveOperations#Default-1.4]
Due [2009-10-01 02:05:28-04 1254377128000000] TimeStamp [2009-10-01 02:05:28-04] id[57671033]
later [false]"
The values shown in this type of messages follow the following pattern:
As you can see, this message is telling us what type of execution (ToDo_Item_Type), process, activity, and
instance the engine is about to run. For a listing and description of the To-Do Types, refer to section 5.1.1. To-Do
Item Types.
The following is an example in text format for version 6.0 and 10.3:
<D>, "Aug 25, 2009 4:30:45 PM", Engine, Main, <5> ET(3), "Executing item: RUN_AUTOMATIC Inst
(1,21,0) Act [StartExecution[End]] Proc [/LeaveProcess#Default-1.0] Due [2009-08-25 17:30:45-
04:00 1251235845000000] TimeStamp [2009-08-25 17:30:45-04:00] id[63150014] later [false]"
The only difference in the pattern is the addition of the Micro-Activity Id, as shown below:
The component elements of the message listed here are not going to be described in this document. If you have
questions about them, please, contact Oracle BPM Support (see section 12. Contacting Oracle BPM Support).
The default location is $BPM_HOME/log and the value can be modified in the following Process Administrator page
(Engines <engine-name> Basic Configuration [Tab]):
The files follow the following name convention: <engine-name>.log.<index>. Where index takes the values from 0
to the maximum number of log files and tz. For instance, for the engine used in this example j2eewl57engine
we will have the following set of files:
C:\bea\albpm5.7\j2eewl\log\j2eewl57engine.log.0
C:\bea\albpm5.7\j2eewl\log\j2eewl57engine.log.1
C:\bea\albpm5.7\j2eewl\log\j2eewl57engine.log.2
C:\bea\albpm5.7\j2eewl\log\j2eewl57engine.log.3
C:\bea\albpm5.7\j2eewl\log\
C:\bea\albpm5.7\j2eewl\log\j2eewl57engine.log.tz
Important: All files listed above are part of the Engine Log. This means that none of the BPM Log applications will
be able to read the logs if any of these files are missing. Therefore, please, include all <engine-name>.log.* files
when sending the Engine Log to a Support analyst. An easy way to make sure all files were gathered is to
compress the whole log directory (for instance, C:\bea\albpm5.7\j2eewl\log).
In cluster environments with more than one Node running on the same Host, you will need to specify a different log
location for each Engine node. This can be achieved by setting the following JVM additional parameter to the Java
Virtual Machine that runs the Engine Node:
-Dfuego.server.log.file=<file-path>/<log-filename>.log
Note that log-filename (that by default corresponds to the engine name) here can be set to any other value. Like,
for instance, the name of the Application Server Node.
Please, have in mind that this parameter will override the value specified by the Process Administrator. Therefore,
any change to the log directory in the Process Administrator will be ignored.
In the following Process Administrator snapshot (Engines <engine-name> Log [Tab]) three settings have
been changed from the default:
1. Messages Logged from Server: these are the internal BPM Engine messages. The value has been set to
Debug level to capture full execution details.
2. Messages Logged from BP-Methods: these are the messages logged by the project code (see section 3.
Project Debugging for more info). The value has been set to Debug level.
3. The Maximum Number of Log Files has been increased to 20 (being this the minimum recommended for
this configuration). This is important since having log levels set in Debug means hundreds of messages
more than in the default scenario and, in turn, this means that the log time frame will be reduced (given that
the number of log files is fixed and that old items are overwritten once the maximum number of files has
been reached).
2.2.2. Enabling Asserts in the JVM that runs the BPM Engine
In BPM versions 5.5 and 5.7 (but not in 6.0 and 10.3), Java Asserts must be enabled in order for Debug messages
from the Engine to be generated. This configuration involves adding an additional JVM argument (-ea) to the start
script. The place where to add this argument depends on what type of engine you are running. The following
sections describe how to enable asserts in the Standalone, WebLogic and WebSphere versions.
For the Standalone Engine (and not for the J2EE versions), the additional JVM argument can be added in the
Execution tab of the corresponding engine within the Process Administrator. An example is shown below:
The following snapshot corresponds to WebLogic Server version 9.2.0. The argument -ea has been added to the
Server Start page of the Server where the BPM Engine is deployed.
If, for any reason, this configuration change does not affect the actual java command executed when WLS starts
(see section 2.2.3.1. Start output of the WebLogic Server), then change the startWebLogic.cmd or .sh
(depending on the OS) adding the -ea option as follows:
In Windows OS (startWebLogic.cmd):
In Unix-like OS (startWebLogic.sh):
JAVA_OPTIONS=$SAVE_JAVA_OPTIONS ea
The following snapshot corresponds to WebSphere Application Server version 6.0.2. The argument -ea has to be
set to the Generic JVM arguments field on the JVM page of the server running the BPM Engine (e.g. Application
servers server1 Process Definition Java Virtual Machine).
Once the changes are completed, you need to restart the Engine. In a Standalone environment this can be done
from the Process Administrator while in a J2EE environment this implies restarting the EAR, the node or the cluster
depending on the deployment topology.
The start log for WLS should look like the following (Note that the JAVA Options shown at the beginning is not
printed by the default script):
***************************************************
* To start WebLogic Server, use a username and *
* password assigned to an admin-level user. For *
* server administration, use the WebLogic Server *
* console at http:\\hostname:port\console *
***************************************************
starting weblogic with Java version:
java version "1.5.0_04"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_04-b05)
Java HotSpot(TM) Client VM (build 1.5.0_04-b05, mixed mode)
Starting WLS with line:
C:\bea\JDK150~1\bin\java -client -Xms256m -Xmx512m -XX:CompileThreshold=8000 -XX:PermSize=48m -
XX:MaxPermSize=128m -Xverify:none -da -Dplatform.home=C:\bea\WEBLOG~1 -Dwls.home=C:\bea\WEBLOG~1\server
-Dwli.home=C:\bea\WEBLOG~1\integration -Dweblogic.management.discover=true -Dwlw.iterativeDev= -
Dwlw.testConsole=-Dwlw.logErrorsToConsole= -
Dweblogic.ext.dirs=C:\bea\patch_weblogic920\profiles\default\sysext_manifest_classpath -ea -
Dweblogic.Name=AdminServer -Djava.security.policy=C:\bea\WEBLOG~1\server\lib\weblogic.policy
weblogic.Server
Finally open the LogViewer and verify that Debug Messages coming from the server are being generated
(Application is Default for messages coming from the Engine and CIL for messages coming from BPM Methods
or custom code).
The BPM LogViewer is a graphical application to read and perform different type of searches of log entries. Do not
confuse it with the simple html Log Viewer, available at the Process Administrator (Engines <engine-name>
Basic Configuration [Tab] Log Viewer). The latter only allows to browse the entries and does not provide the
search functionality that the former does.
If BPM Enterprise is installed on a box with a graphical environment, the LogViewer can be started from the
following script: $BPM_HOME/bin/albpmlogviewer.sh (an example in windows will be: C:\bea\albpm5.7\j2eewl\
bin\albpmlogviewer.exe). The application can read log files located locally or remotely.
For more information on this application, please, refer to the LogViewer documentation at:
http://download.oracle.com/docs/cd/E13165_01/albsi/docs57/olh_logviewer/index.html
The BPM LogReader is a command line application that parses the contents of the Engine log files and writes the
log items in a readable format to the Standard Output. This application is useful in environments where a graphic
terminal might not be available. The recommended use is to redirect the output to a file and then parse it with a tool
like grep or simply open it with a text editor.
Note that the log filename <engine log file location>.log does not require an index after the .log.
This is an example of a grep command to find all items regarding a particular instance (in this case, instance 1234):
The followings are copies of the Windows script, in case your BPM distribution does not contain it:
In a J2EE environment it is also recommended to gather the corresponding Application Server Logs. This is so
because some error messages from the underlying mechanisms that the J2EE Engine uses are not be logged in
the Engine logs.
The following sub-sessions describe the default location for these logs. When in doubt, please contact your
Application Server Administrator.
The default location of the logs for the WebLogic Server is: $BEA_HOME/user_projects/domains/<domain-name>/
servers/AdminServer/logs (for instance, in a single node Windows installation where the domain name is ALBPM
the logs can be found at: C:\bea\user_projects\domains\ALBPM\servers\AdminServer\logs).
The default location of the logs for the WebSphere Application Server is: $WEBSPHERE_HOME/AppServer/
profiles/default/logs/<server-name> (for instance, in a single node Windows installation where the server name is
server1 the logs can be found at: C:\Program Files\IBM\WebSphere\AppServer\profiles\default\logs\server1).
3. Project Debugging
Whenever you experience a problem with the BPM Engine there is chance that the issue is triggered by:
- an aspect of your project design,
- a defect in the code of the project,
- a defect in one of the resources utilized by the Engine (and/or the Project),
- or a defect in the Engine.
In order to rule out the first two causes it is always of great help to include sufficient log messages in the BPM
Project code. In this section we are going to indicate how to do it in a meaningful way.
BPM has a predefined method to add entries to the Engine logs. The method is called logMessage and below is
shown a basic example of its use:
logMessage("message", Severity.DEBUG);
As you might have seen in section 2.1.1. A log entry the list of fields of a log entry does not include Process,
Instance of Method information. Thus, in order to provide meaningful log information we strongly recommend to
include in all your entries as much info about the Instance or Process as possible. Bellow youll find two best
practice examples for the most common cases:
3.2.1. Log message for Instance Methods (this is, that have access to instance information. This includes methods
in most activities except from Global Activites -that do not have instance access-):
3.2.2. Log messages for Static Methods (this is, that do not have access to instance information. This includes
methods in Global Activites -that do not have instance access-):
When sending a Project to Support you should Export the relevant Projects from BPM Studio including all libraries.
This way we will have access to versionable and non-versionable libraries that your project might use.
Below is a snapshot of the option in Studio to perform, what we can call, a full export:
In the case that you are uncertain of what Process version is actually deployed in the environment where the
problem has being found, the way to retrieve it from the Process Administrator is accessing the Project History by
following the link shown below:
Then you can either select the link on Processes or Revisions in the following page:
Until you get to the relevant Revision (depending on the case, it could be the latest one or not) and there youll find
the link to Download the process files for it (called Recover the project revision), as shown below:
It is generally relevant to include in your report an export of the Pending Execution Items. There are two tables that
contain this information: one is PTODOITEMS and the other is PWKTODO and both can be found in the Engine
Database. The tables contain a number of rows in the order of hundreds (or less), so exporting them as comma
separated values and then compressing them should produce a file in the order of 100kb. Please, remember to
include all columns and all rows in the export.
When navigating the Pending Execution Items table (PTODOITEMS), you might find useful to know what the
meaning of the different type ids is. The following is a list valid from version 5.5 onwards:
0 = RUN_AUTOMATIC 16 = ABORT_COPIES
1 = INSTANCE_EXPIRATION 17 = COMPONENT_NOTIFICATION
2 = ACTIVITY_EXPIRATION 18 = INSTANCE_RESUME
3 = MAXRETRY_EXCEPTION 19 = UNSELECT
4 = GENERIC_EXCEPTION 20 = BAM_UPDATER
5 = IPC_SEND 21 = SHUTDOWN
6 = IPC_PROCESS_TW 22 = POLLING_DIRECTORY
7 = IPC_PROCESS_NW 23 = CADUCATOR
8 = WAKEUP_CADUCATOR 24 = GAUTOMATIC_INIT_LISTENING
9 = IPC_INTERRUPTION 25 = PROCESS_CHECKPOINT
10 = GAUTOMATIC_POLLING 26 = RUN_IMMEDIATE
11 = DAILY 27 = ABORT_INSTANCE_EXCEPTION
12 = REPLACEMENT 28 = START_SERVICES
13 = WAIT_NOTIFICATION_TIMEOUT_EXCEPTION 29 = GAUTOMATIC_JMS_LISTENING
14 = ITEM_RECOVERY 30 = UNDEPLOY
15 = GAUTOMATIC_SCHEDULED 31 = GAUTOMATIC_JMX_LISTENING
The description of the To-Do item types exceeds the scope of this document. If you have questions, please, contact
Oracle BPM Support (see section 12. Contacting Oracle BPM Support).
5.2. Obtaining affected entries in the table of Instances and Instance Events
In the case that you have found a problem with a specific instance it is always a good idea to collect the relevant
entries in the table of Instances (PPROCINSTANCE) and, optionally, the entries in the table of Instance Events
(PPROCINSTEVENT). Both of these tables are also located in the Engine Database. The following are examples of
how to obtain these entries:
5.2.a. Retrieves all copies of the Instance 1234 from the PPROCINSTANCE table:
select CREATIONTIME, NTHREADS, PROCESSID, HASPARTFORROLE, ACTIVATIONTIME, ITEMDEADLINE,
SOURCEACTIVITYNAME, PROCESSDEADLINE, NREMARKS, PENDINGITEMID, ACTIVITYNAME, NEXTPARTICIPANT,
PRIORITY, NAME, ACTIVITYDEADLINE, THREADID, PARENTTHREAD, CTHREADS, RECEIVETIME, LASTRESULT,
ROLEIN, INSTID, NCHANGES, NATTACHMENTS, STATE, INSTANCEDATA, AUTHOR, TOTALTHREADS, PARTICIPANT
from <Engine_Schema>.PPROCINSTANCE where INSTID = 1234;
5.2.b. Retrieves all events for the Instance 1234 from the PPROCINSTEVENT table:
select ACTIVATETIME, EVENTDATA, EVENTID, PROCESSID, THREADID, TYPE, ACTIVITYNAME, RECEIVETIME,
CATEGORY, EVENTTIME, INSTID, PARTICIPANT
from <Engine_Schema>.PPROCINSTEVENT where INSTID = 1234;
In the preceding examples we have included all BPM relevant columns excluding project sensitive information (i.e.
external variables: columns that start with V_). But you might as well run a select * if, for instance, the problem
was found on a test environment where external variables do not contain sensitive info for your business.
When navigating the Process Instance table (PPROCINSTANCE), you might find useful to know what the meaning
of the different states is. The following is a list valid from version 5.5 onwards:
Note that several states can be present at the same time. When that is the case, the instance state is the addition
of the current states. For example, if the instance state is RUNNING and SHARING_VARIABLES, the
corresponding state code is 1 + 1024 = 1025. Conversely, if you find a state code of 1088, this means that the
instance state is 1088 = 1024 + 64, which means ABORTED and SHARING_VARIABLES. Another way of looking
at the state code is analyzing it as a binary word where each bit represents a different state. The individual states
th
are encoded with a bit in the n position of the word (being n the power of 2 shown in the list above).
The description of the states exceeds the scope of this document. If you have questions, please, contact Oracle
BPM Support (see section 12. Contacting Oracle BPM Support).
When navigating the Instance Event table (PPROCINSTEVENT), you might find useful to know what the meaning
of the different type ids is. The following is a list valid from version 5.5 onwards:
0 = INSTANCE_CREATED 13 = INSTANCE_UNSELECTED
1 = INSTANCE_END 14 = SUBFLOW_INSTANCE_CREATED
2 = ACTIVITY_IN 15 = INSTANCE_ABORTING
3 = ACTIVITY_OUT 16 = INSTANCE_TO_EXCEPTION
4 = GLOBAL_EXECUTED 17 = INSTANCE_ABORTED
5 = ITEM_SELECTED 18 = ACTIVITY_COMPLETED
6 = ITEM_UNSELECTED 19 = INSTANCE_UPDATED
7 = ITEM_EXECUTED 20 = USER_EVENT
8 = INSTANCE_SUSPENDED 21 = INSTANCE_REASIGN
9 = INSTANCE_RESUMED 22 = MEASUREMENT_START
10 = INSTANCE_GRABBED 23 = MEASUREMENT_STOP
11 = INSTANCE_UNGRABBED 24 = MEASUREMENT_START_STOP
12 = INSTANCE_SELECTED 25 = VARIABLE_MEASUREMENT
The description of the event types exceeds the scope of this document. If you have questions, please, contact
Oracle BPM Support (see section 12. Contacting Oracle BPM Support).
The following is a list of useful queries to the Engine Schema (using Oracle DBs functions and predefined
variables).
To understand in detail some of the information returned from these queries you will need to execute the following
queries against the Directory Schema (formerly known as FDI). These are some of the queries:
One way to perform this is by clicking the Thread Dump button in the Process Administrator Engine page, as
shown below:
This option might not work in all environments. In addition to that, it might be cumbersome to collect or copy the
results. In any case, there are other simply alternatives that will depend on the Operating System.
For instance, in Windows if the command console that runs the Engine JVM is available (like it will be, for instance,
when running WebLogic Server) you can obtain a FTD by pressing the Control+Break keys in the command
console window.
If the command console is not available, you should use an alternative way like the one described in section 6.4.
Obtaining FTDs using other tools.
In Unix-like OSs you can send a signal to the JVM process to generate the FTDs. You simply need to find out what
is the process id (or pid) and then send a SIGQUIT to it.
And once you have the process id, the SIGQUIT can be sent with the following command:
kill -3 <pid>
And advantage of this command is that you can create a simple script that calls it iteratively and with that collect
several consecutive FTDs at a regular time interval. A simple example of such script for a Unix system is:
while true
do
kill -3 $1
sleep .5
done
Here we mention a couple of free tools to obtain Full Thread Dumps. Note that these tools require the process id of
the JVM running the Engine. Weve seen how to obtain it in Unix OSs in the previous section. For Windows OSs
you will find it in the Task Manager window.
jstack is an application to obtain FTDs provided with Sun JVM 1.5.0 (and newer versions). It should be found in the
bin directory of the JDK or JRE Java installation. For more details about its use and syntax refer to:
http://java.sun.com/j2se/1.5.0/docs/tooldocs/share/jstack.html
Other useful tool to obtain FTDs is StackTrace by AdaptJ Systems (http://www.adaptj.com/main/stacktrace). The
tool can be launched online or it can be downloaded from the following site: http://www.adaptj.com/main/download.
In some scenarios for instance, on a JVM crash- a javacore dump is generated. Javacores are a more
comprehensive text outputs than FTDs, but they are rarer and more difficult to obtain. Additionally, not all JVM
support them in the same way. A continuation there is some basic information about them.
A Fatal Error Log is the equivalent of a javacore dump in the Sun Java SE 6 release. For more info refer to:
http://java.sun.com/javase/6/webnotes/trouble/TSG-VM/html/felog.html
This is a tool that reads and analyzes IBM JVM javacores. More info on the tool can be found at:
http://www.alphaworks.ibm.com/tech/jca
This is the best option for gathering information on an Out of Memory condition, since it does not affect
performance and it is easy to configure (compared with Java Profiling tools).
The option is available in Suns JVM 1.4.2 update 12; 5.0 update 9 and newer versions and on IBMs JVM. To
enable it, pass the additional argument to the JVM:
-XX:+HeapDumpOnOutOfMemoryError
Sun jhat is the basic tool for Heap Dump analysis. A recommendation about its use is that the memory assigned to
the application should be as much as possible (you will see that Heap Dumps are quite big and opening them
requires a lot of memory in order to prevent OutOfMemoryExceptions). The following is an example where the
memory has been set to 1024 Mb:
For more details about its use and syntax refer to: http://java.sun.com/javase/6/docs/technotes/tools/share/jhat.html
IBM HeapAnalyzer is nice graphical tool. The only problem with it is that it requires a memory larger than the size of
Java heaps to be able to open them (it will result in an OutOfMemoryException otherwise). In fact, it requires more
memory than jhat. This tool is available at: http://www.alphaworks.ibm.com/tech/heapanalyzer
The following document is of great help: Troubleshooting Guide for Java SE 6 with HotSpot VM. You can find it
at: http://java.sun.com/javase/6/webnotes/trouble/TSG-VM/html/index.html
Note that even when the document is designed for Java SE 6 many recommendations and tools also apply to
earlier versions. For instance, Java SE 6 jhat can open heap dumps generated with Java 1.5.0.
JRockit JVM has a profiling API and the implementation comes with a set of tools: the JRockit Mission Control
tools suite. For details visit:
Profiling and Debugging with WebLogic JRockit:
http://download.oracle.com/docs/cd/E13188_01/jrockit/docs81/portblty/jvmtools.html
JRockit Mission Control: http://dev2dev.bea.com/jrockit/tools.html
JRockit Mission Control Documentation: http://e-docs.bea.com/jrockit/tools/
There is a list of them. We have used some of them (JProfiler, for instance) and they proved to be of great help in
identifying performance issues. Here are some examples:
JProfiler: http://www.ej-technologies.com/products/jprofiler/overview.html
YourKit: http://www.yourkit.com/
JProbe: http://www.quest.com/jprobe/
JIP: http://jiprof.sourceforge.net/
Extensible Java Profiler: http://ejp.sourceforge.net/
9.1. Performance deterioration due to insufficient space in the PAPI Instance cache
This is likely to affect projects with a considerable number of active instances per project and an elevated number
of users executing filters (or opening the Inbox). The PAPI cache is a cache of header information for instances. Its
main purpose is to eliminate unnecessary access to Instance tables by providing an abbreviated description of the
instance state and its main variables. The default size of the PAPI cache is 10,000 instances per process. In some
circumstances this number will fall short and you might find messages in the Engine log indicating the following:
The best indicator for this type of problem is to execute the query 5.3.1. If there are more than 10,000 active
instances per process, you should increase the PAPI cache size. A good rule of thumb is to set the cache size to
double the number of current active instances per process. Values up to 1,000,000 can be used. The setting is
available in the WorkSpace (or Portal) configuration file (or in your custom PAPI application configuration). An
example of the location of this file in BPM 5.7 (which is similar to 5.5) is C:\bea\albpm5.7\enterprise\
webapps\portal\WEB-INF\portal.properties. The property within this file is the following:
fuego.portal.papi.instancesCacheSize=10000
fuego.workspace.papi.instancesCacheSize=10000
Please, remember that after changing this property in a WorkSpace running as a J2EE application you will need to
regenerate the WorkSpace EAR and redeploy it.
The Engine Database is designed to handle hundreds of thousands of active instances. As the total number of rows
in the different engine tables grow, the engine overall performance will deteriorate. Therefore, to reach optimal
performance it is highly recommended to configure the Disposer. This is a mechanism that will either delete or
archive Completed or Aborted Instances. The following screenshot shows where the Disposer related settings can
be configured and the generally recommended values. For more details, please, refer to the BPM Administrators
guide: http://download.oracle.com/docs/cd/E13165_01/albsi/docs57/adminguide/index.html.
In J2EE environments Engine transactions are regulated by the J2EE Server transaction manager. One of the
functionalities of this manager is to coordinate transactions across resources and makes sure that the total time
spent in a particular transaction does not exceed the global transaction timeout (or JTA timeout). Although the
default timeout might be sufficient for small projects in test environments, this value is not recommended for
production environments. In the latter case, the minimum recommended JTA timeout is 5 minutes (300 seconds).
For instructions on how to set this value, please, refer to your J2EE Server documentation.
A JTA Transaction timeout exception can be identified by the following message in the engine log:
Whenever you are experiencing this SQL Exception (no matter what is the full stack trace); the first step you need
to take is to verify that the JTA timeout is not smaller than 5 minutes. If it is, the value should be increased to 5 or
more minutes (a timeout of up to 8 minutes is reasonable for most environments). Note that if the JTA timeout will
affect other projects in the J2EE environment, you should check with the J2EE Administrator before making any
change.
In J2EE environments with short global transaction timeouts (JTA timeout) you might find that the Disposer is
unable to process the default number of instances (1000) in a single transaction. That is why on the latest 5.7 build
(and newer versions) we have made available a JVM property to modify the number of instances deleted per
transaction. The property is the following:
-Dfuego.server.amountInstancesToCommit=200
The value shown above is the recommended setting to be used when the problem has been found on the
environment. The common indicator for this type of problem is a SQLException caused by a transaction rolled
back. The stack trace will contain invocations to these methods:
o fuego.server.persistence.Persistence.getInstancesToDispose()
o fuego.server.execution.Caducator.disposeInstances()
o fuego.server.execution.Caducator.execute()
You should note that the syntax of the method might vary depending on the version. Therefore, when looking at the
logs you should look for key words like: "getInstancesToDispose", "caducator", "dispose", etc
Additionally, you might find the following message:
o "Could not dispose instances from process '/<process_name>#Default-<version>'"
In J2EE environments with short global transaction timeouts (JTA timeout) you might find that the Scheduler is
unable to process the default number of later To Do Items (500) or non-later To Do Items (5000) in a single
transaction. That is why on the latest 5.7 build (and newer versions) we have made available two JVM properties to
modify the number of To Do Items processed per transaction. The properties are the following:
-Dfuego.ebjengine.todoqueue.maxitems.nonlater=1000
-Dfuego.ebjengine.todoqueue.maxitems.later=100
The values shown above are the recommended settings to be used when the problem has been found on the
environment. The common indicators for these types of problems are:
Non-later items timeout problem: You will find a SQLException caused by a transaction rolled back. The
stack trace will contain invocations to these methods:
o fuego.ejbengine.ejb.EngineStartupBean.recoverLostItems()
o fuego.ejbengine.servlet.SchedulerServlet$RecoverLostItemsTask.runImpl()
Later items timeout problem: You will find a SQLException caused by a transaction rolled back. The stack
trace will contain invocations to these methods:
o fuego.ejbengine.ejb.EngineStartupBean.sendLaterItems()
o fuego.ejbengine.servlet.SchedulerServlet$SendItemsTask.runImpl()
You should note that the syntax of the method might vary depending on the version. Therefore, when looking at the
logs you should look for key words like: "recoverLostItems", "sendLaterItems", "Scheduler", etc
If instances in the WorkSpace are not being refreshed appropriately or if there is some other synchronization
problem between the Engine and the WorkSpace, then you should enable the PAPI News Debug property. This
property is a JVM system property that must be specified in the Java Optional Arguments of the JVM running the
Engine and in the JVM running the WorkSpace. Its syntax is show below:
-Dfuego.papi.news.debug=true
For more information on this property, please, refer to the Issue # 7729408 (in https://bug.oraclecorp.com/) or
contact Oracle BPM Support (see section 12. Contacting Oracle BPM Support).
If participants and their entitlements are not being synchronized appropriately or if there is some other
synchronization problem between the Directory and the Engine, then you should enable the Directory Polling
Debug property. This property is a JVM system property that must be specified in the Java Optional Arguments of
the JVM running the Engine and in the JVM running the WorkSpace. Its syntax is show below:
-Dfuego.fdi.polling.debug=true
Note that this property is only available in version 6.0 and 10.3.
For more information on this property, please, refer to the Issue # 8463913 (in https://bug.oraclecorp.com/) or
contact Oracle BPM Support (see section 12. Contacting Oracle BPM Support).
If the Directory has a Hybrid configuration (LDAP + DB) and there are delays in the synchronization of the
Directory, then you should enable the Hybrid LDAP Debug property. This property is a JVM system property that
must be specified in the Java Optional Arguments of the JVM running the Engine and in the JVM running the
WorkSpace. Its syntax is show below:
-Dfuego.fdi.hybrid.ldap.debug=true
For more information on this property, please, refer to the Issue # 8351448 (in https://bug.oraclecorp.com/) or
contact Oracle BPM Support (see section 12. Contacting Oracle BPM Support).
Finally, if you have any questions regarding this document feel free to email at: gustavo.cavallin@oracle.com