Documente Academic
Documente Profesional
Documente Cultură
Contents
LOAD & PERFORMANCE TESTING .................................................... ERROR! BOOKMARK NOT DEFINED.
1 STRESS, LOAD, SOAK, SPIKE TESTS LOAD PROFILES ........................................ ERROR! BOOKMARK NOT DEFINED.
2 LNP TESTING LIFE CYCLE.......................................................................... ERROR! BOOKMARK NOT DEFINED.
3 EXAMPLES OF LNP TESTING OBJECTIVES ...................................................... ERROR! BOOKMARK NOT DEFINED.
3.1 RESPONSE TIME......................................................................................... ERROR! BOOKMARK NOT DEFINED.
3.2 RELIABILITY............................................................................................... ERROR! BOOKMARK NOT DEFINED.
3.3 CONFIGURATION SIZING .............................................................................. ERROR! BOOKMARK NOT DEFINED.
3.4 CAPACITY PLANNING .................................................................................. ERROR! BOOKMARK NOT DEFINED.
3.5 REGRESSION ............................................................................................. ERROR! BOOKMARK NOT DEFINED.
3.6 BOTTLENECK IDENTIFICATION ...................................................................... ERROR! BOOKMARK NOT DEFINED.
3.7 SCALABILITY .............................................................................................. ERROR! BOOKMARK NOT DEFINED.
BPEL PM: INTRODUCTION ............................................................... ERROR! BOOKMARK NOT DEFINED.
1 BPEL PM PROCESS TYPES ....................................................................... ERROR! BOOKMARK NOT DEFINED.
1.1 BY INTERFACE ........................................................................................... ERROR! BOOKMARK NOT DEFINED.
1.2 BY DURABILITY .......................................................................................... ERROR! BOOKMARK NOT DEFINED.
2 TERMINOLOGY ...................................................................................... ERROR! BOOKMARK NOT DEFINED.
3 DIFFERENCE BETWEEN ASYNC & SYNC BPEL .............................................. ERROR! BOOKMARK NOT DEFINED.
4 SOA 11G CONFIGURATION FILES AND LOG FILES LOCATION ............................. ERROR! BOOKMARK NOT DEFINED.
5 DEFAULT DATA SOURCES ......................................................................... ERROR! BOOKMARK NOT DEFINED.
6 PERFORMANCE OBJECTIVE ....................................................................... ERROR! BOOKMARK NOT DEFINED.
7 WHAT TO TUNE .................................................................................... ERROR! BOOKMARK NOT DEFINED.
8 SYSTEM SETUP ...................................................................................... ERROR! BOOKMARK NOT DEFINED.
9 IMPORTANT POINTS ............................................................................... ERROR! BOOKMARK NOT DEFINED.
10 REFERENCE......................................................................................... ERROR! BOOKMARK NOT DEFINED.
JVM TUNING .................................................................................................................................... 15
1 PAUSE TIMES AND THROUGHPUT ....................................................................................................... 16
1.1 GARBAGE COLLECTION VS. THROUGHPUT ................................................................................................... 16
1.2 DEFRAGMENTATION PAUSES AND THROUGHPUT ......................................................................................... 17
2 PERFORMANCE AND MEMORY FOOTPRINT ........................................................................................... 17
3 JVM CHOICE ................................................................................................................................. 17
4 JROCKITS TUNING .......................................................................................................................... 18
4.1 BOARDING INTO SHIP STEP 1 ................................................................................................................. 18
4.1.1 Heap Size ......................................................................................................................................... 18
4.1.2 Garbage Collection.......................................................................................................................... 19
4.1.3 Nursery Size ................................................................................................................................... 20
3.6 AUTOMATICALLY RECOVERING LEAKED CONNECTIONS ..................................... ERROR! BOOKMARK NOT DEFINED.
3.7 LIMITING STATEMENT PROCESSING TIME ...................................................... ERROR! BOOKMARK NOT DEFINED.
3.8 PINNED TO THREAD PROPERTY .................................................................... ERROR! BOOKMARK NOT DEFINED.
3.9 GRIDLINK ................................................................................................. ERROR! BOOKMARK NOT DEFINED.
3.10 SINGLE CLIENT ACCESS NAME.................................................................... ERROR! BOOKMARK NOT DEFINED.
3.11 LOGGING LAST RESOURCE ......................................................................... ERROR! BOOKMARK NOT DEFINED.
4 DATABASE MODE .................................................................................. ERROR! BOOKMARK NOT DEFINED.
5 REFERENCE .......................................................................................... ERROR! BOOKMARK NOT DEFINED.
BPEL ENGINE TUNING ..................................................................... ERROR! BOOKMARK NOT DEFINED.
1 REAL TIME VIEWING OF PERFORMANCE METRICS .......................................... ERROR! BOOKMARK NOT DEFINED.
1.1 FUSION MIDDDLEWARE CONTROL ................................................................ ERROR! BOOKMARK NOT DEFINED.
1.2 DYNAMIC MEMORY SERVICE ....................................................................... ERROR! BOOKMARK NOT DEFINED.
2 BPEL ENGINE THREADING MODEL............................................................. ERROR! BOOKMARK NOT DEFINED.
2.1 DISPATCHER MAXIMUM REQUEST DEPTH...................................................... ERROR! BOOKMARK NOT DEFINED.
2.2 INVOKEMESSAGES IN CACHE ....................................................................... ERROR! BOOKMARK NOT DEFINED.
3 TRANSACTION TIMEOUT .......................................................................... ERROR! BOOKMARK NOT DEFINED.
3.1 SYNCMAXWAITTIME ................................................................................. ERROR! BOOKMARK NOT DEFINED.
3.2 BPEL EJBS ............................................................................................... ERROR! BOOKMARK NOT DEFINED.
3.3 GLOBAL TRANSACTION TIMEOUT AT WEBLOGIC DOMAIN LEVEL ........................ ERROR! BOOKMARK NOT DEFINED.
3.4 WEBSERVICE READ AND CONNECTION TIMEOUT ............................................. ERROR! BOOKMARK NOT DEFINED.
3.5 RELATIONSHIP AMONG TIMEOUT PARAMETERS .............................................. ERROR! BOOKMARK NOT DEFINED.
3.6 WLS TIMEOUT PARAMETERS ....................................................................... ERROR! BOOKMARK NOT DEFINED.
3.6.1 Complete Message Timeout .............................................................. Error! Bookmark not defined.
3.6.2 Idle Connection Timeout: .................................................................. Error! Bookmark not defined.
3.6.3 Tunneling Client Timeout................................................................... Error! Bookmark not defined.
3.6.4 Post Timeout ...................................................................................... Error! Bookmark not defined.
3.6.5 Duration ............................................................................................. Error! Bookmark not defined.
3.6.6 Tunneling Client Timeout................................................................... Error! Bookmark not defined.
4 BPEL PM AUDIT AND LOGGING ................................................................ ERROR! BOOKMARK NOT DEFINED.
4.1 AUDIT TUNING .......................................................................................... ERROR! BOOKMARK NOT DEFINED.
4.1.1 SOA Infrastructure level ..................................................................... Error! Bookmark not defined.
4.1.2 BPEL Engine Level .............................................................................. Error! Bookmark not defined.
4.1.3 AuditStorePolicy ................................................................................ Error! Bookmark not defined.
4.1.4 AuditFlushByteThreshold................................................................... Error! Bookmark not defined.
4.1.5 AuditFlushEventThreshold ................................................................. Error! Bookmark not defined.
4.1.6 AsyncAuditPersisterThreads .............................................................. Error! Bookmark not defined.
4.1.7 AsyncAuditQueueSize ........................................................................ Error! Bookmark not defined.
4.2 LOGGING TUNING...................................................................................... ERROR! BOOKMARK NOT DEFINED.
5 BPEL ENGINE DATABASE SCHEMA ............................................................. ERROR! BOOKMARK NOT DEFINED.
5.1 TWO SCHEMAS ......................................................................................... ERROR! BOOKMARK NOT DEFINED.
5.2 PURGE ..................................................................................................... ERROR! BOOKMARK NOT DEFINED.
10.4.1 Enable SecureFiles for SOAINFRA schema ....................................... Error! Bookmark not defined.
10.4.2 Enabling Encryption ......................................................................... Error! Bookmark not defined.
10.5 SQL PLAN .............................................................................................. ERROR! BOOKMARK NOT DEFINED.
11 REFERENCE......................................................................................... ERROR! BOOKMARK NOT DEFINED.
EM FUSION MIDDLEWARE CONTROL AND WLS ADMIN CONSOLE..... ERROR! BOOKMARK NOT DEFINED.
1
2
3
4
5
6
7
8
9
10
11
12
3.4 ANALYSIS OF SOA SERVER MEMORY USAGE ................................................... ERROR! BOOKMARK NOT DEFINED.
4 REFERENCE .......................................................................................... ERROR! BOOKMARK NOT DEFINED.
LNP TEST CASE DESIGN AND LOAD TEST SELECTION ......................... ERROR! BOOKMARK NOT DEFINED.
1 LNP TEST CASE DESIGN ........................................................................... ERROR! BOOKMARK NOT DEFINED.
2 DESIGN OF EXPERIMENT .......................................................................... ERROR! BOOKMARK NOT DEFINED.
3 REFERENCE .......................................................................................... ERROR! BOOKMARK NOT DEFINED.
APPENDIX A: WEBLOGIC SERVER OVERVIEW ................................... ERROR! BOOKMARK NOT DEFINED.
1
2
3
4
5
6
7
8
9
APPENDIX F: ORA-01438: VALUE LARGER THAN SPECIFIED PRECISION ALLOWED .... ERROR! BOOKMARK
NOT DEFINED.
5 WHAT IS THE ERROR IN LOGS? .................................................................. ERROR! BOOKMARK NOT DEFINED.
APPENDIX K: HEAP DUMP FILES ANALYSIS: JROCKIT AND HOTSPOT JVMS ....... ERROR! BOOKMARK NOT
DEFINED.
1 EXAMPLE ANALYSIS OF A HEAP DUMP FILE USING ECLIPSE MEMORY ANALYZER ... ERROR! BOOKMARK NOT DEFINED.
2 REFERENCE .......................................................................................... ERROR! BOOKMARK NOT DEFINED.
APPENDIX L: CAPACITY PLANNING................................................... ERROR! BOOKMARK NOT DEFINED.
1 CAPACITY PLANNING FOR BPEL PM .......................................................... ERROR! BOOKMARK NOT DEFINED.
1.1 DETERMINING PERFORMANCE GOALS AND OBJECTIVES CURRENT & FUTURE.... ERROR! BOOKMARK NOT DEFINED.
1.2 MEASURING PERFORMANCE METRICS .......................................................... ERROR! BOOKMARK NOT DEFINED.
1.3 IDENTIFYING BOTTLENECKS ......................................................................... ERROR! BOOKMARK NOT DEFINED.
1.4 IMPLEMENTING A CAPACITY MANAGEMENT PLAN .......................................... ERROR! BOOKMARK NOT DEFINED.
2 REFERENCE .......................................................................................... ERROR! BOOKMARK NOT DEFINED.
Exhibits
Exhibit 1: Stress, Load, Soak, Spike Tests Load Profiles ................... Error! Bookmark not defined.
Exhibit 2: LnP Testing Life Cycle and SDLC ....................................... Error! Bookmark not defined.
Exhibit 3: BPEL Processes ................................................................. Error! Bookmark not defined.
Exhibit 4: JVM Tuning Tradeoffs ................................................................................................... 16
Exhibit 5: JVM's Garbage Collection and Throughput .................................................................. 17
Exhibit 6: jRockit Attributes .......................................................................................................... 23
Exhibit 7: HostSpot Attributes ...................................................................................................... 28
Exhibit 8: WLS internal applications at startup - 1 ......................... Error! Bookmark not defined.
Exhibit 9: WLS internal applications at startup - 2 ......................... Error! Bookmark not defined.
Exhibit 10: WLS internal applications at startup - 3 ....................... Error! Bookmark not defined.
Exhibit 11: WLS internal applications at startup - 4 ........................ Error! Bookmark not defined.
Exhibit 12: WLS internal applications at startup - 5 ........................ Error! Bookmark not defined.
Exhibit 13: WLS internal applications at startup - 6 ........................ Error! Bookmark not defined.
Exhibit 14: SOAWorkManager - 1 .................................................... Error! Bookmark not defined.
Exhibit 15: SOAWorkManager - 2 .................................................... Error! Bookmark not defined.
Exhibit 16: SOAWorkManager - 3 .................................................... Error! Bookmark not defined.
Exhibit 17: WLS Muxers ................................................................... Error! Bookmark not defined.
Exhibit 18: FMC Login....................................................................... Error! Bookmark not defined.
Exhibit 19: FMC Farm Summary....................................................... Error! Bookmark not defined.
Exhibit 20: FMC Performance Summary - 1 ..................................... Error! Bookmark not defined.
Exhibit 21: FMC Performance Summary - 2 ..................................... Error! Bookmark not defined.
Exhibit 22: FMC Performance Summary - 3 ..................................... Error! Bookmark not defined.
Exhibit 23: Dynamic Memory Service .............................................. Error! Bookmark not defined.
Exhibit 24: BPEL Engine Threading Model ....................................... Error! Bookmark not defined.
Exhibit 25: Dispatcher Maximum Request Depth - 1....................... Error! Bookmark not defined.
Exhibit 26: Dispatcher Maximum Request Depth - 2....................... Error! Bookmark not defined.
Exhibit 27: InvokeMessages In Cache .............................................. Error! Bookmark not defined.
Exhibit 28: syncMaxWaitTime - 1 .................................................... Error! Bookmark not defined.
Exhibit 29: syncMaxWaitTime - 2 .................................................... Error! Bookmark not defined.
Exhibit 30: syncMaxWaitTime - 3 .................................................... Error! Bookmark not defined.
Exhibit 31: syncMaxWaitTime 4 .................................................... Error! Bookmark not defined.
Exhibit 32: BPEL EJBs - 1 ................................................................... Error! Bookmark not defined.
Exhibit 33: BPEL EJBs - 2 ................................................................... Error! Bookmark not defined.
10
12
Snippets
Snippet 1: Sample Work Manager - 1 .............................................. Error! Bookmark not defined.
Snippet 2: Sample Work Manager - 2 .............................................. Error! Bookmark not defined.
Snippet 3: Sample config.xml........................................................... Error! Bookmark not defined.
Snippet 4: Sample startWeblogic.sh ................................................ Error! Bookmark not defined.
Snippet 5: Using setNull In a Prepared Statement .......................... Error! Bookmark not defined.
Snippet 6: Sample thread waiting for DB Call to finish.................... Error! Bookmark not defined.
Snippet 7: composite.xml ................................................................ Error! Bookmark not defined.
Snippet 8: Drop Histrogram ............................................................. Error! Bookmark not defined.
Snippet 9: Set Table Preferences ..................................................... Error! Bookmark not defined.
Snippet 10: Single Threaded Model - jca file ................................... Error! Bookmark not defined.
Snippet 11: Partitioned Threaded Model - jca file........................... Error! Bookmark not defined.
Snippet 12: Tuning number of files to be processed at a time ....... Error! Bookmark not defined.
Snippet 13: JMS Adapter - Receive Thread - composite.xml ........... Error! Bookmark not defined.
Snippet 14: AQ Adapter - Dequeue Threads - composite.xml......... Error! Bookmark not defined.
Snippet 15: common properties schema - 1.................................... Error! Bookmark not defined.
Snippet 16: common properties schema - 2.................................... Error! Bookmark not defined.
Snippet 17: bpel.xml ........................................................................ Error! Bookmark not defined.
Snippet 18: composite.xml .............................................................. Error! Bookmark not defined.
Snippet 19: SecureFiles .................................................................... Error! Bookmark not defined.
Snippet 20: BasicFiles to SecureFiles ............................................... Error! Bookmark not defined.
13
Tables
Table 1: AuditLevel........................................................................... Error! Bookmark not defined.
Table 2: BPEL Engine - AuditLevel .................................................... Error! Bookmark not defined.
Table 3: AuditStorePolicy ................................................................. Error! Bookmark not defined.
Table 4: SOA Schemas ...................................................................... Error! Bookmark not defined.
Table 5: SOAINFRA Tables ................................................................ Error! Bookmark not defined.
Table 6: MDS Schema tables............................................................ Error! Bookmark not defined.
Table 7: CUBE_INSTANCE................................................................. Error! Bookmark not defined.
Table 8: CUBE_SCOPE ...................................................................... Error! Bookmark not defined.
Table 9: COMPOSITE_INSTANCE ...................................................... Error! Bookmark not defined.
Table 10: COMPOSITE_INSTANCE_FAULT ....................................... Error! Bookmark not defined.
Table 11: AUDIT_TRAIL .................................................................... Error! Bookmark not defined.
Table 12: AUDIT_DETAILS ................................................................ Error! Bookmark not defined.
Table 13: DLV_MESSAGE ................................................................. Error! Bookmark not defined.
Table 14: DLV_SUBSCRIPTION ......................................................... Error! Bookmark not defined.
Table 15: DOCUMENT_CI_REF ......................................................... Error! Bookmark not defined.
Table 16: DOCUMENT_DLV_MSG_REF ............................................ Error! Bookmark not defined.
Table 17: WORK_ITEM ..................................................................... Error! Bookmark not defined.
Table 18: XML_DOCUMENT ............................................................. Error! Bookmark not defined.
Table 19: XML_DOCUMENT_REF ..................................................... Error! Bookmark not defined.
Table 20: HEADERS_PROPERTIES ..................................................... Error! Bookmark not defined.
Table 21: CLUSTER_MASTER ............................................................ Error! Bookmark not defined.
Table 22: CLUSTER_NODE ................................................................ Error! Bookmark not defined.
Table 23: Partial Partitioning ........................................................... Error! Bookmark not defined.
Table 24: Complete Partitioning ...................................................... Error! Bookmark not defined.
Table 25: MDS_NAMESAPCES.......................................................... Error! Bookmark not defined.
Table 26: MDS_PARTITIONS ............................................................ Error! Bookmark not defined.
Table 27: MDS_PATHS ..................................................................... Error! Bookmark not defined.
Table 28: File & FTP Adapter - Inbound Throttling .......................... Error! Bookmark not defined.
Table 29: : File & FTP Adapter - Outbound Throttling ..................... Error! Bookmark not defined.
Table 30: : File & FTP Adapter - Outbound Best Practices............... Error! Bookmark not defined.
Table 31: completionPersistPolicy property .................................... Error! Bookmark not defined.
Table 32: validateSchema value ...................................................... Error! Bookmark not defined.
Table 33: Idempotency value........................................................... Error! Bookmark not defined.
Table 34: nonBlockingInvoke value ................................................ Error! Bookmark not defined.
Table 35: Database initialization parameters .................................. Error! Bookmark not defined.
Table 36: Contention tables ............................................................. Error! Bookmark not defined.
Table 37: Tables under potential of index contention .................... Error! Bookmark not defined.
14
JVM Tuning
SOA Suite 11g ideally uses two JVMs to function.
Admin Server JVM: On this JV, Weblogic Admin Console and the EM Fusion Middleware
Control are deployed. Though, this JVM does not affect performance of BPEL PM but if this JVM
is leading to reduced performance of WebLogic Admin Console and EM Fusion Middleware
Control, it leads to irritation and frustration during monitoring and deployment.
SOA Managed Server: On this JVM entire SOA Suite and B2B product stack are deployed. This
JVM is critical for overall performance of BPEL PM.
The JVM tuning depends upon number of environmental factors and objectives of tuning. Since
each deployment of BPEL PM is unique, so no sure shot formula can be derived but generally
speaking following parameters should be considered while tuning JVM:
Tuning the JVMs should not be speculative but a measured and calibrated. One should rely on
Memory and GC analyzing tools like jConsole, visualVM, Flight Control etc. in conjunction with
LnP Testing activities. Finer tuning should take help of thread dump and GC activities.
JVM tuning leads to trade-offs among garbage collection pause time, application throughput,
and memory footprint.
From JVM metrics purpose:
GC pause time is defined as typical duration for which application threads are paused when GC
is happening. During pause time, application seems to be unresponsive.
Throughput is measured in terms of time spent doing garbage collection vs. the time spent
outside of garbage collection, considered over long periods of time.
Memory footprints refer to amount of memory space utilized by JVM.
15
16
3 JVM Choice
When it comes to JVM, there are two popular choices:
17
1. jRockit
2. HotSpot
Other than Solaris, jRokit is better option to run WLS. I do not have any data to prove this bias
but my experience and conversation with peers.
On the basis of prefix JVM option it can be divided into three parts, JVM Options which starts
with X, -XX and others.
1. JVM Options that begin with -X are non-standard. These are not guaranteed to be
supported on all JVM implementations and subject to change without notice in future
releases of the Java.
2. JVM Options that begin with -XX are not stable and are not recommended for casual use.
These options are subject to change without notice.
3. Apart from above mentioned there are several other options which are stable and
standard across different implementations of JVM.
Fusion Middleware SOA Suite (BPEL Engine) is multithreaded application which can host both
long running processes and short one. Depending upon the processes hosted in a BPEL Engine,
one need to tune JVM.
jRockit is preferred option to run WLS on all platforms other than Solaris on
which HotSpot is first choice.
4 jRockits Tuning
4.1 Boarding Into ship Step 1
In this step we need to understand how to set Heap size, choosing Garbage Collection strategy,
Nursery Size, and Garbage Collection pauses and their implications on performance.
-Xms:<size>[g|G|m|M|k|K]
-Xmx:<size>[g|G|m|M|k|K]
Where:
g
or G Gigabyte (GB)
or M Megabyte (MB)
or K Kilobyte (KB)
For example a managed server running on a machine with 8 GB RAM memory could be started
with the following settings:
java -Xms:8000m -Xmx:8000m MyManagedServer
This starts the JVM with a heap of 8000 MB and allows the heap to grow up to 8000 MB.
4.1.2 Garbage Collection
Garbage collection uses system resources like any other process. By tuning the garbage
collection one can decide how and when these resources will be used. JRockit JVM has three
garbage collection strategies. The default strategy optimizes application throughput, second
one optimizes pauses during garbage collection, and third one focuses on very short and
deterministic garbage collection pauses.
Garbage collection is optimized for application
throughput. This is the default garbage collection strategy.
-XgcPrio:throughput
-XgcPrio:pausetime
collection pauses.
JVM on which managed server (on which BPEL engine is running) is running
should choose garbage collection strategy which optimizes application throughput.
If BPEL engine is hosting short running synchronous processes only and load on
server is constant (not varying within short interval of time), deterministic
strategy might be better option.
19
-XpauseTarget:<time>
A higher pause target usually turns into higher application throughput, so one
should set the pause target as high as BPEL Engine and hence deployed
processes/composites can tolerate.
For example a BPEL engine where transactions normally take 50 ms and time out after 350
ms could be started with the following settings:
java -XgcPrio:deterministic -XpauseTarget:150 MyManagedServer
These options will start up the JVM with garbage collection optimized for very short pauses
with a pause target of 150 ms. This leaves a 200 ms margin before time-out for 350 ms
transactions that are interrupted by a 150 ms garbage collection pause.
20
-XXlazyUnlocking
On all platforms except IA64, lazy unlocking is enabled by default for R27.5 for all garbage
collection strategies except for deterministic.
This option is intended for applications with many unshared locks. Since BPEL
Engine and deployed processes/composites shares short lived locks, specifying
this option may hit performance.
-XXcallProfiling
Call Profiling option may increase performance because JVM can act on those
statistics.
-XlargePages
By default the JVM will continue to run without large pages if large pages cannot be acquired
when -XlargePages is enabled.
-XlargePages:exitOnFailure=true
Use extended option to override default behavior and force the JVM to exit if enough large
pages cant be acquired. This extended option is supported in JRockit JVM R27.5 and later
releases.
This option tells the JRockit to use large pages, for the Java heap. Large pages
allow application to more effectively use the translation look-aside buffer (TLB) in
the processor, if available
4.3.1 Compaction
Compaction of objects is the process of moving objects closer to each other in the heap, thus
reducing the fragmentation and making object allocation easier for the JVM. JRockit JVM
compacts a part of the heap at each garbage collection. Compaction may leads to long garbage
collection pauses.
4.3.1.1 Compaction Ratio
The compaction ratio determines what percent of the heap will be compacted during
garbage collection.
-XXcompactRatio:<percentage>
As a start, one can try lowering the compaction ratio to 1 and measure the pause. If
pause is within tolerable limit, gradually increase compaction ratio till it start affecting
BPEL engine and hence deployed processes/composites.
A good value for the compact ratio is usually between 1 and 20, sometimes even
higher. Compaction ratio must be in tune with compact set limit.
Low value of compaction ratio may increase the fragmentation and the amount of
free space that is too small to be used for object allocation.
-XXcompactSetLimit:<references>
As a start, one can try setting the compact set limit at 10,000 and measure the pause. If
pause is within tolerable limit, gradually increase the compact set limit as long as the
compaction times stay low.
A normal value for the compact set limit is usually between 100,000 and several
million, while lower values are used when the pause time limits are very low.
Setting the compact set limit too low may stop compaction from being done
altogether. Running without any compaction at all may lead to increasing
fragmentation, which will in the end force the JVM to perform a full compaction of
the whole heap at once, which may take several seconds.
-XXcompactSetLimit has no effect when -XgcPrio:deterministic
XgcPrio:pausetime is used.
or -
synchronizing with other threads. When the TLA gets full the thread simply requests a new TLA.
The objects allocated in a TLA are accessible to all Java threads and are not considered thread
local in any way after they have been allocated.
-XXtlaSize:min=<size>,preferred=<size>
The min value is the minimum TLA size, while the preferred value is a preferred size. This
means that TLAs will be of the preferred size whenever possible, but may be as small as the
min size.
Typically the preferred TLA size can be up to twice the size of the largest
commonly used object size in the application. Adjusting the min size may have
an effect on garbage collection performance, but is seldom necessary. A normal
value for the min size is 2 KB.
Increasing the TLA size is beneficial for BPEL Engine and hence deployed
processes/composites (since BPEL engine is a multi-threaded applications and
each thread allocates a lot of objects and size of objects is large). Increasing the
TLA size too much may however cause more fragmentation and more frequent
garbage collections.
5 HotSpots Tuning
HotSpot claims that it is ergonomic. Based upon the platform configuration, it will select a
compiler, Java heap configuration, and garbage collector for best results. But still human
intervention in terms of its tuning is required because of staggering number of variations
possible.
HotSpot tuning focusses around Garbage Collection strategies and mode for server and client.
This book will be focusing on server class machines only. Keeping this assumption, JVM startup
command will contain following options in all scenarios:
-server:
23
-XX:+UseSerialGC
-XX:+UseParallelGC
HotSpot JVM also has one more parallel GC strategy which also runs compaction in parallel to
reduce fragmentation of Heap.
-XX:+UseParallelOldGC
Without parallel compaction, major collections are performed using single thread,
which can limit scalability.
Parallel GC strategy (with or without compaction) is designed for small to medium
scale applications.
The concurrent GC strategy, as name suggests does most of its work while
application is running, so minimizes pause time. This strategy focuses on response
time rather than over all throughput.
-XX:+UseConcMarkSweepGC
Concurrent collections can be performed incrementally as well. During a concurrent phase the
garbage collector thread use one or more processors. The incremental option decreases the
impact of long concurrent collections by periodically stopping the concurrent collection to yield
back the processor to the application. This mode divides the work done concurrently by the
collector into small chunks of time which are scheduled between young generation collections.
This feature is useful when applications require small pause times and machine
has small numbers of processors (e.g., 1 or 2).
24
-XX:+CMSIncrementalMode
To collect statistics and analyze, one should supply following option while starting HotSpot JVM.
-verbose:gc
-XX:+PrintGCDetails
PrintGCDetails option prints amount of young generation, heap recovered and time spent in
doing so.
25
It indicates that the minor collection recovered about 98.6% ((64289-875)/64289) of the young
generation and took 0.0397641 seconds.
The usage of the entire heap was reduced to about 32% ((196554-875)/196554) and that there
was some slight additional overhead for the collection (over and above the collection of the
young generation) as indicated by the final time of 0.0399680 seconds
-XX:+PrintGCTimeStamps
PrintGCTimeStamps
The collection starts about 99.032 seconds into the execution of the application. The minor
collection starts at about the same time.
It indicates that the minor collection does not recovered anything ((8136-8136)/ 8136) of the
young generation and took 0.0000210 seconds.
The tenured generation usage was reduced to about 14% (100- ((18264-2524)/ 18264)) and
took 0. 899311 seconds.
The usage of the entire heap was reduced to about 6% (100- ((45289-2571)/45289)) and that
there was some slight additional overhead for the collection (over and above the collection of
the young generation) as indicated by the final time of 0.1294000 seconds
Like jRockit, HotSpot also has mechanism to define Heap size.
There are two parameters to set Heap Size.
-Xms:<size>[g|G|m|M|k|K]
-Xmx:<size>[g|G|m|M|k|K]
Where:
g
or G Gigabyte (GB)
26
or M Megabyte (MB)
or K Kilobyte (KB)
One should set the minimum and maximum heap size as same value to avoid
unnecessary JVM effort to manage fluctuating heap size.
For example a managed server running on a machine with 8 GB RAM memory could be started
with the following settings:
java -Xms:8000m -Xmx:8000m MyManagedServer
This starts the JVM with a heap of 8000 MB and allows the heap to grow up to 8000 MB.
Garbage Collection (minor as well as major collections) occurs when generations (young or
tenured portion of heap) fill up, throughput is inversely proportional to the amount of memory
available. Total available memory is the most important factor affecting garbage collection
performance.
5.1.3 Parallel GC strategy
JVM sets priority of targets at execution time.
Priority 1: Maximum pause time target
Priority 2: Throughput target
Priority 3: Minimum footprint target
On a machine with P processors, parallel collector uses P garbage collector threads; but, this
number can be adjusted with following option
XX:ParallelGCThreads=<N>
In parallel collector minor collection pauses are shorter. Because there are multiple garbage
collector threads participating in the minor collection there is a small possibility of
fragmentation due to promotions from the young generation to the tenured generation during
the collection. Each garbage collection thread reserves a part of the tenured generation for
promotions and the division of the available space into these "promotion buffers" can cause a
fragmentation effect. Reducing the number of garbage collector threads will reduce
this fragmentation effect as will increasing the size of the tenured generation.
-XX:MaxGCPauseMillis=<N>
This is a hint to HotSpot JVM that pause times of <N> milliseconds or less are desired. By
default there is no maximum pause time target. If a pause time target is specified,
the heap size and other garbage collection related parameters are adjusted in an
27
attempt to keep garbage collection pauses shorter than the specified value. These
adjustments may cause the garbage collector to reduce the overall throughput of
the application.
-XX:GCTimeRatio=<N>
For example, -XX:GCTimeRatio=29 sets a goal of 1/30 or 3.33% of the total time in garbage
collection. Default value on Solaris is 99.
Maximum heap footprint is specified using the
-Xmx:<size>
The collector has an implicit goal of minimizing the size of the heap as long as
the other goals are being met.
If more than 98% of the total time is spent in garbage collection and less than 2% of the heap is
recovered, an OutOfMemoryError is thrown. This feature prevents applications from running
for an extended period of time while making little or no progress because the heap is too small.
HotSpot provides option to suppress this error using
-XX:-UseGCOverheadLimit
28
For BPEL engines managed server parallel GC is sufficient, so will not discuss the details of
concurrent GC strategy.
7 OutOfMemoryError
Out of memory is one of the most common error for an JVM. At any time the JVM may run out
of memory, which effectively crashes the server and requires a restart. There are many causes
for out of memory exceptions. From too small of a heap defined, memory leak, poor quality
code, not enough Perm Gen space defined, poor garbage collection settings, not enough
physical memory being available on the machine where the JVM runs, etc.
Most common reasons of out of memory error are following:
The PermSpace is normally not a target of tuning efforts, usually more space must be allocated
in the -XX:PermSize/-XX:MaxPermSize parameters.
There are a few exceptions though:
Applications that do dynamic class loading and unloading might suffer from a too small
PermSpace
A PermSpace can trigger unnecessary Full GC activity with the Concurrent-Mark-Sweep
(CMS) collector
An oversized PermSpace unnecessarily increases memory footprint which in turn
increases the likelihood to hit the 2 or 4 g process size limit in 32 bit mode
Since process space is limited, running a demanding application under high load often involves finding
an appropriate trade-off between native heap availability, Java heap availability, and threads. Reducing
the Java heaps (-Xmx or -XX:MaxPermSize) will make more native heap available, but at the expense of
reducing the number of Java objects or classes that can be accommodated. Reducing the stack size (-Xss)
will also reduce the load on the native heap, but will reduce the available call stack depth, which might
cause heavily recursive code to fail. Reducing the number of threads is usually only possible by reducing
the load on the application.
If the failure is not the result of a memory leak, and the application cannot be further optimized to
reduce memory needs, then one might consider a move to a 64-bit JVM. However, this will inevitably
use more memory than the same application on a 32-bit JVM, and the system memory must be sized
appropriately.
7.5 Analysis
In case of java.lang.OutOfMemoryError log files are generally unreliable and not of much use.
One should try to understand, what is happening on the system at the time of error. From a
SOA suite perspective one should look for answers of following questions:
Is the BPEL Audit Trail level set at Development causing additional overhead in the JVM
when each Assign is being written to the audit trail? This should always be set to
Production level or less on systems under load.
Are there large payload files being processed at the time of the error?
Is the composite set to do XML Validation which will add processing overhead?
31
Analysis of the composites, are they looping, are there a lot of synchronous BPEL
processes running for long periods of time each consuming memory or any other
application misuse of memory?
One should collect the heap usage and garbage collection statistics for analysis of
java.lang.OutOfMemoryError. A memory leak is evident if the used heap goes up at an ever
increasing rate with garbage collections having little or no impact on decreasing the used heap..
In both the JRockit and HotSpot JDKs one can pass parameters to dump the heap to a binary
heap dump file at the time of out of memory:
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/oracle
Note that the heap dump file will be a little larger than the maximum defined heap size so make
sure that the defined HeapDumpPath has enough disk space to accommodate the file(s).
On JRockit one can also produce an out of memory diagnostic file:
-Djrockit.oomdiagnostics=true -Djrockit.oomdiagnostics.filename=/home/oracle
This is a text file that will give additional information about what is happening with the
memory.
JRockit also has these:
-XX:+|-HeapDiagnosticsOnOutOfMemoryError
-XX:HeapDiagnosticsPath
-XX:HeapDumpSegmentSize
-XX:+|-HeapDumpOnCtrlBreak
-XX:HeapDumpSegmentSize
Apply -XX:+HeapDumpOnCtrlBreak to the start parameters of the JVM and use CTRL-BREAK to
cause a heap dump, or in JRockit Mission Control.
This will create a jrockit_<pid>.hprof heapdump file in the defined directory.
8 Reference
1. Comprehensive list of documents for jRockit:
http://docs.oracle.com/cd/E13150_01/jrockit_jvm/jrockit/webdocs/index.html
2. Understanding of jRockit:
http://docs.oracle.com/cd/E13150_01/jrockit_jvm/jrockit/geninfo/diagnos/part_01.html
32
33