Sunteți pe pagina 1din 56

2006 BEA Systems, Inc.

Monitoring, Tuning, and

Troubleshooting Performance
At the end of this module, you will be able to:
Optimize IPC resources for Tuxedo application
Configure system and server options for improved
Troubleshoot applications with runtime tracing
Use the Tuxedo Administration Console
Module 15
Monitoring, Tuning, and Troubleshooting Performance-1
2006 BEA Systems, Inc. 456
Road Map
1. IPC Resource Requirements
Understanding Shared Memory and Semaphores
Message and Message Queue Sizing
Performance Monitoring
Tracing ATMI Code Execution
2. Runtime and Configuration Performance Options
3. The BEA Tuxedo Administration Console

Monitoring, Tuning, and Troubleshooting Performance-2
2006 BEA Systems, Inc. 457
Reconfiguring Kernel Parameters
Most UNIX Operating Systems will require tuning.
Refer to the documentation (Tuxedo Platform Data
Sheets) for specific information for your operating
Below is an excerpt from the Sun Microsystems Solaris 9
(32-bit) data sheet :
1024 30 SEMMNU semsys:seminfo_semmnu
2000 25 SEMMSL semsys:seminfo_semmsl
5029 10 SEMMNI semsys:seminfo_semmni
5048 60 SEMMNS semsys:seminfo_semmns
300 100 SHMMNI shmsys:shminfo_shmmni
100 6 SHMSEG shmsys:shminfo_shmseg
67108864 131072 SHMMAX shmsys:shminfo_shmmax
Solaris Name
Monitoring, Tuning, and Troubleshooting Performance-3
2006 BEA Systems, Inc. 458
Determining IPC Requirements
You can determine more specific IPC requirements for
your application:
Use tmloadcf c or tmboot c
Determining IPC Requirements for a Specific Tuxedo Application:
C:\> tmboot -c
Ipc sizing (minimum /T values only) ...
Fixed Minimums Per Processor
Variable Minimums Per Processor
------ ------ ------ ------ ------ ------ ------ ------
MYMACHINE 100 12 95 A + 1 77 154 519K

where 1 <= A <= 8.
Monitoring, Tuning, and Troubleshooting Performance-4
2006 BEA Systems, Inc. 459
Windows IPC Resources Page
The BEAProcMGR service sets resources to values
defined in the IPC Resources Page.
The default
values are
shown here.
Monitoring, Tuning, and Troubleshooting Performance-5
2006 BEA Systems, Inc. 460
Tuning Message and Queue Lengths
When Queues Are Too Long:
Can have a negative impact allowing a backlog of requests to
accrue affecting performance
If a need for large queues is perceived, something else is
probably wrong
When Queues Are Too Short:
Asynchronous calls will suffer performance slow-downs
Larger than average load can cause uneven response times
Large individual requests can cause spooling to disk:
Tuxedo uses file transfer if MSGMAX or 75% of MSGMNB is exceeded
Can be advantageous to prevent very rare
large requests from filling queues.
MSGMAX: Maximum Message Size
MSGMNB: Maximum Message Queue
Monitoring, Tuning, and Troubleshooting Performance-6
2006 BEA Systems, Inc. 461
Tunable Parameters for IBM AIX
The latest supported version is AIX 4.53 (32-bit)
No IPC configuration is required for AIX.
To change the value of the maxuproc parameter:
1. Acquire superuser privileges.
2. Determine the recommended value
3. Change the value of the parameter.
4. Reboot the system.

Monitoring, Tuning, and Troubleshooting Performance-7
2006 BEA Systems, Inc. 462
System-Specific Diagnostic Tools
Some tools are specific to the system platform and are
provided by the operating system vendor.
Use UNIX System utilities such as:
top (Displays system and process information)
sar (System Activity Report)
In Windows:
Event Viewer
Task Manager
Performance Monitor

Monitoring, Tuning, and Troubleshooting Performance-8
2006 BEA Systems, Inc. 463
Windows Task Manager
Monitoring, Tuning, and Troubleshooting Performance-9
2006 BEA Systems, Inc. 464
Unix Monitoring Tools
Most Unix systems provide tools for monitoring
system performance, such as:
sar: system activity (Solaris)
mpstat: per-processor statistics (Solaris)
vmstat: virtual memory statistics
netstat: network statistics
iostat: I/O statistics
Monitoring, Tuning, and Troubleshooting Performance-10
2006 BEA Systems, Inc. 465
Example vmstat Output
Example of vmstat output:

$ vmstat 1
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
2 0 0 120 30180 1736 46696 0 0 424 25 134 77 5 9 86
3 0 0 120 13908 1776 63192 0 0 8220 0 268 275 15 41 44
0 1 1 120 5012 1784 71668 0 0 4344 5196 452 337 9 32 59
0 1 1 120 4756 1784 71796 0 0 128 7824 294 94 0 6 94
2 0 1 120 3200 1788 72436 0 0 1028 3364 266 91 2 6 92
0 2 0 140 3628 1800 70796 0 0 5368 0 216 286 41 30 30
1 0 0 140 23144 1856 51904 0 0 1968 28 339 500 49 16 36
4 1 0 140 20324 1968 53188 0 0 2152 0 210 368 72 7 21
6 0 0 140 15940 2020 54992 0 0 1796 0 413 315 85 10 5
6 3 0 140 12096 2112 56732 0 0 1880 0 468 478 80 19 1
10 0 0 140 8660 2152 58736 0 0 2040 0 232 448 81 19 0
5 3 0 140 4556 2184 61052 0 0 2340 0 482 1198 35 32 34

Monitoring, Tuning, and Troubleshooting Performance-11
2006 BEA Systems, Inc. 466
A general tuning methodology:
1. State goals and define the system.
2. List services and outcomes.
3. Select metrics.
4. List parameters.
5. Select factors to study.
6. Select an evaluation technique.
7. Select workload.
8. Design experiments.
9. Analyze and interpret data.
10. Present results.
11. Repeat.

Jain, Raj. The Art of Computer Systems Performance Analysis. New York, NY: John
Wiley& Sons, April 1991.
Monitoring, Tuning, and Troubleshooting Performance-12
2006 BEA Systems, Inc. 467
Methodology Tips
Perform benchmarks on:
Functional areas (subsystems) of the application
The full (integrated) application
Test performance early and often, but defer tuning
until forced to do so.
Do the following to maintain test integrity:
Always generate load on separate machines from those
hosting server instances
Minimize the amount of data collection software running on
the machines hosting server instances
Account for the impact of both
Monitoring, Tuning, and Troubleshooting Performance-13
2006 BEA Systems, Inc. 468
Tracing ATMI calls
Tuxedo provides the facility to trace ATMI library code
execution within client and server processes.
Allows the application developer (or administrator) to
trace the flow of execution within a Tuxedo application.
The default ulog receiver outputs trace data:
to include entry and exit time stamps.
written by to the ULOG file.
The utrace receiver outputs trace record information:
by calling a user-defined tputrace() library routine.
Users can customize trace record information and output location.
Version 9.0 + only
Monitoring, Tuning, and Troubleshooting Performance-14
2006 BEA Systems, Inc. 469
Using the tmtrace Facility
tmtrace Environment Variable
To trace a client and all ATMI calls made: TMTRACE=on
To see all client and server trace points,
both explicit and implicit:
To trace service requests from a client
but restrict the tracing output from the
client to the bare minimum of information
about tpcall(3c) requests:
To trace the invocations of all service
requests performed by application
changetrace tmadmin sub-command
To change the trace specification of a
specific currently-running server:
changetrace [-g groupname]
[-i srvid] newspec
Monitoring, Tuning, and Troubleshooting Performance-15
2006 BEA Systems, Inc. 470
Section Review
Determine the IPC resource requirements for a Tuxedo
Allow for optimum message queue sizing
Trace ATMI code execution within a running
In this section, we learned how to:
Monitoring, Tuning, and Troubleshooting Performance-16
2006 BEA Systems, Inc. 471
Road Map
1. IPC Resource Requirements
2. Runtime and Configuration Performance
Message Size Compression Thresholds
System and Server Tuning Options
Bundling Services in Servers
Measuring Service Execution Times
Multiple-Server-Single-Queue (MSSQ) Sets
3. The BEA Tuxedo Administration Console

Monitoring, Tuning, and Troubleshooting Performance-17
2006 BEA Systems, Inc. 472
Protecting Shared Memory
In the development cycle, programmer code can
inadvertently corrupt shared memory.
You can protect system tables in shared memory with
The SERVERS section also provides an override setting.
selected option cannot be changed
libraries will not attach
libraries attach at all times (default)
Description Option
Protecting Shared Memory:
Monitoring, Tuning, and Troubleshooting Performance-18
2006 BEA Systems, Inc. 473
Specified in MACHINES section, SPINCOUNT is the
number of attempts made to the BB before sleeping.
Default is 0, which allows each process to use value built into
the binary or use the TMSPINCOUNT environment variable.
Always set SPINCOUNT to 1 for a single CPU machine.
Experiment with a SPINCOUNT between 5,000 and
50,000 for multiple CPU machines.
Tune by trial, while watching a busy server with semOPS
Setting SPINCOUNT per Machine Type:
gumby # a 4-CPU Solaris machine
PLUTO # a 1-CPU Windows machine
Monitoring, Tuning, and Troubleshooting Performance-19
2006 BEA Systems, Inc. 474
Service Lookup Caching
This feature is turned on by default; a performance
improvement over previous versions of Tuxedo.
Saves some BB overhead by caching service (and interface)
entries within the calling process.
Configuration parameter is SICACHEENTRIESMAX
Used in the MACHINES or SERVERS section.
Set value to 0 to turn off or change to an appropriate value
for your application. Default: 500
This feature can also be set/reset per process with the
TMSICACHEENTRIESMAX environment variable.
Versions 8.0 + only
Changing the number of cached services:
Monitoring, Tuning, and Troubleshooting Performance-20
2006 BEA Systems, Inc. 475
Message Compression Thresholds
Remote compression should be based on network
speed and congestion:
Low-speed network compress all messages.
Workstation clients use the compression threshold specified
by the WSL CLOPT -c option.
Local compression is used to prevent disk spooling.
Compress large messages if message queue size is an issue.
Using CMPLIMIT parameter in MACHINES section:

Compressing both remote and local messages:
Compress all internal messages larger than 2k
Compress all remote messages larger than 8k
Monitoring, Tuning, and Troubleshooting Performance-21
2006 BEA Systems, Inc. 476
OPTIONS Parameter Values
NO_AA : turns off the extended security plug-in calls:
Username/password authentication or ACL authorization
security is not affected.
NO_XA : turns off XA transactions for the entire
application domain.
Use only if you are not using distributed (XA) transactions or
/Q queuing features.
It is also recommended to set the MAXGTT parameter to 0 in
the RESOURCES section.
Versions 8.0 + only
Protecting shared memory:
Monitoring, Tuning, and Troubleshooting Performance-22
2006 BEA Systems, Inc. 477
Single-Threaded Option
Use the TMNOTHREADS environment variable for
processes within your application.
Set TMNOTHREADS=Y for all single-threaded clients
and single-threaded servers:
In the runtime environment for remote clients.
In the ENVFILE for servers and native clients.
Results in multi-threaded support code being bypassed
providing improved performance.
Tuxedo versions 8.1+ can still make use of the
Multi-threaded BRIDGE processes ignore TMNOTHREADS.
Versions 8.0 + only
Monitoring, Tuning, and Troubleshooting Performance-23
2006 BEA Systems, Inc. 478
Service/Server Bundling
The following criteria should be considered in bundling
Execution time
Service dependencies
Message buffer sizes and queue sizes
Expected number of requests
Executable size
Shared libraries (BEA Tuxedo/other)
Server/service replication
Process limits
Business rules
The application architect usually decides this !
Monitoring, Tuning, and Troubleshooting Performance-24
2006 BEA Systems, Inc. 479
Repackaging Servers
When service execution time differs by more than
20%, consider re-partitioning server functionality
instead of using LOAD for performance tuning.

Server App 1b
Servant Object A
Method1() [1.0 secs]
Method2() [1.1 secs]
Server App 1b

Service1() [1.0 secs]
Service2() [1.1 secs]
Server App 1a

Service3() [0.3 secs]
Service4() [0.2 secs]
After repackaging, overall response times
for requests to Service3 and Service4 will
probably improve because of less wait time
spent on the servers queue
Mix of services in Server with
large variation of service times
Server App 1
Servant Object A
Method3() [0.3 secs]
Method4() [0.2 secs]
Servant Object A
Method1() [1.0 secs]
Method2() [1.1 secs]
Server App 1

Service3() [0.3 secs]
Service4() [0.2 secs]

Service1() [1.0 secs]
Service2() [1.1 secs]
Monitoring, Tuning, and Troubleshooting Performance-25
2006 BEA Systems, Inc. 480
Repackaging Advertised Services
In most cases, the administrator does not have access to
the source code for the Tuxedo application servers.
Repackaging of services can still be accomplished by
using the CLOPT s option.
Original Configuration:
New Configuration:
CLOPT=-s SVC1:foo1
CLOPT=-s SVC2:foo2
Obtaining SERVICE to routine mapping:
C:\> serverApp1 -v
Monitoring, Tuning, and Troubleshooting Performance-26
2006 BEA Systems, Inc. 481
Measuring Service Execution Times
Tuxedo can be configured to measure the time it takes
to perform each service.
Use CLOPT -r option to enable service timings to be
written to stderr.
The -r option should normally be used in test environments.
Use the CLOPT -e option to specify the log file name.
(default is stderr)

CLOPT=-A -r -e svctimes.txt
Monitoring, Tuning, and Troubleshooting Performance-27
2006 BEA Systems, Inc. 482
Performance Logging Example
This file will contain a line entry for every service call
processed and from every server configured to output
to the log file.
Monitoring, Tuning, and Troubleshooting Performance-28
2006 BEA Systems, Inc. 483
Performance Logging Report
Use the txrpt utility to
produce a formatted
report from a service
execution log file.

-t Sort by descending elapsed service time
-n Report only services specified by name
-d Report only requests occurring on mm/dd
-s Report only requests starting after time
-e Report only requests ending after time
Output the Service Log Report:
C:\> txrpt < svctimes.txt
Monitoring, Tuning, and Troubleshooting Performance-29
2006 BEA Systems, Inc. 484
Multiple-Server-Single-Queue (MSSQ)
Services A,B,C
By default, each Tuxedo application server is assigned
its own IPC request message queue.
Multiple-Server-Single-Queue (MSSQ) sets enable two
or more servers to dequeue the same request queue.
Services A,B,C
Services A,B,C
Monitoring, Tuning, and Troubleshooting Performance-30
2006 BEA Systems, Inc. 485
When to Use MSSQ
Consider using an MSSQ set for:
High volume (request queue is always busy), quick
turnaround (fast response time)
Moderate number of servers that offer the same set of
services (rule of thumb 2 to 10)
Message request sizes are reasonable relative to the IPC
request queue size limit
MSSQ may not be suitable for servers expecting large
message sizes that may quickly fill the request queue.
Be aware that an MSSQ set still uses the same queue size
limited by the Operating System kernel parameter.
Monitoring, Tuning, and Troubleshooting Performance-31
2006 BEA Systems, Inc. 486
Configuring an MSSQ Set
Services A,B,C
Services A,B,C
Services A,B,C
Configuring an MSSQ Set:
When MIN>1, specifying a name for the
common request queue enables an MSSQ
set for this group of servers
REPLYQ is not required, but it is
recommended to avoid possible lost
replies from other service calls.
Monitoring, Tuning, and Troubleshooting Performance-32
2006 BEA Systems, Inc. 487
Automatic Start of Additional Servers
Use the CLOPT -p option to automatically start
additional servers and to shutdown servers when the
load has abated.
If the number of requests on
the queue exceed 10 for a
period of 7 seconds, an
additional server is started
until MAX servers are active.
If the number of requests on
the queue fall below 5 for a
period of 120 seconds, one
server is stopped until only
MIN servers are active.
Configuring an MSSQ Set:
CLOPT=-A p 5,120:10,7
Monitoring, Tuning, and Troubleshooting Performance-33
2006 BEA Systems, Inc. 488
Section Review
Configure the additional system and server runtime
configuration options
Optimize performance for application that do not use
all of the Tuxedo architecture
Determine service execution times
Configure and deploy MSSQ sets

In this section, we learned how to:
Monitoring, Tuning, and Troubleshooting Performance-34
2006 BEA Systems, Inc. 489
Lab Exercise
For details on the exercise, refer to the Lab Guide.
If questions arise, ask the instructor.
The instructor will determine the stop time.
Lab 16 PERF: Configure and Use Additional
Performance Enhancement Options
Monitoring, Tuning, and Troubleshooting Performance-35
2006 BEA Systems, Inc. 490
Road Map
1. IPC Resource Requirements
2. Runtime and Configuration Performance Options
3. The BEA Tuxedo Administration Console
Configuring the Web Server and Listener
Launching the Web browser
Connecting to a Domain Configuration
Monitoring and Configuration Options
Creating new Domain Objects
Monitoring, Tuning, and Troubleshooting Performance-36
2006 BEA Systems, Inc. 491
The Tuxedo Administration Console
The Administration Console is a Java-based applet that
you can download into your Web browser.
It can be used from a remote site to:
Monitor system events, logs, and statistics.
Dynamically create and configure administration objects.
Create new domain configurations.
Manage application resources.
Internet Explorer 6.0 using Java 1.4.x plug-in has been
validated by BEA however:
Other web browsers (with a JRE) will work on most

Monitoring, Tuning, and Troubleshooting Performance-37
2006 BEA Systems, Inc. 492
Setting Up Your Environment
Any commercial Web server can be used; BEA also
provides tuxwsvr with the system software.
Prior to starting the Web Server, the following
environment variables must be set:
TUXDIR: Set to the Tuxedo product installation directory
PATH: Must include the %TUXDIR%\bin directory
The tuxwsvr process runs in the background until the
machine shuts down or until it is terminated.
In UNIX, use kill -9 on the process id.
In Windows, use the Task Manager > Process Tab (End
Monitoring, Tuning, and Troubleshooting Performance-38
2006 BEA Systems, Inc. 493
Starting the Web Server tuxwsvr
During the software installation, the tuxwsvr.ini
file is automatically created.
Located in the %TUXDIR%\udataobj directory
You may edit this file if you wish to move the Java files to
another directory.
The l and i command-line options are required.
The l network address will be used for your browser URL.
The i is the initialization file (normally set to the standard
tuxwsvr.ini file).
Starting the tuxwsvr process from the command-line:
C:\> tuxwsvr l //student12:4321
i %TUXDIR%\udataobj\tuxwsvr.ini
Monitoring, Tuning, and Troubleshooting Performance-39
2006 BEA Systems, Inc. 494
Starting the Listener Process - wlisten
During the software installation, the wegui.ini file
is automatically created.
Located in the %TUXDIR%\udataobj\webgui directory
Contains all wlisten initialization parameter options
Edit this file if default values are not appropriate:
Uses port 4003 runs in the background no LLE etc.
The i command-line option is optional.
Used to specify a different file other than the default.
Starting the wlisten process from the command-line:
C:\> wlisten
Monitoring, Tuning, and Troubleshooting Performance-40
2006 BEA Systems, Inc. 495
Launching the Web Browser
Enter the following URL in your browser:
This is the IP Address of your Web Server.
In our example, the l option for the tuxwsvr process:
The BEA Tuxedo
Administration Console
splash screen displays.
Scroll past the licensing
information to the bottom of
the page; click the button to
enter the login page.
Monitoring, Tuning, and Troubleshooting Performance-41
2006 BEA Systems, Inc. 496
Login Information
After a few moments (depends on your machine and
The BEA Tuxedo Administration Console Login window
should display in a Java applet.
If you do not see the login window, you will need to download a Java
1.4.x plug-in for your browser.
The Password must be present in the
file located in %TUXDIR%\udataobj
Optionally provide a login name (no spaces)
It must remain active during your
session. You may minimize it to
get it out of your way.
NOTE: After the Console
displays, do not close this
browser window!
Monitoring, Tuning, and Troubleshooting Performance-42
2006 BEA Systems, Inc. 497
Administration Console Main Window
When you first enter the main window there is no
associated Tuxedo domain.
Subsequent logins will remember your last working domain.
Monitoring, Tuning, and Troubleshooting Performance-43
2006 BEA Systems, Inc. 498
Connecting to a Domain Configuration
To associate the Console with a pre-
configured Tuxedo domain:
Click on Domain > Open...
The domain does not have to be running; it is
only looking for the tuxconfig binary file.
A Java Applet window is displayed.
Enter the name and location of TUXCONFIG.
Enter the DOMAINID (if it has been
Monitoring, Tuning, and Troubleshooting Performance-44
2006 BEA Systems, Inc. 499
Monitoring the Domain
Monitoring, Tuning, and Troubleshooting Performance-45
2006 BEA Systems, Inc. 500
Toolbar Buttons
Invokes the context-sensitive help. Use the cross-hairs cursor to
click on a specific field or area.
Set the default mode (edit or view). Set location for online docs.
Displays the tab pages that allows for graphical monitoring info.
Displays the window for monitoring System Event Broker events.
Displays the ULOG file for a specific machine.
Used to initiative server group migration.
Deactivates (shutdown) all or part of a Tuxedo application domain.
Activates (boot) all or part of a Tuxedo application domain.
Opens a pop-up window to search for an administrative object.
Updates the tree view and configuration tool pane.
Interrupts the current request and returns control to the user.
Description Button
Opens the Admin Console on-line help in a separate Web browser.
Monitoring, Tuning, and Troubleshooting Performance-46
2006 BEA Systems, Inc. 501
Administration Console Tree View
The Tree View pane appears
in the left column of the
main GUI window.
Clicking on any object
corresponds to that MIB
class and retrieves its
attribute values.
Right-click an object to
reveal a quick-edit menu:
Options depend on object type
Icon color reveals the
objects state.
Monitoring, Tuning, and Troubleshooting Performance-47
2006 BEA Systems, Inc. 502
The Configuration Tool Pane
Each object has a corresponding
set of tabs in the Configuration
Tool pane.
The tab pages are electronic forms
that display information about the
administrative object.
The attributes that can be changed
are presented as input form fields.
In some cases, attempting to
change an attributes value will
return a MIB SET error:
Consult the MIB class documentation
Monitoring, Tuning, and Troubleshooting Performance-48
2006 BEA Systems, Inc. 503
Viewing System Statistics...

Each object has a corresponding set of tabs in the
Configuration Tool pane.
Monitoring, Tuning, and Troubleshooting Performance-49
2006 BEA Systems, Inc. 504
...Viewing System Statistics
Tracking client
selected server
Monitoring load
across services
Monitoring, Tuning, and Troubleshooting Performance-50
2006 BEA Systems, Inc. 505
Viewing the ULOG
The Logfile Tool provides filtering and formatting
options to view ULOG entries.
Once messages have been downloaded, clicking on a message
will bring up the HTML Message Catalog for that entry.
Monitoring, Tuning, and Troubleshooting Performance-51
2006 BEA Systems, Inc. 506
Creating New Domain Objects
New MIB class objects can be created:
Prior to booting the application
Dynamically while the application is active
The functionality is similar to tmconfig or ud32.
Create a new machine, group, server.
Monitoring, Tuning, and Troubleshooting Performance-52
2006 BEA Systems, Inc. 507
Starting and Stopping Servers
Most of the tmboot and
tmshutdown options are
The console terms are
activate and deactivate.
Monitoring, Tuning, and Troubleshooting Performance-53
2006 BEA Systems, Inc. 508
Section Review
Configure and deploy the Tuxedo Web Server and
Launch the Tuxedo Administration Console.
Use the more common features within the
Administration Console.
In this section, we learned how to:
Monitoring, Tuning, and Troubleshooting Performance-54
2006 BEA Systems, Inc. 509
Lab Exercise
For details on the exercise, refer to the Lab Guide.
If questions arise, ask the instructor.
The instructor will determine the stop time.
Lab 17 PERF: Configure, Deploy, and Test
the Tuxedo Administration Web Console
Monitoring, Tuning, and Troubleshooting Performance-55
2006 BEA Systems, Inc.
Module Review
Optimize IPC resources for Tuxedo application
Configure system and server options for improved
Troubleshoot applications with runtime tracing.
Use the Tuxedo Administration Console.
In this module, we learned how to:
Monitoring, Tuning, and Troubleshooting Performance-56