BMC BladeLogic Server Automation Best Practices for Deployment and Configuration

BMC BladeLogic Server Automation Best Practices for for Deployment and Configuration

REVISION HISTORY
Date March, 2011 Product version 8.1.00 Revisions Initial version.

Page 2

BMC BladeLogic Server Automation Best Practices for for Deployment and Configuration

TABLE OF CONTENTS
Revision history ............................................................................................................................................................... 2 Understanding job behavior ............................................................................................................................................ 4 Job execution framework ............................................................................................................................................ 4 Compliance ................................................................................................................................................................. 5 Component Discovery Jobs ........................................................................................................................................ 6 NSH Script Jobs ......................................................................................................................................................... 7 Deploy Jobs ................................................................................................................................................................ 7 Provision Jobs ............................................................................................................................................................ 9 Patching Jobs ............................................................................................................................................................. 9 Virtualization Jobs..................................................................................................................................................... 11 Administrative jobs.................................................................................................................................................... 12 Deployment guidance ................................................................................................................................................... 13 BMC BladeLogic Server Automation components .................................................................................................... 13 Simple installations ................................................................................................................................................... 16 Large-Scale installations ........................................................................................................................................... 17 Geographically-distributed installations .................................................................................................................... 19 Configuration guidance ................................................................................................................................................. 20 About Java memory .................................................................................................................................................. 20 About thread pools.................................................................................................................................................... 21 About database connections .................................................................................................................................... 21 Recommendations for job servers ............................................................................................................................ 22 Recommendations for Configuration servers ............................................................................................................ 24 Recommendations for NSH Proxy servers ............................................................................................................... 26 Appendix: TCP/UDP Port Usage................................................................................................................................... 27

Page 3

BMC BladeLogic Server Automation Best Practices for for Deployment and Configuration

UNDERSTANDING JOB BEHAVIOR

This section provides a brief overview of the runtime behavior of the various job types, with emphasis on computation, storage, and network resource requirements.

JOB EXECUTION FRAMEWORK All BMC BladeLogic Server Automation jobs execute in the job execution framework. This section describes the overall operation of the framework. Jobs, targets, and work items The execution of a BMC BladeLogic Server Automation job begins with the job being placed in a work queue of jobs waiting to execute. Each job server maintains a pool of threads, called the job thread pool, dedicated to executing jobs in the job queue. The main work of a job is the creation and management of individual work items and their results. Work items are separately-schedulable units of work that are undertaken as part of the execution of a job. It is in the execution of work items that the job carries out its responsibilities. Like jobs, work items are executed by job servers. A work item may or may not be executed by the same job server that is responsible for executing the job that created it. Work items scheduled for execution are maintained in a work item queue in the job server. A pool of threads, known as work item threads, is maintained by each job server for the execution of work items. A work item is almost always bound to one target host, or to a component on a target. Further, a job may generate multiple work items for each target, corresponding to different steps or stages of the job. Thus, a job that is scheduled to execute against 1,000 target hosts can create and schedule execution of possibly several thousand work items. Lightweight work items Some work items are designated as lightweight work items because their execution consumes significantly fewer server resources than does the execution of normal work items. In addition to the work item thread pool, each job server also maintains a separate thread pool for lightweight work items, the lightweight work item thread pool. If the lightweight work item thread pool is not empty (that is, has a non-zero size), threads from this pool manage lightweight work items; otherwise, all work items (lightweight and nonlightweight) are managed by the normal work item thread pool. Asynchronous BlExec tasks While a work item is executing, a work item thread is assigned exclusively to that work item. Some work items, however, spend much of their time waiting for results from operations being carried out remotely, for example, on a target host. A work item thread assigned to such a work item blocks while it waits. This can present a potential resource issue, as the work item thread is not available to service other work items with more active processing needs. This resource utilization concern is addressed by the introduction of asynchronous tasks. Asynchronous tasks are a relatively new feature of the job execution framework, having been introduced in BMC BladeLogic Server Automation version 8.0. When a work item must perform a remote operation, instead of performing that remote operation and waiting for a response, the work item can instead create and queue an asynchronous BlExec task to perform the operation. Then the work item itself terminates. Asynchronous BlExec tasks are managed by the BlExec service, which maintains yet another thread pool, but the tasks occupy a thread from the pool only when they have active processing to perform. While it waits for a response from the remote target, an asynchronous BlExec task does not consume any thread resources. A work item must be explicitly written to make use of asynchronous tasks; work items whose implementation is not asynchronous task aware still perform remote operations directly and cause their work item threads to wait for the operation to complete. Currently, some BMC BladeLogic Server Automation jobs are asynchronous task aware, and some are not.

Page 4

BMC BladeLogic Server Automation Best Practices for for Deployment and Configuration
Version 7.6 and earlier Version 8.0 Version 8.1 The BlExec service and asynchronous BlExec tasks are not available earlier than BMC BladeLogic Server Automation 8.0. In BMC BladeLogic Server Automation 8.0, NSH Script jobs and patch analysis jobs take advantage of asynchronous BlExec tasks. In BMC BladeLogic Server Automation 8.1, Deploy jobs and Virtual Guest Jobs also take advantage of asynchronous BlExec tasks.

COMPLIANCE This section describes Snapshot, Audit, and rules-based Compliance Jobs. Snapshot Jobs Snapshot jobs collect information about assets from a target and convey that information to the Application Server. As it is received by the Application Server, information about each asset is stored in a local (on the Application Server) file with a .snp (snapshot) suffix. The lone exception to this rule is that when a files content is included in a snapshot, the files content is received as a separate file and not included in the .snp file. One .snp file is constructed for each component part. When construction of the .snp file is completed on the Application Server, it is then scanned and compared to the most recent prior snapshot from that target, if one exists. This comparison is performed between the new .snp file, and a .bnp (baseline snapshot) representing the previous scan. The exact means by which the .bnp file is obtained depends on the BMC BladeLogic Server Automation version; however, processing steps follow this flow: Only in version 7.6 and earlier Version 8.0; Version 7.6 and earlier If the last scan of this particular target ran on the same job server and the old .bnp from that job run is still available on the job server, then the old .bnp file is used directly. Otherwise, the next processing step takes place. If the .bnp file from the last scan of this particular target is available on the file server (see below), it is copied from the file server to the local Application Server and then used for comparison. Otherwise, the next processing step takes place. The job server uses the data in the database to reconstruct a .bnp file as it would have existed following the last scan of this particular target. Version 8.0; Version7.6 and earlier Version 7.6 and earlier After the snapshot job executes, the .snp file is renamed to .bnp and copied to the file server. The .bnp file also remains on the job server. (In BMC BladeLogic Server Automation version 8.0 and later, the file is deleted).

After the comparison between the two .bnp files (that is, the .bnp of the current snapshot of the asset and the .bnp file from the prior snapshot), only the differences between the two versions are stored in the BMC BladeLogic Server Automation database. Therefore, repeated snapshots of an asset that changes very little or not at all result in relatively little information being stored in the database. For example, in the case of an unchanged file, no additional data is recorded in the database. Files are the most common type of asset used in Snapshot and Audit Jobs. To perform a full snapshot of a file, the entire contents of that file must be transferred to the Application Server. On the other hand, capturing just the MD5 digest (checksum) of a file, instead of the actual contents, results in the calculation of the 128-bit MD5 digest value on the target and transmission of only that 128-bit value to the Application Server for storage in the .snp file. The table shows a summary of resource usage for Snapshot Jobs. Application Server CPU High Network Traffic High Database Load Moderate Agent Low

Page 5

BMC BladeLogic Server Automation Best Practices for for Deployment and Configuration
Audit Jobs Audit Job behavior is largely similar to that of Snapshot Jobs, in that it involves constructing and comparing two snapshot files on the Application Server (with a .snp suffix this time), and recording only the differences. For Audit Jobs, master .snp files are always constructed as part of the job. For live audits, a snapshot of the master target is performed and the results captured in .snp files. For a snapshot-based audit, the master .snp file is generated from data in the database. The master .snp files are copied to the file server, if necessary, and shared among any Application Servers that run work items for the Audit Job. After the master .snp files have been constructed, each target of the audit job is processed by first constructing a target .snp file and then comparing the master and target .snp files. Differences between the two .snp files are recorded in the database. For each difference detected, the asset from the target .snp file is persisted in the database, regardless of how many earlier audits may have detected the same difference. After each audit target is processed, the target .snp files for that target are discarded. Upon completion of the Audit Job, master .snp files are marked for deletion on the file server. The table shows a summary of resource usage for Audit Jobs. Application Server CPU High Compliance Jobs Compliance jobs, also called rule-based compliance jobs, operate by: collecting asset information on the target transferring that data back to the Application Server applying the user-specified rules to the returned data to assess the targets compliance Unlike Audit and Snapshot jobs, Compliance Jobs do not use temporary snapshot files (.snp or .bnp) on the Application Server. When a Compliance Job runs, each work item operates by looping through the component parts of its assigned component; for each component part, it decides whether or not it needs to retrieve data from the target. If it does need to retrieve assets for the current component part, a single request is issued to collect the required information for each of the assets to be tested. As each requested asset arrives on the Application Server, the relevant conditions are applied to determine compliance. The result (compliant, noncompliant, or noncompliant with exception) of applying each condition is recorded in the database. In the case of a noncompliant result, the specific value under test is also recorded in the database. For example, a non-complying condition on file size causes the actual file size to be recorded in the database, but not the contents of the file. Compliance autoremediation If a target is noncompliant and if the Compliance Job has the Allow Auto-remediation option specified, then each rule failure selects a BLPackage to be included in a combined remediation BLPackage for that host. (Any other hosts with the same combination of failing rules will use the same remediation package.) The Compliance Job then runs a BLPackage Deploy Job against the noncompliant targets. The Compliance Job does not complete until the BLPackage Deploy Job has completed. The table shows a summary of resource usage for Compliance Jobs that perform autoremediation. Application server CPU High Network Traffic Moderate High Database Load Moderate Agent Moderate Network Traffic High Database Load Moderate Agent Low

COMPONENT DISCOVERY JOBS Component Discovery Jobs first use component applicability rules to select appropriate targets from the requested list of targets. Each suitable target is then contacted and sufficient assets collected to perform a test of the signature condition for the target. The table shows a summary of resource usage by Component Discovery Jobs.

Page 6

BMC BladeLogic Server Automation Best Practices for for Deployment and Configuration
Application Server CPU Low Moderate Network Traffic Low Moderate Database Load Moderate Agent Low

NSH SCRIPT JOBS Scripts executed by NSH Script Jobs are categorized by the four radio buttons presented in the jobs Add Script dialog: Execute the script separately against each host (Type 1) Execute the script once, passing the host list as a parameter to the script (Type 2) Copy and execute the script against each host separately (Type 3) Execute the script using the PERL interpreter, passing the host list as a parameter (Type 4) From a performance perspective, Type 3 jobs differ from the other types in that they execute the script on the target, rather than on the Application Server. Even for scripts executed on the Application Server however, any nexec commands are executed on the target. Version 8.0 and later As of BMC BladeLogic Server Automation version 8.0, users have the option of selecting asynchronous execution for Type 3 NSH Script Jobs. Choosing this option causes the job to be executed using asynchronous BlExec tasks; see Asynchronous BlExec tasks on page 4.

Process Spawner NSH Script Jobs invoke the actual NSH scripts in a separate process. That separate process can be created and managed either by the Application Server or by a separately-running application known as the Process Spawner. Use of the Process Spawner can significantly reduce the overhead of creating and tearing down the process used to execute the NSH script. Version 8.0; Version 7.6 and earlier Version 8.1 and later Prior to BMC BladeLogic Server Automation version 8.1, use of the Process Spawner can result in deadlocks or hangs under high workloads. BMC does not recommend using the Process Spawner in these versions. Issues with deadlock and hangs are resolved in release 8.1. Use of the Process Spawner offers significant performance benefits for NSH Script jobs. BMC recommends its use.

The table shows a summary of resource usage by the Process Spawner. Application server CPU Varies Network Traffic Varies Database Load Low Agent Varies

DEPLOY JOBS This section describes file and package deploy jobs. File Deploy Jobs A File Deploy Job arranges to deploy a file from any NSH-accessible location to one or more remote targets. A File Deploy Job operates by first constructing and then executing an .nsh script to copy (push) the requested file. This script runs on the Application Server. For a direct File Deploy Job, the script running on the Application Server copies the file from its source to each target. The Application Server acts as an intermediary; there is no direct data transfer between the source and the target. For an indirect File Deploy Job, the script running on the Application Server copies the file to one or more remote repeaters, with the Application Server again acting as an intermediary; a script then runs on each repeater to push the file to the final target.

Page 7

BMC BladeLogic Server Automation Best Practices for for Deployment and Configuration
When pre-commands or post-commands are specified as part of a File Deploy Job, these commands are executed on the remote target. The table shows a summary of resource usage by File Deploy Jobs. Application Server CPU Moderate Network Traffic High Database Load Moderate Agent Moderate

The server from which the files are deployed can experience heavy load during a File Deploy Job. Similarly, any Repeaters involved can experience heavy load during a File Deploy Job. BLPackage Deploy Jobs A BLPackage is an aggregation of many types of server objects, including, for example, registry keys and configurations within files, not just files. These server objects are packaged together for unattended deployment on multiple remote hosts. Phases of the BLPackage Deploy Job BLPackage Deploy Jobs comprise a sequence of work items run in the following phases: Phase Simulate Staging Work Item Description This is a dry run or preflight phase to verify that conditions exist which should lead to a successful execution. Any necessary files are copied to the target in preparation for deployment, possibly by way of repeater servers. If the package uses the agent mounts source option, no staging is required. Execute Pre commands, if any, on the target. Run installation commands on the target. Commit Execute Post commands, if any, on the target. Undo If the deployment is unsuccessful, its effects are reverted on the target. Notes Asynchronous BlExec task High file server load

Lightweight Work Item Lightweight Work Item, Asynchronous BlExec task Lightweight Work Item Asynchronous BlExec task

The Staging phase has the potential to generate significant workloads on the file server (or other server providing the package source files). The Commit phase, in contrast, presents almost no load to the BMC BladeLogic Server Automation infrastructure, as most of the work for this phase is carried out on the target hosts. With the exception of work items for predeploy and postdeploy commands, work items for the BLPackage Deploy Jobs Commit phase are implemented as Lightweight Work Items. See Lightweight work items, page 4. Version 8.1 and later As of BMC BladeLogic Server Automation version 8.1, several phase work items have been enhanced to use asynchronous BLExec tasks for execution, allowing for increased throughput even without populating the thread pool for lightweight work items. See Asynchronous BlExec tasks, on page 4.

The table shows a summary of resource usage by BLPackage Deploy Jobs. Application Server CPU Low Network Traffic High Database Load Low Agent Low

The Staging phase has the potential to generate significant workloads on the file server (or other server providing the package source files. The Commit phase, in contrast, presents almost no load to the BMC BladeLogic Server Automation infrastructure, as most of the work for this phase is carried out on the target hosts

Page 8

BMC BladeLogic Server Automation Best Practices for for Deployment and Configuration
PROVISION JOBS A Provision Job establishes the necessary network resources required for a target machine to be provisioned upon reboot; then the job monitors the progress of the provisioning activity as it occurs on the target. As the target device reboots, it requests progressive instructions from BMC BladeLogic servers and downloads boot images and operating system installation files from servers on the network. The devices boot process varies, depending on the type of target device, but in all cases, as the target device is rebooting, it requires network access to servers from which it can retrieve instructions and downloadable artifacts. Provisioning details Windows and Linux provisioning Provisioning support for Windows and Linux devices is based on the Pre-Execution Environment (PXE) standard. When booting under the control of a provisioning job, a Windows or Linux target contacts the following: DHCP server, to identify a PXE server PXE server, for initial booting instructions TFTP server, from which it downloads a pre-boot kernel image, in most cases. Application Server, from which it retrieves the system package Data store, for operating system installation files Generally speaking, none of these activities impose significant computational demands on the supporting servers, but the network link between the target device and the data store server may experience substantial bandwidth usage. Solaris provisioning The Oracle JumpStart technology used for provisioning Solaris machines relies on three separate JumpStart functions: JumpStart Boot Server JumpStart Install Server JumpStart Configuration Server These functions may be provided independently, or combined into one or two actual JumpStart servers. The boot server must be on the same network as the provisioning target. Of the three functions, the install server (the data store) bears the greatest load. AIX provisioning The IBM AIX Network Installation Manager (NIM) technology uses a NIM master to control the provisioning target, while installation files are served off an NFS server (data store). HP-UX provisioning The HP-UX Ignite technology uses a single Ignite master to control the provisioning target and to provide the operating system installation files. The table shows a summary of resource usage by Provision Jobs. Application Server CPU Very Low Network Traffic Low Database Load Low Agent Low

Provisioning servers (whatever the type) must be available to the target host being provisioned. The data store server may experience moderate to high load during provisioning.

PATCHING JOBS In BladeLogic, software patches released by a patch vendor (that is, Microsoft, Red Hat, Adobe) are conceptualized as comprising metadata, describing the patch and its applicability, and a payload, containing the actual bits of the patch.

Page 9

BMC BladeLogic Server Automation Best Practices for for Deployment and Configuration
Patches are stored in a repository in the computing environment, which is an NSH-accessible directory somewhere in the BladeLogic environment. An offline or air-gapped environment is one in which the repository does not have direct access to the internet and therefore patches cannot be directly downloaded from the vendor site to an offline repository. A common strategy for populating an offline repository is to transfer patch content on removable media with the help of a BMC-provided download utility. Patches are organized into patch catalogs, according to filters defined in BladeLogic. For example, the catalog for Windows patches is separate from the catalog for Red Hat patches. If a repository is to include Windows patches, and the Application Server is running on a Linux host, then you must identify a Windows Helper Server Location when you create the repository. The Windows Helper Server Location is a user-defined temporary directory on a Microsoft Windows server which is used to decrypt files downloaded from the vendor site. Catalog Update Jobs You can create Catalog Update Jobs for each type of patch repository. Online patch catalogs are updated by downloading additional content from vendor and/or metadata-provider websites. The Application Server running a catalog update job for an online repository requires web access to these sites (that is, the Application Server must be configured to allow traffic to pass through any firewalls and web proxy servers). Offline patch catalogs are updated by transferring content from a local server, which typically mounts removable storage media onto which patch information is already loaded. The table shows a summary of resource usage by Catalog Update Jobs. Application Server CPU High Patch Analysis Jobs On all supported platforms as of BMC BladeLogic Server Automation version 8.0, patch analysis processing takes place on the affected target. That is, the relevant metadata (typically less than 5 MB) is transferred from the repository to the target agent, where it is decoded; the target agent then performs the necessary calculation to determine which patches to install on the target. Version 7.6 and earlier Version 8.0, Version 8.1 and later Prior to BMC BladeLogic Server Automation version 8.0, patch analysis for Solaris was performed primarily on the Application Server, rather than on the target. This can present a moderate to high work load on the Application Server. As of BMC BladeLogic Server Automation version 8.0, patch analysis for Solaris now occurs on the target. Further, patch analysis for all target types now uses an asynchronous agent call, allowing greater concurrency on the Application Server. Network Traffic High Database Load Moderate Agent Low

The table shows a summary of resource usage by Catalog Update Jobs. Application server CPU Low* Network Traffic Low Database Load Low Agent Moderate High*

*See version-specific notes, above, for Solaris patch analysis. Patch Remediation Jobs A patch remediation job does the following: Runs a patch download job to download patch payloads of missing patches that have not yet been downloaded. For Windows and Solaris, you can run analysis with just the metadata, without downloading the payload. Red Hat requires a payload download. So you can create a Windows patch catalog with all Windows 2008 patches and only download payloads of the patches that are found missing. Based on the patch analysis results, the patch remediation job runs an algorithm that creates a set of BLPackages and BLPackage Deploy Jobs. If multiple servers have the same set of missing patches, the patch remediation job creates a single Deploy Job with BLPackages that target the servers. If different

Page 10

BMC BladeLogic Server Automation Best Practices for for Deployment and Configuration
servers have different patches missing, the patch remediation job creates a Deploy Job for each unique set of missing patches. The BLPackage Deploy Jobs are wrapped into a Batch Job. The Batch Job then executes immediately (if specified), or is scheduled to execute at a later time. The table shows a summary of resource usage by Patch Remediation Jobs. Application server CPU Low Network Traffic High Database Load Low Agent Low

See BLPackage Deploy Job, above, for more information about the resource demands of the deploy operations. Patch resources are stored in the patch repository, which may experience heavy workload during the Staging phase of deployment.

VIRTUALIZATION JOBS Virtual Guest Job A Virtual Guest Job constructs a virtual guest, with or without an operating system, from a known VCenter or other virtual infrastructure. Configuration decisions for the new virtual guest are captured in a Virtual Guest Package. Internally, Virtual Guest Jobs operate as BLPackage Deploy Jobs, so the behavior and resource demands of a Virtual Guest Job correspond to those of the Deploy Job. Virtual Guest Jobs require minimal Application Server resources. For some steps in its operation, a Virtual Guest Job communicates with the VCenter through a custom object (CO) that must be installed on the Virtual Center host. The table shows a summary of resource usage by Virtual Guest Jobs. Application Server CPU Very Low Network Traffic Low Database Load Low Agent Low

Virtual Guest Jobs make demands on the Virtual Center host to accomplish construction of the virtual guest. Virtual Infrastructure Discovery Job Also called a sprawl job, a Virtual Infrastructure Discovery Job scans the network to identify ESX servers or other virtual hosting environments and then interrogates them to identify guests hosted by that computer. The table shows a summary of resource usage by Virtual Infrastructure Discovery Jobs. Application Server CPU Low Moderate UCS Provisioning Jobs A Cisco Unified Computing System (UCS) chassis comprises a number of hardware blades which act as a pool of computing resources. The chassis also includes a hardware entity (the Fabric Interconnect) that manages all the computing, network, and storage connectivity resources. The BladeLogic UCS custom object (CO) communicates with this hardware entity. A UCS template is a configuration that contains settings to configure a blade to become a server. This template contains server identity information (MAC address), networking configuration, and storage configurations (WWNN and WWPN). The UCS Provisioning Job takes a predefined template, applies the template to a stateless blade (so that the blade becomes a server with an identity), and then provisions the server. The table shows a summary of resource usage by UCS Provisioning Jobs. Network Traffic Moderate Database Load Low Moderate Agent Low

Page 11

BMC BladeLogic Server Automation Best Practices for for Deployment and Configuration
Application Server CPU Very Low Network Traffic Low Database Load Low Agent Low

UCS Provisioning Jobs make demands on the UCS Fabric Interconnect to accomplish the actual construction of the virtual guest.

ADMINISTRATIVE JOBS Update Server Properties Job The Update Server Properties Job invokes miscellaneous remote commands to obtain server name, operating system type and version, IP address, etc. The table shows a summary of resource usage by Update Server Properties Jobs. Application Server CPU Low ACL Push Jobs At its core, an ACL Push Job computes a set of entries for the user file on each target, and then overwrites the user file (the file in the targets rsc directory whose name is user) with those entries. The table shows a summary of resource usage by ACL Push Jobs. Application Server CPU Low Network Traffic Low Database Load Low Agent Low Network Traffic Low Database Load Low Agent Low

Distribute Configuration Objects Jobs The table shows a summary of resource usage by Distribute Configuration Objects Jobs. Application Server CPU Low Network Traffic Low Moderate Database Load Low Agent Low Moderate

Decommission Configuration Object Jobs The table shows a summary of resource usage by Decommission Configuration Object Jobs. Application Server CPU Low Network Traffic Low Database Load Low Agent Low

Page 12

BMC BladeLogic Server Automation Best Practices for for Deployment and Configuration

DEPLOYMENT GUIDANCE
A BMC BladeLogic Server Automation deployment typically involves a large number of individual software elements arrayed across a number of physical servers deployed around the environment. This section discusses performance and other considerations for the deployment of the various BMC BladeLogic Server Automation software elements.

BMC BLADELOGIC SERVER AUTOMATION COMPONENTS This section describes the components that may constitute a BMC BladeLogic Server Automation installation. Database server At the center of every BMC BladeLogic Server Automation installation is the BMC BladeLogic database server. Application Servers are tightly coupled to the database and impose significant demands on the server that hosts the database. BMC recommends the use of a dedicated physical machine or cluster to host the database server for BladeLogic. The database server or cluster should be on the same LAN as the BMC BladeLogic Server Automation Application Server. High latency on the link between the Application Servers and the database server can cause unacceptable performance for BladeLogic. Additionally, there is anecdotal evidence that high packet loss rates on the Application-Server-to-database link may cause issues for (expose defects in) the Oracle JDBC driver. Application Servers A BMC BladeLogic Server Automation deployment comprises one or more Application Server (appserver) processes. The number and configuration of Application Servers in a deployment depends on many factors, including the number of targets to be managed and the expected job load for the system. Within limits, it is acceptable to run multiple Application Servers on a single physical server while still maintaining acceptable performance. For more information, see Adding Application Server instances, on page 17. An Application Server can fulfill any of several distinct profiles, or combinations of profiles, depending on its configuration. Configuration servers BMC BladeLogic Server Automation clients (rich client UI, BLCLI command line client, bmi.exe) connect to configuration servers to allow interaction with the BladeLogic system. A configuration (UI) server is an Application Server of type CONFIGURATION of type ALL (which includes CONFIGURATION). A configuration server provides middle-tier functionality, answering requests from BMC BladeLogic Server Automation client applications both for data and for operations on that data. Job servers Application Servers configured as job servers are responsible for the execution of BMC BladeLogic Server Automation jobs. In many environments, job servers are limited by internal resource contention, but, surprisingly, do not impose excessive workload on the hardware. Accordingly, it may be advisable to configure multiple job servers on the same physical machine in order to make more complete use of the available hardware resources. Alternatively, in a virtualized environment, you can run multiple job server guest VMs on the same physical server. NSH Proxy Servers NSH Proxy Servers perform a specialized role in BladeLogic installations, as described in NSH proxies, page 16.

Page 13

BMC BladeLogic Server Automation Best Practices for for Deployment and Configuration
Process Spawners A BladeLogic Process Spawner offers improved performance for NSH Script jobs under certain circumstances. See Process Spawner considerations, page 24. PXE servers A BMC BladeLogic Server Automation PXE server, technically a specially configured Application Server, performs a specialized role in support of provisioning jobs. PXE servers are discussed in Servers for provisioning, page 14. Authentication servers Although not a separate Application Server profile, at least one Application Server in a BMC BladeLogic Server Automation environment must be configured as an authentication server. As its name implies, the authentication server verifies the identity of a BMC BladeLogic Server Automation user, after which the user is allowed to interact with the BladeLogic client. Authentication servers do not normally experience a high work load. A typical practice is to configure one of the configuration Application Servers also to act as an authentication server. It is not normally necessary to configure more than one authentication server for a single BladeLogic environment. File server Every BMC BladeLogic Server Automation environment includes a server designated as the file server. Any server running an RSCD agent can be designated as the file server for the installation. Both the performance of the file server and the network connection between job servers and the file server have a critical impact on Deploy Jobs. Using NFS as a file server Because NFS sharing provides higher performance than NSH data transfer, BMC BladeLogic Server Automation performance can be enhanced by employing an NFS-based network-attached storage (NAS) device and mounting the storage on each physical computer hosting an Application Server. In this configuration, localhost should be designated as the file server, so that each Application Server treats the shared mount point as local storage. Repeaters For environments in which deploy job performance over the WAN is a concern, BMC recommends the use of one or more repeaters at each data center. Properly configured, a repeater serves as a staging location at each site for packages as they are deployed. Advanced Repeaters An Advanced Repeater server is simply a repeater that uses BMC BladeLogic Advanced Repeater technology to enable file servers and repeater servers to store and share data more efficiently. BMC recommends the use of advanced repeaters whenever repeaters are deployed across a WAN. BMC BladeLogic Server Automation consoles Communication between the BMC BladeLogic Server Automation console and Application Servers requires significant bandwidth. BMC recommends deploying consoles to servers on the same LAN as the Application Servers to which they connect. It is possible, but not recommended, for the console and Application Server to be separated by a longer network link, but performance under that configuration may be unacceptable. For environments in which a population of geographically-dispersed users must all have access to the BMC BladeLogic Server Automation console, BMC recommends running the console on a Citrix Presentation Server. This allows users who are offsite from the presentation server to run remote instances of the UI without experiencing excessive latency. Servers for provisioning BMC BladeLogic Server Automation provisioning works with different provisioning technologies, depending on the type of server being provisioned.

Page 14

BMC BladeLogic Server Automation Best Practices for for Deployment and Configuration
Windows and Linux provisioning PXE servers support Windows and Linux provisioning jobs by providing boot-time services to target devices. Each target device needs to have access to a local PXE server, usually on the same LAN. Although PXE servers do communicate with the database, the data volume of that communication is relatively low. It is therefore acceptable to install geographically-removed PXE servers, which must communicate with the database over longer network legs. The PXE server and the TFTP server must reside on the same physical server. A provisioning target also needs access to the BladeLogic Application Server and a data store. Traffic to the Application Server is relatively light, so it is not necessary for the Application Server to be geographically proximate to the provisioning target. However, if the provisioning target will be retrieving files from a data store, it is preferable that the data store be local to the provisioning target.

Solaris provisioning The Oracle Solaris JumpStart technology identifies a JumpStart boot server, a JumpStart config server, and a JumpStart install server, all of which may be (and commonly are) hosted on the same physical device. JumpStart servers should be located on the same LAN as the Solaris servers being provisioned, but may be remote from the BMC BladeLogic Server Automation Application Server. To use a JumpStart server with provisioning jobs, you must install an RSCD agent on the JumpStart server. AIX provisioning The IBM AIX NIM technology requires a NIM Master server on the same LAN as the AIX servers being provisioned. To use a NIM Master with provisioning jobs, you must install an RSCD agent on the NIM Master server.

Page 15

BMC BladeLogic Server Automation Best Practices for for Deployment and Configuration
HP-UX provisioning The HP-UX Ignite technology requires an Ignite Master server on the same LAN as the HP-UX servers being provisioned. To use an Ignite Master with provisioning jobs, you must install an RSCD agent on the Ignite Master server. Proxies NSH proxies Historically, NSH proxy servers played an important role in negotiating fire walls in large scale deployments, but this practice is no longer recommended. In BMC BladeLogic Server Automation versions 7.x and later, NSH proxies are used mainly as a security enhancement measure. SOCKS proxies To access targets that are behind a firewall (because they are in a remote data center, for example) or otherwise not directly accessible from the Application Servers, BMC recommends using SOCKS Proxy Servers. In this situation, establish a SOCKS Proxy Server in each remote data center and configure any intervening firewalls to allow the Application Servers to contact the SOCKS proxy over port 1080. Configure the Application Server to establish communications with the remote targets by using the SOCKS proxy, usually over port 4750, rather than contacting the remote hosts directly. A SOCKS proxy normally requires minimal computing power but can be expected to have network bandwidth demands commensurate with its role as a communication concentrator for the remotely-managed targets.

SIMPLE INSTALLATIONS This section describes basic considerations applicable to all BMC BladeLogic Server Automation installations. See also Large-Scale installations on page 17, and Geographically-distributed installations, page 19, for additional considerations. You can set up a very small-scale installation using just two physical management servers to host BMC BladeLogic Server Automation. One computer is dedicated to hosting the database, while the other hosts all the essential BladeLogic components: Application Server offering: Job server Configuration (UI) server Authentication server File server Management console UI (BMC BladeLogic Server Automation console) This simple installation highlights the fact that BMC BladeLogic Server Automation makes significant use of the associated database. Accordingly, BMC recommends a dedicated database server or cluster to support a BMC BladeLogic Server Automation installation.

Page 16

BMC BladeLogic Server Automation Best Practices for for Deployment and Configuration

For demonstration or other specialized purpose, it is possible to host all the components on one machine, including the database server. However, doing so is likely to lead to unacceptable performance in most cases.

LARGE-SCALE INSTALLATIONS Most customer environments are too large to be managed by the simple 2-server infrastructure described in the previous section. Fortunately, you can add BMC BladeLogic Server Automation components to provide greater management capacity. This section describes the use of additional infrastructure to provide greater capacity for a BMC BladeLogic Server Automation installation. Adding Application Server instances To meet the demands of a larger data center, it is likely that additional Application Servers will need to be deployed. Most commonly it is necessary to add job servers to provide support for a larger number of managed servers. In some cases, it may also be necessary to deploy additional Configuration (UI) servers to support a larger user population. Increasing job throughput To execute more jobs against more targets in a given period of time, it is usually necessary to increase the number of work item threads (WITs) available to execute jobs. The number of WITs is a configurable option of each job server, but the number of WITs per job server is normally limited by the amount of memory available in a single Application Server. In most cases, then, adding WITs means configuring another Application Server. It is frequently the case that a physical server has CPU and other resources sufficient to host several times the total number of WITs that can be run in a single Application Server. A rule of thumb is to install Application Servers on physical servers based on the assumption that each Application Server requires: Two CPU cores Physical memory sufficient for the Application Server process (4 GB for a 32-bit Application Server and 8-10 GB for a 64-bit Application Server). Under these guidelines, a typical eight-core server computer with sufficient memory can support three to four Application Servers. In figuring required RAM for the physical server, remember to allow memory for the operating system and for other processes running on the computer, including the Application Server launcher. See Configuration guidance on page 20 for more detailed suggestions on memory and WIT settings for job servers.

Page 17

BMC BladeLogic Server Automation Best Practices for for Deployment and Configuration
Support for more users For environments supporting a large user population, you may need to increase the number of Configuration (UI) Servers in the installation. The workload required to support a user varies widely, of course, but as a starting point, in the absence of additional information, BMC typically recommends: Install one Configuration server for every 50 concurrent logged-in users. Expect as many as 20% of total users to be logged in at any one time. In combination, these guidelines call for one Configuration server for every 250 users. Limits to growth Neither Oracle nor SQL Server has a theoretical limit on the number of database connections that a database server can support. However, the actual physical resources available on the database server impose a practical limit on the number of database connections that that particular database server can maintain. This, in turn, limits the total number of Application Servers a particular BMC BladeLogic Server Automation implementation can support. You can control the minimum and maximum number of database connections maintained by an Application Server through user-configurable settings for the various database connection pools. If you plan to establish an extremely large BMC BladeLogic Server Automation implementation, you should use the information in the Configuration Guidance section to estimate the total number of database connections required for the implementation. Then work with the local DBA and database vendor to ensure that the database server is capable of supporting that load. Scaling the file server The BMC BladeLogic Server Automation design requires a designated File Server to host the files in the BladeLogic Depot. The File Server is simply a server running the RSCD agent. A NAS filer using NFS or SMB can act as a kind of virtual file server, a configuration offering several benefits in terms of performance and scalability. In this configuration, a share exported by the filer is mounted at the same mount point on each computer hosting an Application Server, making the share appear to be local storage for each Application Server. The File Server is then defined to be localhost, and the file storage path is that on which the shared storage is mounted. This configuration offers potentially improved performance because the NFS protocol used by the filer exhibits better performance over the network than does the NSH protocol. In addition, this configuration allows the use of clustered NAS servers, allowing for redundancy and higher performance. Considerations for virtualized environments When BladeLogic Application Servers are hosted in virtual (guest) machines in a virtualized environment, the choice naturally arises whether it is better to deploy multiple Application Servers in a single virtual machine, or spread the Application Servers across separate virtual machines. In both cases, the Application Servers themselves are likely running on the same physical host computer. In virtualized environments, BMC recommends deploying Application Servers in separate virtual machines. Further, the Application Server performs best when the virtual machine hosting it is configured to have one dedicated virtual CPU (vCPU). In addition, for best performance, avoid allocating more vCPUs than the physical host has physical CPU cores. See BMC BladeLogic Application Server Running on VMware ESX: Performance and Scalability Best Practices, http://documents.bmc.com/supportu/documents/60/54/106054/106054.pdf; and BMC BladeLogic 8.0 Application Server Running on Red Hat Xen: Performance and Scalability Best Practices, http://documents.bmc.com/supportu/documents/29/84/142984/142984.pdf . Load balancing In large deployments involving multiple instances of some or all BMC BladeLogic Server Automation components, it may be necessary to provide load balancing services to ensure that the extra resources being applied are being utilized appropriately. Job servers effectively perform their own load balancing, scheduling jobs and work items according to availability. No additional load balancing considerations are applicable for job servers.

Page 18

BMC BladeLogic Server Automation Best Practices for for Deployment and Configuration
Two strategies for load balancing are commonly applied for configuration (UI) servers: For cases where the user population and behaviors support it, you can achieve a crude but effective load balancing simply by assigning different users to use different configuration servers. For more homogeneous load balancing, you must add an external load balancer to the installation and use it to distribute the load across configuration servers. The BIG-IP product by F5 is a common choice for this purpose. Load balancing for authentication servers (for example, for purposes of failover) must similarly be provided by an external load balancer.

GEOGRAPHICALLY-DISTRIBUTED INSTALLATIONS For a variety of reasons, it is unusual for the largest customer environments to be entirely contained within a single data center. In these cases it is necessary to consider not just the scale of the BMC BladeLogic Server Automation installation but also its geographic distribution. This section describes additional infrastructure recommended for managing servers in remote data centers. SOCKS proxies For a remote data center accessible only through a firewall, BMC recommends the use of a SOCKS proxy in the remote data center. The firewall can be configured to route connections on port 1080 (the SOCKS proxy port) to the SOCKS Proxy Server, which then brokers a connection to the actual target server agent (on port 4750, the RSCD agent port). Repeaters You can configure repeaters as staging areas for deployment files. Appropriate use of repeaters in remote data centers can significantly reduce the amount of network traffic that must be carried over long (slow and/or expensive) data lines. The BMC BladeLogic Advanced Repeater is an enhancement to the repeater architecture that provides scalable transport of data over wide-area networks. Advanced repeaters also offer additional features, such as bandwidth throttling and secure communications, that may be important for large-scale installations. BMC recommends the use of advanced repeaters for geographically-distributed deployments, with at least one repeater configured in each remote data center. Deploy Jobs with targets in remote data centers should normally be configured to use indirect push staging. (This staging pushes to the repeater and then pushes from the repeater to the target.) Provisioning servers As a rule, support for provisioning targets in remote data centers must be provided from provisioning servers located in the remote data center. Depending on the provisioning technology used, each remote data center must provide support for one or more provisioning-related services, for example DHCP, PXE and TFTP servers. Citrix Presentation Server If, in addition to remote managed servers, the installation also requires access to BMC BladeLogic Server Automation for remote users, BMC recommends the use of a Citrix Presentation Server. For performance reasons, it is usually not practical to deploy a management console (BMC BladeLogic Server Automation console) at a remote site, due to the bandwidth and latency requirements for the console-to-configuration server link.

Page 19

BMC BladeLogic Server Automation Best Practices for for Deployment and Configuration
CONFIGURATION GUIDANCE
This section offers guidance on appropriate settings for the configuration parameters for a BMC BladeLogic Server Automation Application Server. Refer to the BMC BLADE L OGIC SERVER AUTOMATION ADMINISTRATION GUIDE for details on using the blasadmin tool to control the configuration parameters. Apart from some general discussion, this section organizes BMC configuration recommendations according to type for single-purpose Application Servers. An Application Server can, of course, be configured to provide the combined services of the single-purpose Application Server (an Application Server of type ALL). In this case, you must modify the parameter recommendations, usually by adding recommended values for the same parameter for different Application Server types.

ABOUT JAVA MEMORY Effective operation of a large Java system like the BladeLogic Application Server depends critically on the availability of sufficient heap memory. This section provides an overview of some considerations that apply to correctly sizing Java memory for BladeLogic. Process space, Java heap, and native heap A Java process comprises two distinct memory areas: the Java heap and the native heap. Both heaps, together with the Java executable code itself, must fit within the footprint of a single process. Increasing the maximum size of the Java heap necessarily decreases the maximum possible size of the native heap that can fit within a certain process size. The java heap contains Java objects and accounts for most of the memory required by a running Application Server. The Java heap is managed by the Java garbage collector, and so is sometimes called GC heap. The native heap (also sometimes called the C heap) contains thread stacks, file handles, and other objects not managed by the Java garbage collector. If the maximum Java heap size is set too low, it is possible to run out of Java heap memory. If the maximum Java heap size is set too high, it is possible to run out of native heap memory. To complicate matters further, peak memory use for either the Java heap or the native heap depends on the precise work load being considered, as well as timing effects between concurrently-operating threads. Therefore, the recommendations that follow are merely that: recommendations, not guarantees or absolute limits. 32-bit processes A process running under any 32-bit operating system is limited to 4 GB of virtual address space, from which the operating system must reserve a significant portion for itself. For example, 32-bit Windows divides the entire address space in half, allowing an application process only 2 GB total private process space. For large Java applications like Application Servers, this process space limit imposes a ceiling on the number of threads that can be accommodated within a single Application Server. 64-bit processes A process running under a 64-bit operating system has access to a much larger virtual address space. Compared to a 32-bit Java process performing equivalent work, a 64-bit Java process also requires a larger Java heap, typically 50% or more larger than the 32-bit Java process. For an example, see http://users.elis.ugent.be/~leeckhou/papers/SPE06.pdf. Recommended Java heap settings This section describes recommended Java heap sizes for Application Servers running under different operating systems. These are recommendations only and must be adjusted in light of observed conditions, especially out-of-memory errors.

Page 20

BMC BladeLogic Server Automation Best Practices for for Deployment and Configuration
For 32-bit processes, BMC recommends operating system-specific Java heap size values according to the table below. Due to memory constraints, job servers using 32-bit processes should be configured to use no more than 50 work item threads. For 64-bit processes, BMC recommends that the Java heap size be increased as indicated in the table, if there is sufficient physical memory to support this setting. blasadmin Setting Module AppServer Setting MaxHeapSize Description and Recommendation Specifies the maximum heap size for this Application Server. See operating system-specific recommendations for this value summarized in the table below.

Max Java Heap Recommendations Operating System Windows Linux Solaris 32-bit 1024 MB 1536 MB 2048 MB 64-bit 6144 MB 6144 MB Not applicable

ABOUT THREAD POOLS An Application Server maintains several thread pools, each dedicated to a specific purpose. Selecting appropriate sizes for each of the various thread pools is one of the most important configuration choices for an Application Server. Each thread consumes resources, especially memory, even when idle. Java threads may also consume operating system resources such as thread handles. While executing, a thread consumes even more memory. Regardless of additional performance considerations, available process size limits the number of threads available in an Application Server. Threads within the same process share certain data structures, especially caches, which are not shared between threads in different processes. Increasing the number of threads within a single process, especially threads within a particular thread pool, has two consequences: Serendipity: Because there are more threads contributing to the process-wide caches, any given item request from any thread is more likely to be fulfilled from the cache, because another thread is more likely to have already placed the element in the cache. This phenomenon has a mildly positive effect on overall performance as the number of threads increases. Contention: Because some operations on some data structures require exclusive access, as the number of threads increases, there is a greater likelihood of one thread having to wait for another threads exclusive access to conclude. This effect degrades per-thread performance as the number of threads increases; that is, increasing the number of threads is subject to diminishing returns, sometimes sharply so. As the number of threads in a process grows, the negative contention effects grow more rapidly than do the positive serendipity effects. Doubling the number of threads in a pool improves performance, but doesnt double it. Each additional thread provides a smaller and smaller net benefit, while still consuming as much memory and other resources as any other thread.

ABOUT DATABASE CONNECTIONS Connections between an Application Server and the database are managed in three connection pools, with each pool devoted to a different purpose. Each connection pool allows the configuration of a minimum and maximum number of connections, although BMC recommends leaving the minimum value at zero for all connection pools.

Page 21

BMC BladeLogic Server Automation Best Practices for for Deployment and Configuration
Configuring a database pools maximum size to be too high wastes resources, and for large installations, may risk exceeding the total capacity of the database server. You must also ensure that the database server has sufficient capacity to service all the connections from all the connection pools for all the Application Servers in the environment. BMC recommends working with the DBA and database vendor to ensure that you have this capacity, particularly for very large installations. Conversely, configuring a database pool with a maximum size that is too low can degrade performance, as a thread requesting a database connection from an empty connection pool blocks until a connection becomes available.

RECOMMENDATIONS FOR JOB SERVERS This section provides general recommendations for configuring Application Servers established as job servers. Recommendations for the work item thread pool The work item thread pool is the thread pool whose configuration has the greatest effect on overall job performance. However, selecting the best size for this thread pool involves trade-offs, and the best size will be different for different environments. Most jobs generate one or more work items per target host, so a job targeted at thousands of servers can be expected to result in thousands of work items being queued for processing. Further, the work items themselves tend not to be CPU intensive. In light of these considerations, it is usually desirable to allocate a generous number of work item threads for a job server. For 32-bit Application Servers, BMC recommends a setting of 50 work item threads. Configuring a larger number of work item threads risks an OutOfMemoryError under the process size limitations of 32-bit processes. For 64-bit Application Servers, larger available process spaces make it possible to use a larger number of work item threads. The number of work item threads to configure is primarily determined by the effects of contention between work item threads. Version 7.6 and earlier, Version 8.0, Version 8.1 Up through BMC BladeLogic Server Automation 8.1, BMC recommends establishing additional Application Servers instead of increasing the number of work item threads. BMC recommends 50 work item threads for each of these Application Servers.

blasadmin Setting Module AppServer Setting MaxWorkItemThreads Description and Recommended Value Number of threads that can be used to execute job parts, for parallelism. BMC recommends 50 work item threads for both 32-bit and 64-bit Application Servers. Recommendations for the lightweight work item thread pool Lightweight work item threads are of benefit primarily for Deploy Jobs. For installations in which Deploy Jobs represent a significant fraction of the workload, BMC suggests a value of 200 threads for lightweight work items. The default value of 0 threads for lightweight work items uses ordinary work item threads for the execution of all work items, lightweight or not. blasadmin Setting Module AppServer Setting MaxLightweightWorkItem Threads Description and Recommended Value Number of threads that can be used to execute lightweight job parts, for parallelism. BMC recommends a value up to 200 if Deploy Jobs are a primary use; otherwise 0.

Page 22

BMC BladeLogic Server Automation Best Practices for for Deployment and Configuration
Recommendations for the BlExec service and thread pool An Application Servers BlExec service maintains a pool of threads for the execution of asynchronous tasks involving communication with remote targets. It is not normally necessary to change the BlExec services configuration settings; the default values produce good results in most cases. blasadmin Setting Module AppServer Setting EnableAsyncExecution Description and Recommended Value Enables/disables the async execution framework for jobs that allow it, for example, Deploy Jobs. You can set the value to true or false; the default setting is true. BMC recommends leaving async execution enabled. BlExec MaxSocketConnections Maximum simultaneous sockets open by the BlExec service. BMC recommends using the default value of 500 in most cases. A higher value allows more simultaneous connections; a lower value reduces the demand for file descriptors. BlExec NumWorkerThreads Number of worker threads used by the BlExec service. BMC recommends using the default value of 20. Other parameters For completeness, this section describes some additional configuration parameters related to thread pool sizes for job servers. These configuration parameters do not normally require adjustment from their default values. blasadmin Setting Module AppServer Setting MaxApprovalThreads Description Number of Approval Threads. This parameter governs a small pool of threads used to communicate with BMC Atrium Orchestrator. AppServer MaxJobs Maximum number of jobs the Application Server can execute simultaneously. The maximum number of jobs the Application Server allows to run simultaneously, regardless of the availability of resources to execute the jobs. AppServer MaxJobThreads Maximum number of threads that can be used to execute a job, for parallelism. Maximum size for the job execution pool. The job execution pool is distinct from the work item thread pool. While most of the work involved in executing a job is delegated to work items, some of the work, such as creating the work items themselves, remains the responsibility of the job itself, and will be carried out by the job execution pool. JobFactory GlobalDefaultJobParallelism Global default value for Job Parallelism made available to user. This parameter has no direct effect on the operation of the Application Server. Rather, it is the default value that appears in the UI for a jobs maximum parallelism option, when that option is selected.

Page 23

BMC BladeLogic Server Automation Best Practices for for Deployment and Configuration
Recommendations for database connections For best job server performance, BMC recommends allowing the job execution connection pool to grow up to twice the number of work item threads (MaxWorkItemThreads in AppServer module). Version 7.6 and earlier, Version 8.0 Up through BMC BladeLogic Server Automation 8.0, logging of output from NSH script jobs is handled with connections from the general database connection pool, while other jobs use the job execution database connection pool. For best performance of NSH script jobs in these versions of BladeLogic, BMC recommends setting the maximum size for the general database connection pool to twice the number of work item threads.

blasadmin Setting Module Database Setting MaxJobExecutionConnections Description and Recommended Value Maximum connections in the pool for job execution thread group. BMC recommends a value that is twice the number of work item threads (MaxWorkItemThreads). Database MaxGeneralConnections Maximum connections in the pool for general thread group. For BMC BladeLogic Server Automation version 8.0 and earlier, BMC recommends a value of 2 * MaxWorkItemThreads, especially for environments that depend heavily on NSH script jobs. For version 8.1 and later, the default value should be adequate. Process Spawner considerations As memory size increases, so too does the cost of spawning a new process directly from the Application Server. For NSH script jobs, a job server can be configured to use a Process Spawner to spawn subprocesses, rather than spawning them directly. The Process Spawner is simply a process with a small memory footprint that can spawn new processes without the penalty of the Application Servers large memory footprint. As the configured size of the Application Server grows, the benefit of using the Process Spawner increases. blasadmin Setting Module ProcessSpawner Setting SpawnExternally Description and Recommendation Processes should be spawned outside the Application Server or not. BMC recommends using the Process Spawner for all job servers for BMC BladeLogic Server Automation version 8.1 and beyond.

RECOMMENDATIONS FOR CONFIGURATION SERVERS Estimating client connections Parameter value settings for configuration (UI) servers should be based on the number of client connections you anticipate being made to the configuration server. This value is the total of the number of client connections from UI clients (RCP) and from BLCLI clients. The number of client connections opened by a UI client varies over time and depends on the operations that the user is engaged in at any given moment. As an initial estimate, BMC recommends a planning figure of 2.5 client connections for each concurrent GUI user. The BLCLI client uses exactly one client connection for its execution, and is usually much more short-lived than an interactive users GUI session.

Page 24

BMC BladeLogic Server Automation Best Practices for for Deployment and Configuration
Thus, the peak demand estimate for client connections for the configuration server is: 2.5 * (number of simultaneous GUI users) + (number of simultaneous BLCLI commands) If multiple configuration servers with a load balancer will be established, then the total load for client connections can be divided across the number of configuration servers that will be established. Recommendations for the client connection service and thread pool The client connection service is responsible for managing connections from client processes in Application Servers acting as configuration (UI) servers. The client connection service maintains a pool of threads for servicing client requests. The following table describes the parameters that most strongly affect the performance of the client connection service. In most cases, it is not necessary to change these parameters from their default values. blasadmin Setting Module AppServer Setting MaxClientContexts Description and Recommended Value Number of maximum client connections to the Application Server. BMC recommends estimating peak client connection demand, as described in the previous section, and setting MaxClientContexts to this value. In the absence of sufficient information from which to form an estimate for peak client connection demand, BMC recommends using the default value of 200. AppServer MaxWorkerThreads Number of client connection worker threads. BMC recommends using the default value of 10, or approximately 5% of the value of MaxClientContexts. Recommendations for database connections Similarly, for best configuration (UI) server performance, BMC recommends allowing the pool of database connections for client service threads to grow up to twice the number of client connection service threads (MaxWorkerThreads in AppServer module). blasadmin Setting Module Database Setting MaxClientConnections Description and Recommended Value Maximum connections in the pool for client connections. BMC recommends a value that is twice the number of client connection threads.

Page 25

BMC BladeLogic Server Automation Best Practices for for Deployment and Configuration

RECOMMENDATIONS FOR NSH PROXY SERVERS BMC recommends the use of NSH Proxy servers as a best practice for security. For best performance, configure the NSH Proxy server for the anticipated number of concurrent NSH connections it will be expected to handle. blasadmin Setting Module AppServer Setting MaxNshProxyContexts Description and Recommended Value Maximum number of NSH proxy connections to the Application Server. Set this value to the maximum number of concurrent NSH connections the proxy will be expected to handle. AppServer MaxNshProxyThreads Number of NSH proxy threads. This value can be significantly less than MaxNshProxyContexts, to account for idle NSH connections. In the absence of usage estimates specific to the installation, BMC suggests an initial estimate of 20% of MaxNshProxyContexts. Database MaxClientConnections Maximum connections in the pool for client connections. For an Application Server configured to act exclusively as an NSH proxy server, this value should be the same as MaxNshProxyThreads. For an Application Server configured to act as both a configuration server and an NSH proxy server, this value should be the sum of MaxWorkerThreads and MaxNshProxyThreads.

Page 26

BMC BladeLogic Server Automation Best Practices for for Deployment and Configuration

APPENDIX: TCP/UDP PORT USAGE

The following table summarizes the use of TCP/UDP ports across all the elements of a BladeLogic installation: Port 25 25 67 Protocol
SMTP (TCP) SMTP (TCP) DHCP (UDP)

From Application Server BDSSA server PXE client

To Mail Server SMTP server DHCP service

Notes SMTP For emailing scheduled reports and notifications PXE boot broadcasts a DHCP request that includes PXE info. By default, PXE Server binds to 67 UDP. Extended DHCP response to an initial extended DHCP request

68 69 80 80 161 162 443 445 1080 1433 1433 1433 1521 1521 1521 4011 4750

DHCP (UDP) TFTP (TCP/UDP) HTTP (TCP) HTTP (TCP) SNMP (UDP) SNMP (UDP) HTTPS (TCP) SMB (TCP) TCP MS-SQL (TCP) MS-SQL (TCP) MS-SQL (TCP) TNS (TCP) TNS (TCP) TNS (TCP) DHCP (UDP) RSCD (TCP)

DHCP PXE client HTTP client PXE client Application Server Application Server HTTPS client PXE client SOCKS client Application Server BDSSA server PXE Server Application Server BDSSA server PXE Server PXE client Application Server Advanced Repeater

PXE client TFTP Server BDSSA server PXE Server

HTTP; used for Linux PXE Servers SNMP SNMPTRAP

BDSSA server PXE Server SOCKS proxy SQL Server DB SQL Server DB SQL Server DB Oracle DB Oracle DB Oracle DB PXE Server RSCD Agent PXE discovery when co-located with DHCP Primary communication channel from Application Server to each managed host SMB; used for Windows PXE Servers SOCKS Proxy protocol

5282

HTTP (TCP)

Adv. File Server

Page 27

BMC BladeLogic Server Automation Best Practices for for Deployment and Configuration
Port 7717 Protocol
TCP

From File Server

To Adv. File Server

Notes Marimba publishing -- if the File Server and Adv. File Server are not co-located usually local traffic only usually local traffic only usually local traffic only usually local traffic only Cognos report BladeLogic SSO JMX listener -- usually local traffic only

7717 7717 7717 8080 9300 9640 9700 9701 9702 9831
9836 (base+36*)

TCP TCP TCP HTTP (TCP) TCP TCP JMX (TCP) TCP TCP TCP TCP

Transmitter Administrator Proxy Administrator Certificate Manager Adv. Repeater Cognos client BDSSA server BLASAdmin console Application Server Console Provisioning Client Application Server

Adv. File Server Adv. Repeater BMCCM Tuner Adv. Repeater BDSSA server BDSSA Auth. Server Application Server Launcher Launcher Launcher Application Server RMI Registry

SSL Provisioning (user guide p. 853, steps 7 and 9)

9838 (base+38*) 9840 (base+40*) 9840 (base+40*) 9841 (base+41*) 9842 (base+42*) 9850-9899 (MinPortMaxPort**)

TCP TCP TCP TCP TCP

Jconsole Application Server RCP (Client UI) RCP (Client UI) NSH, Application Server Application Server Auth. Service Auth. Service Application Server NSH Proxy

JMX listener for Application Server Authentication Service

TCP

Application Server

RMI communication ports

* Application Server ports are normally configured from a base port, with 9800 being the default base port. A second Application Server on the same host will typically have a base port of 9900, and so on. Arbitrary port assignments can be made in all cases. ** The MinPort-MaxPort range is configurable, with 9850-9899 being the default for a single Application Server.

Page 28

BMC BladeLogic Server Automation Best Practices for for Deployment and Configuration

Business runs on IT. IT runs on BMC Software. Business thrives when IT runs smarter, faster and stronger. Thats why the most demanding IT organizations in the world rely on BMC Software across distributed, mainframe, virtual and cloud environments. Recognized as the leader in Business Service Management, BMC offers a comprehensive approach and unified platform that helps IT organizations cut cost, reduce risk and drive business profit. For the four fiscal quarters ended September 30, 2010, BMC revenue was approximately $1.96 billion. Visit www.bmc.com for more information.
BMC, BMC Software, and the BMC Software logo are the exclusive properties of BMC Software, Inc., are registered with the U.S. Patent and Trademark Office, and may be registered or pending registration in other countries. All other BMC trademarks, service marks, and logos may be registered or pending registration in the U.S. or in other countries. All other trademarks or registered trademarks are the property of their respective owners. 2011 BMC Software, Inc. All rights reserved. AIX and IBM are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. Linux is the registered trademark of Linus Torvalds. Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners. UNIX is the registered trademark of The Open Group in the U.S. and other countries.

*195833*

Page 29