Running Oracle Database 10g or 11g on an HP-UX ccNUMA-based server

Updated for Oracle 11gR2 and HP-UX 11i v3

Technical white paper

Table of contents
Executive summary............................................................................................................................... 2 ccNUMA: an introduction ..................................................................................................................... 4 ccNUMA infrastructure ..................................................................................................................... 5 HP-UX 11i v3 and LORA mode ....................................................................................................... 6 Virtualization, CLM, and LORA mode .............................................................................................. 7 Oracles NUMA optimizations .............................................................................................................. 9 Enabling or disabling the Oracle NUMA optimizations ........................................................................ 9 Determining the ccNUMA configuration............................................................................................ 10 How the optimizations work ............................................................................................................ 11 Configuring the server and Oracle to match one another .................................................................... 11 A clear case for NUMA optimization: small vPar in large nPar ............................................................ 13 Dynamic reconfiguration considerations with ccNUMA optimizations ................................................... 14 Summary .......................................................................................................................................... 15 Appendix: Oracle versions, default NUMA enablement ......................................................................... 16 For more information .......................................................................................................................... 17

Executive summary
Many HP servers are based on a ccNUMA (cache-coherent Non-Uniform Memory Access) architecture; these servers provide optional features which allow programmers to optimize application performance by tailoring their application to the non-uniform nature of the underlying host server. Depending on the application and workload, by taking advantage of these features and reducing higher-latency memory references, modest performance gains can be achieved. Starting with release 10g, the Oracle database includes such NUMA-optimization features; these can be enabled or disabled at database startup time, and they are enabled by default on some Oracle versions, and disabled on others. To be effective, these optimizations require that the servers memory is allocated mainly to the individual locality domains (cell-local memory or socket-local memory) rather than to the system as a whole (interleaved memory); newer HP ccNUMA servers default to such a configuration, but older ccNUMA servers do not. It is very important for performance reasons that the setting of Oracles NUMA optimizations matches the configuration of the server: never rely on the default settings and configurations to produce an optimal result. Furthermore, dynamic resource reconfiguration will have significant consequences to any running NUMA-optimized Oracle Database instances. Because the optimizations are configured by Oracle only at the time the instance starts up, subsequent changes to the structure (number of locality domains) of the underlying host will, at best, result in the instance being optimized for the wrong structure (which will incur sub-optimal performance); at worst, they may cause the database instance to fail. Thus, if dynamic reconfiguration is to be undertaken, either: disable Oracles NUMA optimizations, or manage any dynamic reconfiguration very carefully, following the recommendations spelled out in a companion white paper, Dynamic server resource allocation with Oracle Database 10g or 11g on an HP-UX ccNUMA-based server
Table 1. Summary of rules governing Oracle NUMA optimization with dynamic system reconfiguration

Oracle NUMA optimization disabled or off (ensure adequate interleaved memory for Oracle SGA) enabled (ensure adequate cell-local memory in each domain for Oracle SGA)*

Dynamic Server Reconfiguration

No Dynamic Server Reconfiguration

OK

OK

OK with restrictions (see companion white paper) but generally not recommended.

OK

* (default on older HP servers is: 100% ILM, 0% CLM)

Applicable servers:: All ccNUMA-based HP-UX servers (all cell-based servers and all servers based on the Intel Itanium 9300 processor) which are running Oracle Database 10g or later. Target audience: Oracle DBAs and those who administer ccNUMA HP-UX systems which will host Oracle databases.

Support: This whitepaper makes recommendations based on HPs best knowledge of the stated configuration options at this time (June 2010). We do not and cannot imply Oracle support for any of these configuration options statements of Oracle support can only be made by Oracle.

Figure 1. Traditional Uniform-Memory-Access (UMA) design. All processors, and all processes (P1, P2, etc), have equal latency to their objects in any part of memory. The server is scaled up by connecting more processors to the bus (which eventually causes the bus to be a bottleneck and reduce scalability).

ccNUMA: an introduction
Cache-coherent non-uniform memory access (ccNUMA) is an architectural technique which has been used successfully to build large, multi-processor servers in a way that avoids the bus bottlenecks which characterize traditional uniform server design (see Figure 1). HPs first generation of ccNUMA servers are cell-based: they are comprised of multiple smaller cells of CPUs, memory, and I/O cards linked together by at least one low-latency cross-bar switch. These individual cells can be operated independently or in groups of two or more cells; each such operating unit is known as an nPar or hardware partition. When multiple cells are incorporated into the same hardware partition, processes running in any cell can access memory owned by any other cell (Figure 2). Accessing memory from a remote cell results in slightly greater latency than an access to memory in a process own cell. Since memory-access time is a component of CPU-busy time, an increase in memory latency equals an increase in CPU-busy time.
Table 2. HP servers based on ccNUMA architecture

Cell-based ccNUMA servers rp7410, rp7420, rp7440, rx7620, rx7640, rp8400, rp8420, rp8440, rx8620, rx8640, HP 9000 Superdome, HP Integrity Superdome

Itanium 9300-based ccNUMA servers rx2800 i2, BL860c i2, BL870c i2, BL890c i2, Superdome 2

In 2010, HP introduced new Integrity servers based on the latest Intel Itanium processor (the fourcore/eight-hyper-thread Itanium 9300 series). Like their predecessors, these servers are modular in nature: the blade-based two-socket BL860c i2 can be used by itself, doubled to form a four-socket server (BL870c i2), or quadrupled to form an eight-socket server (BL890c i2). A non-modular rackmounted version of the two-socket server (rx2800 i2) is also available. The Itanium 9300 is designed so that each processor socket has its own memory; all memory in these servers is allocated (more or less) evenly among the sockets. In a two-socket server, a process will have faster access time to objects in the memory on the same socket as the core where the process is running than it would to objects in the memory of the other socket. When multiple two-socket blades are lashed together to form larger servers, in addition to the latency differential between the sockets in the same blade, there are further increases in latency when accessing memory objects on the other blades which comprise the server. So while previous ccNUMA servers featured locality domains based on the cell concept, these newest ccNUMA servers locality domains are based on their socket layout. ccNUMA-based servers provide excellent performance and scalability without requiring that workloads be adapted in any way for the ccNUMA architecture. However, certain application workloads may realize small or even modest performance gains if they are made aware of the servers hierarchical memory layout and are able to adapt themselves to minimize interactions between the locality domains (the cells or sockets) that comprise the server. By restricting a process to a certain domain and assigning its memory objects to the memory of that same domain, a process will minimize memory latency. Processes which do large amounts of data manipulation in memory are thus candidates for ccNUMA optimization.

Figure 2. Non-Uniform Memory Access. Small UMA servers (cells) are connected together through one or more high-speed switches to form a larger server. Any process has access to any region of memory, but latency times are lower if the object is closer to the accessing process. In this example, P1s access to its object is faster than P2s, which is faster than P3s.

ccNUMA infrastructure
While the memory in a ccNUMA server is organized in a hierarchical manner, applications are traditionally designed with the assumption that all parts of memory are equal. Thus, some or all memory in a ccNUMA server can be designated as interleaved memory (ILM), which is designed to emulate, as much as possible, a uniform architecture. Interleaved memory is managed as a single unit across all locality domains in the server or partition; an object placed in ILM will be striped across all locality domains, and access times, on average, will be the same regardless of where (in which locality domain) the accessing process is located. The alternative to configuring memory as ILM is to configure it in its native ccNUMA state, which is called Cell-Local Memory (CLM), or sometimes Socket-Local Memory (SLM). CLM/SLM is configurable on a per-cell/per-socket basis; an object placed in CLM (SLM) in a cell or socket will be completely contained within that cell or socket. In general, one would only place objects in CLM if the applications which access those objects have been optimized for ccNUMA (i.e., if the applications processes can be localized to a single locality domain); for non-optimized applications, ILM would usually be the best choice. The memory in HP ccNUMA servers is configurable as ILM or CLM; this configuration is set at powerup and is fixed while the server is running. The server management interface can be used to view and modify the ILM/CLM configuration, with any changes taking effect at the next reboot. The default configuration depends on the particular server model: for Itanium 9300-based servers, the default is 87.5% CLM, but for all other servers, the default is 100% ILM. It is important to know the ILM/CLM configuration of your server.

A note about terminology: the earliest ccNUMA servers were all cell-based, so its not unusual to refer to the individual locality domains as cells, which is why we have Cell-Local Memory as opposed to Domain-Local Memory (though the term Socket-Level Memory is sometimes used). Today, we use the term locality domain (or LDOM, or sometimes just locality or domain) to mean cells or sockets any set of resources which have equal latency to a region of memory. Furthermore, we might refer specifically to a locality domain by its number: LDOM 0 through LDOM n.

Figure 3. Interleaved Memory (ILM) is spread across the Locality Domains (LDOMs); Cell-Local Memory (CLM) is contained within each LDOM. An object placed in ILM will thus be striped across the LDOMs; a process accessing it will find some accesses to be local (same LDOM), others remote. An object placed in CLM will be local for all processes in that same LDOM (but all accesses from remote LDOMs will have higher latency). In this example, P1 is accessing an object in ILM (with varying access times since one part of the object is in the same LDOM, but most of it is in remote LDOMs) and an object in the CLM in its own LDOM (where all accesses are local). When P2 accesses the object in LDOM 0, all accesses are remote.

HP-UX 11i v3 and LORA mode

HP-UX 11i v3 has been enhanced to provide tools and features which facilitate application optimization for the ccNUMA architecture. The most evident of these is a new system variable, LORA_MODE, which was introduced with HP-UX 11i v3 update 3 (September 2008). The term LORA comes from Locality-Optimized Resource Allocation, the science of optimizing for ccNUMA. LORA_MODE is intended to indicate whether the server is configured adequately to permit locality optimization (ccNUMA optimization) to be effective. A LORA_MODE value of 1 (one) indicates a sufficient configuration; currently, HP-UX will set this variable to 1 if 86-90% of the servers memory is configured as CLM, and if this CLM is evenly distributed across the locality domains. LORA_MODE cannot be changed directly, but a kernel tunable parameter (numa_mode) can be used to control LORA_MODE. The kernels default value for numa_mode is 0 (zero), which allows HP-UX to set LORA_MODE based solely on the hardware configuration. Setting numa_mode to 1 (one) will force LORA_MODE to 1, while setting numa_mode to -1 (minus one) will force LORA_MODE to zero.

If numa_mode is set to: 0 (default) 1 -1

Then LORA_MODE will be: Set by HP-UX to 1 if adequate CLM, 0 otherwise 1 0

The purpose of LORA_MODE is to provide a way of indicating to any optimized applications that the server has sufficient CLM to support those optimizations. (An application that is optimized for ccNUMA requires sufficient CLM in which to place its memory objects; without enough CLM, the optimizations would have no or perhaps even a detrimental effect on performance.) To determine the value of LORA_MODE, use the HP-UX getconf command: getconf LORA_MODE As will be discussed in more detail below, Oracles most recent database version, 11gR2, checks LORA_MODE before engaging its optimizations. An additional HP-UX 11i v3 parameter which affects ccNUMA behavior is numa_policy, which governs the allocation of memory (in CLM or ILM). The valid settings of numa_policy are: 0: (default): autosense the right policy based on the (LORA) mode in which HP-UX is operating 1: (default in LORA mode 1): honor requests from the application, otherwise place the object in the CLM of the domain where the process is most likely to run 2: override requests; always allocate in ILM if possible 3: honor requests by the application, otherwise allocate in the CLM of the closest domain, except for text/library objects, which will be placed in ILM if possible 4: (default for LORA mode 0): allocate non-LORA-intelligently (i.e., favor ILM for shared objects and CLM for private objects, but honor requests made by the application)

In short, if the application specifies the location of a newly created object, that location will generally be used unless numa_policy=2, or unless theres not enough CLM or ILM available, in which case the object will be placed in the next-best location. If the application does not specify the location of the object, it will default according to the setting of numa_policy. The HP-UX command kctune can be used to determine the current setting of either numa_mode or numa_policy: kctune numa_mode kctune numa_policy

Virtualization, CLM, and LORA mode

Most virtualized servers take on the ccNUMA characteristics of their constituent components, and HPUX, and any applications running on these servers, will act accordingly. An nPar or vPar which is comprised of multiple locality domains will exhibit the same ccNUMA behavior as a physical server constructed from the same hardware components; HP-UX 11i v3 will apply the same LORA_MODE tests, and applications will derive the same benefit from NUMA optimization. Just as its important to properly configure a physical server with sufficient CLM for application optimizations to be effective, it is important to properly configure a multi-domain vPar or nPar which will host an optimized application. To verify the amount of cell-local and interleaved memory configured in an nPar, use the HP-UX parstatus command as follows: parstatus w (to determine the nPartition-number of the system) parstatus V p nPartition-number When configuring a vPar, be aware that some of the memory you specify will be used for vPar management overhead, so be sure to configure your vPar with more memory than you require, keeping in mind that LORA_MODE will be set to 1 only if HP-UX sees that 86-90% of the vPars

memory is configured as CLM. When you start the vPar for the first time, verify your CLM/ILM amounts, and (for HP-UX 11i v3) the state of the LORA_MODE variable (using getconf as described above). The HP-UX machinfo command will tell you how much total memory and interleaved memory that HP-UX sees: machinfo m PSETs will also affect the ccNUMA characteristics seen by any applications running within them. A PSET that is wholly within a single locality domain will cause any applications running within it to behave as if they were running on a non-ccNUMA server. Likewise, if an nPar or vPar or does not span multiple domains, then it will be considered a non-ccNUMA server, even if that partition is part of a larger server that itself comprises multiple domains. HP Integrity Virtual Machines are the lone exception: VM guests will always appear as a single locality domain, regardless of the layout of the underlying physical infrastructure. One last fact about CLM and ILM with vPars: while CLM for each locality in the vPar is obviously contained within those localities, ILM will be interleaved across ALL the localities that constitute the server (or nPar) of which the vPar is a part. An example will help illustrate this fact: imagine an nPar of eight cells that contains a two-cell vPar (see Figure 4). You might expect that the ILM for this vPar would only span cells 0 and 1, but actually, it will span all eight cells so when a process in our vPar needs to access an ILM-based object, it will wind up accessing memory in cells which are not even part of our vPar. (And as you can see, some of the memory in cells 0 and 1 are allocated to ILM that will be available to objects belonging to other vPars, which means that we will have to share some of our memory bandwidth with processes from outside our own vPar.) This last fact may surprise you if you thought that by setting up a vPar, your computing activity would be completely isolated within the vPar, youve just learned that you were wrong activity on every vPar DOES affect performance on all vPars because of this ILM characteristic. The performance advantages of using NUMA optimization to reduce the use of ILM (or, at the very least, of forcing objects to be placed in CLM instead of ILM) should thus be very clear. We will discuss this issue and how Oracle is affected once weve discussed Oracles NUMA optimizations in general.

Figure 4. An example of an eight-cell nPar (or server) with a vPar consisting of cells 0 and 1. Note that while the vPars CLM is contained completely within cells 0 and 1, the vPars ILM is a portion of the overall ILM allocated to the entire nPar/server - it is interleaved across all the cells.

Oracles NUMA optimizations

To exploit the characteristics of ccNUMA to improve performance, Oracle Database (version 10g and later) has been enhanced to accommodate Oracle processes and their corresponding memory objects within the same domains. Such enhanced behaviors are known as Oracles NUMA optimizations (deliberately without the cc), and are engaged only on multi-domain servers and only when they are enabled via the appropriate Oracle initialization parameter. Each time a NUMA-optimized Oracle database instance is started, it will modify its configuration in order to best conform to the ccNUMA characteristics of the host under which it is running.

Enabling or disabling the Oracle NUMA optimizations

Oracles NUMA optimizations are controlled through an initialization parameter which is only checked at instance startup time. The parameter can be changed at any time but it will have no effect until the next time the database instance is started. The name of this parameter depends on the version of Oracle, and the default state of the parameter depends on the specific release and patch level (which is why HP recommends that this parameter always be explicitly set!). (The specific default behavior of each Oracle version is included as an appendix to this paper.) Oracle versions 10gR2 and 11gR1use a hidden (so-called underbar) parameter to control the ccNUMA optimizations: _enable_NUMA_optimization. Oracle typically does not support these underbar parameters and this one is no exception; however, certain Oracle documentation does refer to changing this parameter in order to enable or disable the optimizations.
Note The use of underbar parameters to control certain Oracle behaviors is unsupported by Oracle. Furthermore, since such parameters are unsupported and undocumented, they may change from one release of Oracle to the next. We therefore recommend that you consult your Oracle support resources before making any such changes to a supported Oracle database instance.

Oracle version 11gR2 uses a different underbar parameter: _enable_NUMA_support. This parameter is FULLY supported by Oracle (even though it starts with an underbar!). NOTE: when upgrading to 11gR2 from a previous release of Oracle, make sure to remove any references to the old parameter, _enable_NUMA_optimization. This parameter is unfortunately not ignored by Oracle 11gR2 it still affects some aspects of the optimizations and should thus be avoided. When a NUMA-optimized Oracle 11gR2 instance is started up, a message indicating the use of NUMA optimizations (NUMA system found and support enabled) is displayed on the console and in the instances alert log. (No such indication is given for older Oracle versions.) A complete description of the particular NUMA behaviors associated with each version of Oracle Database is included as Table 3. For Oracle Database version 10.2.0.4, the optimizations are not implemented correctly and should never be used: __enable_NUMA_optimization should always be set to false (and the server should thus be configured ILM-heavy) for Oracle 10.2.0.4.

Table 3. Oracle NUMA behavior. Binding of none indicates that Oracle does not restrict the location of the process(es).Parallel Query Slaves are created and bound one per LDOM in a round-robin (RR) fashion when in NUMA mode. With NUMA optimizations off, the location of the SGA (including the DB cache) is not specified by Oracle, so it defaults to the location implied by the HP-UX parameter numa_policy. In addition, the optimizations will not be engaged if Oracle detects only one locality domain (cell or socket) when the database is started. See HP-UX 11i v3 and LORA mode, above, for definition and default behavior of numa_policy.

ing

lo c a ti on

sa re

at i on

bin d

ve r sio n

ara me te

ery

ffe rb bu og oth

ind
none none none

ing er bin din g

LO RA _M OD

z at ion

loc

in g ind wr b bw r

NU MA p

_c a che

GA

fixe d_S

Ora c le

tim i

alle l
none

qu par
none none

_eNo=false 0, 1, or n/a disabled default _eNo=true 0, 1, or n/a _eNs=false _eNs=true 0 or 1 0 1

default

lcpu / 8 none # LDOMs each LDOM

none

engaged CLM (even) default disabled default unengaged default default default

LDOM RR LDOM 1 LDOM 0 none none

lcpu / 8 none # LDOMs none # LDOMs each LDOM

engaged CLM (even) default

LDOM RR LDOM 1 LDOM 0

_eNo = _enable_NUMA_optimization (10gR2 and 11gR1) _eNs = _enable_NUMA_support (11gR2) depends on numa_policy: either ILM, or CLM of a single LDOM lcpu = logical CPU count {= # cores, times 2 if hyperthreading enabled} *10.2.0.4 only: optimizations are broken. _eNo should ALWAYS be set to false

Determining the ccNUMA configuration

At instance startup, if the initialization parameter governing Oracles NUMA optimizations has been enabled, Oracle performs a check to determine the ccNUMA state of the host server (which could be a physical server, nPar, vPar, Integrity Virtual Machine, or other partition). The nature of this check depends on the release of Oracle: For Oracle 11gR2, the servers LORA_MODE variable is checked if it is zero, the optimizations will be disabled for that execution of the instance. If LORA_MODE is one, Oracle still has other checks to perform: For all Oracle versions, Oracle counts the number of active locality domains (i.e., domains containing one or more CPUs) present on the host server. If Oracle detects only one locality domain, NUMA optimizations will be disabled for that execution of the instance; if multiple domains are detected, the optimizations will be enabled.

Thus, while all versions of Oracle do check to make sure that the server has more than one locality domain available before engaging optimized behavior, only version 11gR2 checks whether the server is configured with adequate CLM (LORA_MODE is 1) to render the optimizations effective. If the server has no CLM and the optimizations are enabled anyway, Oracle will function normally but the optimizations will be useless see below. Recall that the default configuration for many HP servers is zero CLM, so it is important to explicitly configure your server with enough CLM if you plan to use Oracles NUMA optimizations.

10

lgw r, l

#d

op

db

db

How the optimizations work

When Oracles NUMA optimizations are engaged, the following actions are taken as the instance is started: The shared data area (System Global Area, or SGA) is split into multiple pools of memory, one pool for each locality domain in the server. (The fixed SGA is not included see below.) Multiple database writer process are launched, one in each locality domain, in order to more efficiently write out the dirty database buffers contained in that domains SGA pool. The log-writer process is bound to locality domain 1, and all log buffers are allocated from CLM in that domain. Background processes that are instantiated as multiple copies (e.g., the parallel processing processes ora_pNNN, and query processing processes ora_qNNN) are distributed across the locality domains in a round-robin fashion. All other Oracle background processes (pmon, smon, ckpt, etc.) are bound to the locality domain in which they are initially placed by HP-UX to ensure that their memory accesses remain local as much as possible. A small portion of the SGA, known as the fixed SGA, is not explicitly assigned by Oracle to any particular location. It is thus governed by numa_policy when LORA_MODE=0 (or when numa_policy is set to favor ILM), it will go into ILM, and when LORA_MODE=1 (or when numa_policy is set to favor CLM), it will be placed in the CLM of a single domain, usually LDOM 0. (Note that in previous versions of this white paper we reported that the fixed SGA would go into ILM when the NUMA optimizations are engaged such was our understanding of Oracles intent, but empirical evidence proves otherwise.) Thus, when LORA_MODE=1 (unless numa_policy is explicitly set to 2, forcing everything into ILM), the only ILM that will be used by a NUMA-optimized Oracle instance will be its shared text i.e., the Oracle code itself, which is currently on the order of 350 MB.
Note Oracles shadow client processes are not NUMA optimized, because the TNS listener process, tnslsnr (which spawns the Oracle shadow client processes for incoming database connections), is not ccNUMA-aware. Without explicit customization of the listener process, allocation of shadow processes (and their private objects) will follow the same algorithm thats long been used for shadow process distribution. As a best practice, we continue to recommend the use of mpsched P LL p <lsnr-pid> as a means of enforcing a least-loaded scheduling policy on the spawned shadow processes. Such enforcement ensures that the shadow processes will be distributed evenly across the available CPU resources on the host server and that their allocation of private data will be optimized for their resident LDOM.

Since the NUMA configuration is determined only at instance startup, Oracle cannot respond to dynamic system reconfiguration. See Dynamic reconfiguration considerations with ccNUMA optimizations, below, for further discussion.

Configuring the server and Oracle to match one another

It is important to be sure that Oracle and its host server are configured compatibly, either for NUMAoptimized behavior or for non-optimized behavior. The default configuration for some server models is perfectly mis-matched with the default Oracle configuration of certain Oracle versions (older servers default to 100% ILM, while NUMA optimizations are enabled by default on some older Oracle versions; Itanium 9300-based servers default to 87.5% CLM, but the NUMA optimizations are

11

disabled by default for the latest Oracle versions). We strongly recommend that the defaults never be taken (or at least that the configurations be explicitly examined to make sure they are well aligned). For NUMA-optimized behavior When Oracles NUMA optimizations are desired, it is critical to make sure that sufficient CLM is allocated on the system. The default configuration for HP cell-based servers 0% cell-local memory - should be changed on any multi-cell server that will host a NUMA-optimized Oracle database. To take advantage of Oracles NUMA optimizations, configure the system with both cell-local memory and interleaved memory (ILM) Oracle uses a tiny amount of ILM even when the NUMA optimizations are engaged (and HP-UX requires some ILM as well). When insufficient cell-local memory is available in a locality domain, the Oracle memory structures are created in interleaved memory (across all domains) and/or CLM in other localities, where they will still be accessible by Oracles processes, but where most memory accesses will be non-local. So while the Oracle instance will function completely normally, it will be expending the overhead required to eliminate inter-domain accesses, but it cannot be successful: the overhead will be wasted, and a performance penalty will result. How much cell-local memory should be configured on a system? The answer depends largely on two factors: HP-UX requirements. Starting with HP-UX version 11.31 (11i v3), the operating system itself is optimized to use cell-local rather than interleaved memory, so it is important to follow HP-UX standard recommendations for configuring cell-local memory. If you are upgrading from an earlier version of HP-UX, be sure to take into account the higher cell-local memory recommendations for 11.31. Combined memory requirements of all Oracle instances. As noted above, Oracle instances with NUMA optimizations enabled will request CLM in the amount of the SGA (System Global Area) for the instance. A given server should be configured with CLM at least as large as the sum of all NUMA-enabled Oracle instances that will be run on that server. Additional information about CLM and ILM memory recommendations can be found in the white paper Locality-Optimized Resource Alignment (see For more information section, below). Typically, HP recommends 87.5% (7/8) of all memory be configured as CLM, to account for both OS and application needs; this is the default value for the new Itanium 9300-based Integrity servers. Moreover, the 87.5% setting is the level at which HP-UX 11.31 will set the LORA_MODE configuration variable to a value of 1, which will allow Oracle 11gR2 to engage its NUMA optimizations. In general, when setting up partitions (nPars, vPars, PSETs) which will host an Oracle database, include the minimum number of domains required to satisfy Oracles resource requirements, and keep the domains relatively well balanced (equalize memory and processors). Be sure to allocate CLM and ILM per the instructions above. When configuring nPars/vPars, consider the physical location of the I/O cards used to access that partitions devices: it would make sense to configure your partition with dedicated cores from the locality domain(s) associated with the I/O bay(s) containing those cards, in order to avoid cross-locality accesses when processing I/O interrupts. For non-NUMA-optimized behavior If your Oracle instance will be run with its NUMA optimizations disabled, Oracle will not make use of CLM at all, and will need sufficient interleaved memory (ILM) for all its data structures. In this case, be sure to configure your system to be ILM-heavy, and, if you are running an Oracle version prior to 11gR2, set _enable_NUMA_optimization to false to ensure that the optimizations are disabled. Oracle 11gR2 will properly detect ILM-heavy configurations (by detecting LORA_MODE equal to 0)

12

and automatically disable NUMA optimizations regardless of the value of the _enable_NUMA_support parameter. When an unoptimized Oracle instance requests space for its SGA, the space will be allocated according to the setting of numa_policy. If numa_policy is set to favor ILM (the default for LORA_MODE 0) but the system doesnt have enough ILM, the space will be allocated from the celllocal memory of the first LDOM that has it available. Likewise, if numa_policy is set to favor CLM (the default for LORA_MODE 1), the SGA will be placed in the CLM of a single LDOM. In either case, this will result in an imbalance in the available memory distribution across the LDOMs, and HP-UX will favor the remaining LDOMs whenever new processes are created. Thus, the SGA will be in one LDOM, while most of Oracles processes will be in the other LDOMs, practically guaranteeing that most memory accesses will be non-local. Clearly, when running Oracle with its NUMA optimizations disabled, its important to make sure that theres plenty of ILM (and that numa_policy is set to favor ILM). Configuring your server and your Oracle instance: summary While a CLM/ILM configuration that does not match the NUMA optimization state of the Oracle instance will not impede Oracles proper operation, it will result in sub-optimal performance. This implies a clear best practice when deploying Oracle instances on HP ccNUMA servers: make sure that the systems configuration and Oracles configuration match one another. Configure BOTH the server and Oracle for ccNUMA (lots of CLM; enable Oracles optimizations) or configure them both to operate without the optimizations (lots of ILM; disable Oracles optimizations). The best practice for Oracle Database 11gR2 is somewhat simpler: set _enable_NUMA_support to TRUE and then the instance will properly enable or disable the optimizations based on HP-UXs LORA_MODE setting. (As previously mentioned, do not assume that the default settings are optimal!)

A clear case for NUMA optimization: small vPar in large nPar

Weve already discussed the situation (see Figure 4 and the paragraphs above it) of the small vPar carved out of a large nPar; now lets consider the specific case of using such a vPar as an Oracle Database server. Recall that our example vPar consists of two of the eight cells which comprise the nPar; its got a fair amount of CLM in each cell, and a certain amount of ILM which is interleaved across all eight cells. Assume that the remaining six cells are allocated to one or more other vPars which will be running their own workloads. First, lets consider a non-NUMA-optimized Oracle Database instance in our vPar (see Figure 5). All of Oracles SGA will be placed in ILM (and will thus be interleaved across all eight cells) an Oracle process in either cell of our vPar will need to perform, on average, seven memory accesses from other cells for every one memory access within the local cell. Not only is this bad for the performance of our database instance, but all those inter-cell memory accesses will affect the workloads running in whatever vPars are assigned those other cells! Furthermore and likewise, heavy workload activity in those other vPars can and will affect the performance of our database server.

13

Figure 5. An eight-cell nPar (or server) with a vPar, consisting of cells 0 and 1, running a non-NUMA-optimized Oracle instance. The SGA will be located in ILM, interleaved across all eight cells (including six cells that are otherwise not part of the vPar).

Performance of both our database and of the other applications in other cells would clearly benefit from turning our Oracle Databases NUMA optimizations on, because Oracles SGA will be placed in the CLM of the two cells in our vPar.

Dynamic reconfiguration considerations with ccNUMA optimizations

HP server virtualization capabilities (nPars, vPars, etc) allow you to define virtualized servers as a subset of available physical resources, and to re-define those servers dynamically; CPUs and, in some cases, memory can be added or deleted without shutting down the operating system. When an Oracle instance is open and running on a virtualized server with NUMA optimizations disabled, the operating system manages the locality of all Oracles processes and memory allocations. Thus, dynamic changes to the locality domains of the underlying host are transparent to a non-NUMAoptimized Oracle instance. HP-UX takes care of any necessary process or memory migration, and the overall number of inter-domain references may be better, worse, or the same after a dynamic change. But for a NUMA-optimized Oracle instance, dynamic reconfiguration of the underlying host can have a significant impact. In general, dynamic reconfiguration of a server or partition that hosts a NUMAoptimized Oracle instance will increase the likelihood of processes migrating to remote localities, thus reducing the effectiveness of Oracles NUMA optimizations. Furthermore, dynamic removal of resources can cause an Oracle instance to crash under certain circumstances. Therefore, employing Oracles NUMA optimizations on servers/partitions which will be dynamically reconfigured is not recommended, even though it is possible under some conditions. If dynamic reconfiguration is desirable, the recommendation is to disable Oracles NUMA optimizations and let HP-UX handle process locality. Nevertheless, dynamic reconfiguration IS supported for NUMA-optimized Oracle databases, and it can be done safely (without risk of Oracle database crashes) if certain rules are followed. These rules, and the reasons for them, are discussed fully in a companion white paper, Dynamic server resource allocation with Oracle Database 10g or 11g on an HP-UX ccNUMA-based server. If you are considering the use of dynamic resource allocation with a NUMA-optimized Oracle database, it is critical to read and understand the contents of that white paper.

14

Summary
Oracles NUMA enhancements can improve performance when running database instances on HP ccNUMA servers. Since the memory configuration of the underlying server is critical to the performance of an Oracle instance, care must be exercised to ensure that the memory configuration matches the NUMA mode of the instance (sufficient CLM must be configured for a NUMA-optimized instance, whereas a non-NUMA instance requires sufficient ILM). Make it a point to override the defaults (if necessary) and match the server configuration with the Oracle configuration: decide whether you wish your Oracle instance to run with its NUMA optimizations on or off, then configure BOTH Oracle AND your server accordingly!

15

Appendix: Oracle versions, default NUMA enablement

The initialization parameter which governs the enablement of Oracles NUMA parameters, and its default setting, depends on the Oracle release, version, and certain patches. Below is a complete list. Because of the confusion surrounding the default optimization state, HP highly recommends that the appropriate initialization parameter ALWAYS be set explicitly. In all cases, the optimizations will not be used if only one domain (cell or socket) is visible to Oracle when the database is started. 10gR2 was the first to support NUMA optimizations on HP-UX. The optimizations are controlled by the (unsupported) _enable_NUMA_optimization parameter 10.2.0.3 : optimizations enabled by default 10.2.0.4 : optimizations enabled by default BUT DO NOT WORK (bug 9668940). Optimizations should be explicitly disabled. 10.2.0.5 (future) : optimizations off by default; 9668940 will be fixed. Oracle recommends patch 8199533 to switch optimizations off by default for 10.2.0.3 and 10.2.0.4.

11gR1 (11.1.0.6 and 11.1.0.7) : optimizations (_enable_NUMA_optimization) enabled by default but Oracle recommends patch 8199533 to switch optimizations off by default 11gR2: optimizations controlled by the (supported) parameter _enable_NUMA_support (but if the underlying hosts LORA_MODE parameter is set to 0, the optimizations will not be used) 11.2.0.1 : optimizations disabled by default When upgrading to 11gR2 from an earlier version, make sure to DELETE any references to the obsolete parameter _enable_NUMA_optimization to avoid potential issues.

16

For more information

For an excellent discussion of the art of tuning applications to optimize ccNUMA performance, see the white paper Locality-Optimized Resource Alignment at http://docs.hp.com/en/14655/ENWLORA-TW.pdf. For best practices and considerations regarding the use of dynamic resource configuration with Oracle Database, see the white paper Dynamic server resource allocation with Oracle Database 10g or 11g on an HP-UX ccNUMA-based server. For specific rules and the latest information on Oracle support of dynamic reconfiguration and/or ccNUMA optimizations, see the note entitled Oracle Database ccNUMA support and dynamic partitioning on HP-UX published on Oracles MySupport site (http://metalink.oracle.com; search for Document ID 761065.1. Note: this site has membership requirements).

To help us improve our documents, please provide feedback at http://h20219.www2.hp.com/ActiveAnswers/us/en/solutions/technical_tools_feedback.html.

Copyright 2009 2010 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein. Oracle is a U.S. registered trademarks of Oracle Corporation. Intel and Itanium are trademarks of Intel Corporation in the U.S. and other countries. 4AA2-4194ENW, Created January 2009; Updated September 2010, Rev. #1