Documente Academic
Documente Profesional
Documente Cultură
This document describes the HugePages feature in the Linux kernel available
for 32- and 64-bit architectures. There has been some confusion among the
terms and uses related to HugePages. This document should clarify the
misconceptions about the feature.
SCOPE
Information in this document is useful for Linux system administrators and
Oracle database administrators working with system administrators.
This document covers information about HugePages concept that applies to
very large memory systems for 32- and 64-bits architectures including some
configuration information and references.
DETAILS
Introduction
HugePages is a feature integrated into the Linux kernel with release 2.6. This
feature basically provides the alternative to the 4K page size (16K for IA64)
providing bigger pages.
Regarding the HugePages, there are some other similar terms that are being
used like, hugetlb, hugetlbfs. Before proceeding into the details of
HugePages, see the definitions below:
Common Misconceptions
This section aims to give a general picture about memory access in virtual
memory systems and how pages are referenced.
When a single process works with a piece of memory, the pages that the
process uses are reference in a local page table for the specific process. The
entries in this table also contain references to the System-Wide Page Table
which actually has references to actual physical memory addresses. So
theoretically a user mode process (i.e. Oracle processes), follows its local
page table to access to the system page table and then can reference the
actual physical table virtually. As you can see below, it is also possible (and
very common to Oracle RDBMS due to SGA use) that two different O/S
processes can point to the same entry in the system-wide page table.
When HugePages are in the play, the usual page tables are employed. The
very basic difference is that the entries in both process page table and the
system page table has attributes about huge pages. So any page in a page
table can be a huge page or a regular page. The following diagram illustrates
4096K hugepages but the diagram would be the same for any huge page
size.
HugePage sizes vary from 2MB to 256MB based on kernel version and
HW architecture (See related section below.)
o TLB entries will cover a larger part of the address space when
use HugePages, there will be fewer TLB misses before the entire
or most of the SGA is mapped in the SGA
o Fewer TLB entries for the SGA also means more for other parts of
the address space
Eliminated page table lookup overhead: Since the pages are not
subject to replacement, page table lookups are not required.
HW Platform
The actual size of the HugePage on a specific system can be checked by:
$ grep Hugepagesize /proc/meminfo
The table below shows the sizes of HugePages on different configurations.
Note that these are general numbers taken from the most recent versions of
the kernels. For a specific kernel source package, you can check for the
HPAGE_SIZE macro value (based on HPAGE_SHIFT) for a different (more
recent) kernel source tree.
HW Platform
Source Code
Tree
Kernel
2.4
Kernel
2.6
i386
4 MB
4 MB *
2 MB
2 MB
ia64
256 MB
256 MB
ppc64/powerpc N/A **
16 MB
s390
N/A
1 MB
s390
N/A
N/A
* Some older packaging for the 2.6.5 kernel on SLES8 (like 2.6.5-7.97) can
have 2 MB Hugepagesize.
** Oracle RDBMS is also not certified in this configuration. See Document
341507.1
HugePages Reservation
The AMM and HugePages are not compatible. One needs to disable AMM on
11g to be able to use HugePages. See Document 749851.1 for further
information.
What if Not Enough HugePages Configured?
To avoid / help with such situations Bug 10153816 was filed to introduce a
database initialization parameter in 11.2.0.2 (use_large_pages) to help
manage which SGAs will use huge pages and potentially give warnings or not
start up at all if they cannot get those pages.
Parameters/Setup
Document 317055.1 How to Configure RHEL 3.0 32-bit for Very Large
Memory with ramfs and hugepages
Start instance(s)
Calculate hugetlb_pool using script from Note 401749.1
Shutdown instances
Set kernel parameter:
# sysctl -w vm.hugetlb_pool=<value from above>
and make sure that the parameter is persistent to reboots. e.g. On Asianux
1.0 by editing /etc/sysctl.conf adding/modifying as below:
vm.hugetlb_pool=<value from above>
5. Check available hugepages:
$ grep Huge /proc/meminfo
6. Restart instances
7. Check available hugepages:
Start instance(s)
Calculate nr_hugepages using script from Document 401749.1
Shutdown instances
Set kernel parameter:
# sysctl -w vm.nr_hugepages=<value from above>
and make sure that the parameter is persistent to reboots. e.g. On SLES9:
# chkconfig boot.sysctl on
5. Check available hugepages:
$ grep Huge /proc/meminfo
6. Restart instances
7. Check available hugepages:
$ grep Huge /proc/meminfo
Notes:
APPLIES TO:
Oracle Database - Enterprise Edition - Version 9.2.0.1 and later
Linux OS - Version Enterprise Linux 4.0 to Oracle Linux 6.0 with Unbreakable
Enterprise Kernel [2.6.32] [Release RHEL4 to OL6]
Linux x86-64
Oracle Linux
Red Hat Enterprise Linux (RHEL)
SUSE Linux Enterprise Server (SLES)
PURPOSE
This document aims to provide.
SCOPE
Information in this document is useful for Linux system administrators and
Oracle database administrators working with system administrators.
This document covers information about Linux HugePages for 64-bit
architectures. For more generic and uses on 32-bit and for references please
see Document 361323.1
The configuration steps provided here is primarily for Oracle Linux. Still the
same concepts and configurations should apply to other Linux distributions.
DETAILS
Introduction
higher than with regular pages. This reduces the number of times page
tables are walked to obtain physical address from a virtual address.
How to Configure
soft
hard
memlock
memlock
60397977
60397977
There is no harm in setting this value large than your SGA requirements.
The parameters will be set by default on:
Step 2: Re-logon to the Oracle product owner account (e.g. 'oracle') and
check the memlock limit
$ ulimit -l
60397977
Step 3: If you have Oracle Database 11g or later, the default database
created uses the Automatic Memory Management (AMM) feature which is
incompatible with HugePages. Disable AMM before proceeding. To disable,
set the initialization parameters MEMORY_TARGET and
MEMORY_MAX_TARGET to 0 (zero). Please see Document 749851.1 for
further information in case you encounter the error below:
ORA-00845: MEMORY_TARGET not supported on this system
Step 4: Make sure that all your database instances are up (including ASM
instances) as they would run on production. Use the
script hugepages_settings.sh in Document 401749.1 to calculate the
recommended value for the vm.nr_hugepages kernel parameter. e.g.:
$ ./hugepages_settings.sh
...
Recommended setting: vm.nr_hugepages = 1496
$
Note: You can also calculate a proper value for the parameter yourself but
that is not advised if you do not have extensive experience with HugePages
and concepts.
Step 5: Edit the file /etc/sysctl.conf and set the vm.nr_hugepages parameter
there:
...
vm.nr_hugepages = 1496
...
This will make the parameter to be set properly with each reboot.
Step 6: Stop all the database instances and reboot the server
(Although settings could have been done dynamically they would not be
effective to the extent we require before a reboot. The best practice is to do
a persistent system configuration and perform a reboot to complete the
configuration as presented through the steps above)
What If the Database / SGA Configurations Change?
The performed configuration is basically based on the RAM installed and
combined size of SGA of database instances you are running. Based on that
when:
After the system is rebooted, make sure that your database instances
(including the ASM instances) are started. Automatic startup via OS
configuration or CRS, or manual startup (whichever method you use) should
have been performed. Check the HugePages state from /proc/meminfo. e.g.:
HugePages_Rsvd:
HugePages_Surp:
446
0
The values in the output will vary. To make sure that the configuration is
valid, the HugePages_Freevalue should be smaller than HugePages_Total and
there should be some HugePages_Rsvd.HugePages_Rsvd counts free pages
that are reserved for use (requested for an SGA, but not touched/mapped
yet).
The sum of Hugepages_Free and HugePages_Rsvd may be smaller than your
total combined SGA as instances allocate pages dynamically and proactively
as needed.
Troubleshooting
Some of the common problems and how to troubleshoot them are listed in
the following table:
Symptom
Possible Cause
Troubleshooting
Action
System is running
out of memory or
swapping
Databases fail to
start
One of the
database fail to
start while another
is up
Cluster Ready
Services (CRS) fail
to start
Database started
successfully and
the performance is
slow