Documente Academic
Documente Profesional
Documente Cultură
06 May 2008
Learn about shared library mechanisms and memory footprints on IBM® AIX®. This
article is essential for developers writing server code or administrators managing
production AIX systems. It offers developers and administrators commands and
techniques, and gives the understanding necessary to analyze memory requirements
of server processes on AIX. It also helps developers and administrators avoid
resource shortages that can't be identified with other standard runtime analysis tools
such as ps or topas. The article is intended for systems administrators or developers
of native applications on AIX.
Introduction
This article examines how shared libraries occupy memory on 32-bit AIX 5L™ (5.3),
demonstrating the following commands:
• ps
• svmon
• slibclean
• procldd
• procmap
• genkld
• genld
The article discusses the virtual address space of processes, as well as the kernel
shared-library segment, how to examine them, and how to interpret the output of the
various diagnostic utilities mentioned above. The article also discusses how to
diagnose situations where the kernel shared segment is full and possible
approaches to resolving that situation.
In the examples throughout, we happen to use the processes from the software
product Business Objects Enterprise Xir2®. This is arbitrary, as the concepts will
apply to all processes running on AIX 5L.
Review
Just so we are all in the same mindframe, let's review a little on 32-bit architecture.
In doing so, I'll resort to the employ of the most useful 'bc' command-line calculator.
In a 32-bit processor, the registers are capable of holding 2^32 possible values,
$ bc
2^32
4294967296
obase=16
2^32
100000000
That is a 4 gigabyte range. This means a program running on the system is able to
access any function or data address in the range of 0 and 2^32 - 1.
$ bc
2^32 - 1
FFFFFFFF
obase=10
2^32 - 1
4294967295
Now, as you know, any operating system has potentially hundreds of programs
running at the same time. Even though each one of them is capable of accessing a
4GB range of memory, it doesn't mean that they each get their own 4GB allotment of
physical RAM. That would be impractical. Rather, the OS implements a
sophisticated scheme of swapping code and data between a moderate amount of
physical RAM and areas of the file system designated as swap (or paging) space.
Moreover, even though each process is capable of accessing 4GB of memory
space, many don't even use most of it. So the OS only loads or swaps the required
amount of code and data for each particular process.
This mechanism is often referred to as virtual memory and virtual address spaces.
When an executable file is run, the Virtual Memory Manager of the OS looks at the
code and data that comprise the file, and decides what parts it will load into RAM, or
load into swap, or reference from the file system. At the same time, it establishes
some structure to map the physical locations to virtual locations in the 4GB range.
This 4GB range represents the process' maximum theoretical extent and (together
sometimes with the VMM's structures that represent it), is known as the virtual
address space of the process.
On AIX, the 4GB virtual address space is divided into sixteen 256-megabyte
segments. The segments have predetermined functions, some of which are
described below:
To demonstrate this, take a running AIX machine and see which shared libraries are
presently loaded:
> su
# genkld
Text address Size File
d1539fe0 1a011 /usr/lib/libcurses.a[shr.o]
d122f100 36732 /usr/lib/libptools.a[shr.o]
d1266080 297de /usr/lib/libtrace.a[shr.o]
d020c000 5f43 /usr/lib/nls/loc/iconv/ISO8859-1_UCS-2
d7545000 161ff /usr/java14/jre/bin/libnet.a
d7531000 135e2 /usr/java14/jre/bin/libzip.a
.... [ lots more libs ] ....
d1297108 3a99 /opt/rational/clearcase/shlib/libatriastats_svr.a
[atriastats_svr-shr.o]
d1bfa100 2bcdf /opt/rational/clearcase/shlib/libatriacm.a[atriacm-shr.o]
d1bbf100 2cf3c /opt/rational/clearcase/shlib/libatriaadm.a[atriaadm-shr.o]
.... [ lots more libs ] ....
d01ca0f8 17b6 /usr/lib/libpthreads_compat.a[shr.o]
d10ff000 30b78 /usr/lib/libpthreads.a[shr.o]
d00f0100 1fd2f /usr/lib/libC.a[shr.o]
d01293e0 25570 /usr/lib/libC.a[shrcore.o]
d01108a0 18448 /usr/lib/libC.a[ansicore_32.o]
.... [ lots more libs ] ....
d04a2100 fdb4b /usr/lib/libX11.a[shr4.o]
d0049000 365c4 /usr/lib/libpthreads.a[shr_xpg5.o]
d0045000 3c52 /usr/lib/libpthreads.a[shr_comm.o]
d05bb100 5058 /usr/lib/libIM.a[shr.o]
d05a7100 139c1 /usr/lib/libiconv.a[shr4.o]
d0094100 114a2 /usr/lib/libcfg.a[shr.o]
d0081100 125ea /usr/lib/libodm.a[shr.o]
d00800f8 846 /usr/lib/libcrypt.a[shr.o]
d022d660 25152d /usr/lib/libc.a[shr.o]
As an interesting observation, we can see on this machine right away Clearcase and
Java™ are running. Let's take any one of these common libraries, say,
libpthreads.a. Browse the library and see which functions it implements:
Hmm, that was cool. Now let's see which currently running processes have it loaded
currently on the system:
Cool! Now let's get the same thing, but eliminate the redundancies:
dtgreet
dtlogin
dtsession
dtterm
dtwm
java
portmap
radiusd
rdesktop
rpc.mountd
rpc.statd
rpc.ttdbserver
ttsession
xmwlm
There, now we have a nice, discrete list of binaries that are currently executing and
all load libpthreads.a. Note that there are many more processes on this system
than this at this time:
# ps -e | wc -l
85
# ps -e | grep java
34648 - 4:13 java
#
# procmap 34648 | grep libpthreads.a
d0049000 217K read/exec /usr/lib/libpthreads.a[shr_xpg5.o]
f03e6000 16K read/write /usr/lib/libpthreads.a[shr_xpg5.o]
d0045000 15K read/exec /usr/lib/libpthreads.a[shr_comm.o]
f03a3000 265K read/write /usr/lib/libpthreads.a[shr_comm.o]
#
# ps -e | grep automountd
15222 - 1:00 automountd
25844 - 0:00 automountd
#
# procmap 15222 | grep libpthreads.a
d0049000 217K read/exec /usr/lib/libpthreads.a[shr_xpg5.o]
f03e6000 16K read/write /usr/lib/libpthreads.a[shr_xpg5.o]
d0045000 15K read/exec /usr/lib/libpthreads.a[shr_comm.o]
f03a3000 265K read/write /usr/lib/libpthreads.a[shr_comm.o]
d10ff000 194K read/exec /usr/lib/libpthreads.a[shr.o]
f0154000 20K read/write /usr/lib/libpthreads.a[shr.o]
#
# ps -e | grep portmap
12696 - 0:06 portmap
34446 - 0:00 portmap
#
# procmap 12696 | grep libpthreads.a
d0045000 15K read/exec /usr/lib/libpthreads.a[shr_comm.o]
f03a3000 265K read/write /usr/lib/libpthreads.a[shr_comm.o]
d10ff000 194K read/exec /usr/lib/libpthreads.a[shr.o]
f0154000 20K read/write /usr/lib/libpthreads.a[shr.o]
#
# ps -e | grep dtlogin
6208 - 0:00 dtlogin
6478 - 2:07 dtlogin
20428 - 0:00 dtlogin
#
# procmap 20428 | grep libpthreads.a
Notice that each process loads it at the same address each time. Don't be confused
by the constituent listings for the .o's in the library. On AIX, you can share archive
libraries (.a files, customarily) as well as dynamic shared libraries (.so files,
customarily). The purpose of this is to be able to bind symbols at link time, just like
traditional archive linking, yet not require the constituent object (.o file in the archive)
be copied into the final binary image. No dynamic (or runtime) symbol resolution is
performed, however, as is the case with dynamic shared libraries (.so/.sl files).
Also note libpthreads.a code sections, those marked read/exec, are loaded into
segment 0xd. That segment, as mentioned above, is designated on AIX as the
segment for shared library code. That is to say, the kernel loads the shareable
segments of this shared library into an area that is shared by all processes running
on the same kernel.
You might notice that the data sections are also loaded to the same segment: the
shared library segment 0xf. That doesn't mean, however, that each process is also
sharing the data section of libpthreads.a. Loosely defined, such an arrangement
wouldn't work, as different processes would need to maintain different data values at
different times. Segment 0xf is distinct for each process using libpthreads.a,
even though the virtual memory address is the same.
The svmon command can show us the segment IDs in the Virtual Memory Manager
(Vsid) for processes. We'll see the shared-library code segments all have the same
Vsid, while the shared-library data segments all have distinct Vsids. The Esid,
meaning Effective Segment ID, is the segment ID within the scope of the process's
address space (just terminology; don't let it confuse you).
# svmon -P 17314
-------------------------------------------------------------------------------
Pid Command Inuse Pin Pgsp Virtual 64-bit Mthrd 16MB
17314 dtexec 20245 9479 12 20292 N N N
Vsid Esid Type Description PSize Inuse Pin Pgsp Virtual
0 0 work kernel segment s 14361 9477 0 14361
6c01b d work shared library text s 5739 0 9 5786
19be6 f work shared library data s 83 0 1 87
21068 2 work process private s 56 2 2 58
18726 1 pers code,/dev/hd2:65814 s 5 0 - -
40c1 - pers /dev/hd4:2 s 1 0 - -
#
# svmon -P 20428
-------------------------------------------------------------------------------
Pid Command Inuse Pin Pgsp Virtual 64-bit Mthrd 16MB
20428 dtlogin 20248 9479 23 20278 N N N
Vsid Esid Type Description PSize Inuse Pin Pgsp Virtual
# bc
ibase=16
E0000000-D0000000
268435456
ibase=A
268435456/(1024^2)
256
That looks good. Like stated above, each segment is 256MB. Ok, now let's see how
much is currently being used.
$ echo "ibase=16; $(genkld | egrep ^\ \{8\} | awk '{print $2}' | tr '[a-f]' '[A-F]' \
| tr '\n' '+' ) 0" | bc
39798104
$
$ bc <<EOF
> 39798104/(1024^2)
> EOF
37
That is saying that there is 37MB currently being used. Let's start up XIr2, and
compare:
$ echo "ibase=16; $(genkld | egrep ^\ \{8\} | awk '{print $2}' | tr '[a-f]' '[A-F]' \
| tr '\n' '+' ) 0" | bc
266069692
$
$ bc <<EOF
> 266069692/(1024^2)
> EOF
253
Now there is 253MB being used. That is very close to the limit of 256MB. Let's pick a
random process, like WIReportServer, and see how many shared libraries made it
into shared space, and how many had to be mapped privately. Since we know the
shared segment begins at address 0xd000000, we can filter that out of the output
from procmap. Remember, only code sections are mapped to segment 0xd, so we'll
just look for the lines that are read/exec:
It looks like the above four libraries couldn't be mapped into the shared segment.
Consequently, they were mapped to the private segment 0x3, which is used for any
general memory allocated by a call to the mmap() routine.
There are a few conditions that force a shared library to be mapped privately on
32-bit AIX:
The instance for account 'jbrown' was started first, and the instance for account
'sj1xir2a' was started second. If we were to do something obscure and risky like
setting at the appropriate place in our bobje/setup/env.sh file,
LIBPATH=~jbrown/vanpgaix40/bobje/enterprise115/aix_rs6000:$LIBPATH
before starting the second instance, we would see the footprints normalized, (I
switch to the process boe_fcprocd, as I couldn't get WIReportServer to start for this
LIBPATH test).
And we see procmap shows us the files are loaded from ~jbrown as expected:
53470 : /crystal/sj1xir2a/xir2_r/bobje/enterprise115/aix_rs6000/boe_fcprocd
-name vanpg
10000000 10907K read/exec boe_fcprocd
3000079c 1399K read/write boe_fcprocd
d42c9000 1098K read/exec
/home7/jbrown/vanpgaix40/bobje/enterprise115/aix_rs6000/libcrypto.so
33e34160 167K read/write
/home7/jbrown/vanpgaix40/bobje/enterprise115/aix_rs6000/libcrypto.so
33acc000 3133K read/exec
/home7/jbrown/vanpgaix40/bobje/enterprise115/aix_rs6000/libcpi18nloc.so
33ddc697 349K read/write
/home7/jbrown/vanpgaix40/bobje/enterprise115/aix_rs6000/libcpi18nloc.so
Clean up
Once applications are shut down, shared libraries may still reside in the shared
segment 0xd. In such case, you can use the utility 'slibclean' to unload any shared
libraries that are no longer referenced. The utility requires no arguments:
slibclean
There is also the utility genld, which when passed the -l option, can show you output
like procmap, but for all existing process on the system,
genld -l
Sometimes, after running slibclean, you may still be prohibited from copying a
shared library. For example:
$ cp /build/dev/bin/release/libc3_calc.so /runtime/app/lib/
cp: /runtime/app/lib/libc3_calc.so: Text file busy
You may have run slibclean already, and running 'genld -l' doesn't show any process
having this library loaded. Yet the system still has this file protected. You can
overcome this limitation by first deleting the shared library in the target location, and
then copying the new shared library:
$ rm /runtime/app/lib/libc3_calc.so
$ cp /build/dev/bin/release/libc3_calc.so /runtime/app/lib/
Summary
I hope you've been able to see in more detail how shared libraries occupy memory
and the utilities used to examine them. With this you'll be better able to assess the
sizing requirements for your applications and analyze the constituents of memory
footprints for processes running on AIX systems.
Resources
Learn
• Performance Management Guide: Using Shared Memory has some general
notes on segments in 32-bit AIX as well as use of EXTSHM, shmat() and
mmap() subroutines.
• When Segments Collide ((Punit Shah and Sumit Chawla, developerWorks, July
2004) offers a good analysis of virtual address space, architecture thereof in
AIX and runtime use demonstrated by the Java VM.
• BIN DIRECTORY FILLING UP AFTER COMPILES OF SHARED LIBRARY a
short trouble-shooting tip discusses the slibclean and genkld commands.
• General Programming Concepts: Writing and Debugging Programs Chapter 19
Shared Libraries and Shared Memory provides information about the operating
system facilities provided for sharing libraries and memory allocation.
• Solaris Internals: Solaris 10 and OpenSolaris Kernel Architecture, Richard
McDougall, Jim Mauro, July2006 Part 4, and Chapter 21 provide great
information of virtual memory architecture on a comparative unix
implementation. Chapter 14 provides a good demo of analyzing file mappings in
active VM segments.
• AIX 5L Porting Guide provides details on the types of problems most likely to be
encountered when porting applications from other UNIX-based platforms to the
AIX 5L Operating System.
Discuss
• Participate in the AIX and UNIX forums:
• AIX 5L -- technical forum
• AIX for Developers Forum
• Cluster Systems Management
• IBM Support Assistant
• Performance Tools -- technical
• Virtualization -- technical
• More AIX and UNIX forums
George Cross
George Cross has experience in developing server applications in C++ on unix and is
a Senior Software Developer at Business Objects Americas. He holds a Bachelor's
degree in Computer Science from Simon Fraser University.