Special Section: PARALLEL application development SOLUTIONS

Extending OpenMP to Clusters

OpenMP is a well-known parallel The OpenMP memory model allows Cluster OpenMP does not perform
programming paradigm for shared-memory individual reads and writes to memory well for all types of programs, but
multiprocessors. In the past, OpenMP to be done in any order, as long as the programs with certain characteristics
has been confined to Symmetric Multi- synchronization operations (flushes) are can achieve reasonably good performance
Processing (SMP) machines and teamed done in a strict order–the same order in on a cluster, compared with attainable
with Message Passing Interface (MPI) which they appear in the original user’s performance on a hardware shared
technology to make use of multiple SMP program. The lack of ordering between memory machine.
systems. A new system, Cluster OpenMP, reads and writes to memory makes their
is an implementation of OpenMP that concurrent execution possible, but all
can make use of multiple SMP machines flushes in a program must be serialized,
without resorting to MPI. This advance adding overhead to the program.
has the advantage of eliminating the need
to write explicit messaging code, as well Latency to L1 1-2 cycles
as not mixing programming paradigms.
Latency to L2 5-7 cycles
The shared memory in Cluster OpenMP
is maintained across all machines through Latency to L3 12-21 cycles
a distributed shared-memory subsystem. Latency to memory 180-225 cycles
Cluster OpenMP is based on the relaxed
Gigabit Ethernet latency to remote node ~ 28000 cycles
memory consistency of OpenMP, allowing
shared variables to be made consistent InfiniBand* latency to remote node ~ 23000 cycles
only when absolutely necessary.
Figure 1. Itanium® processor latency to cache and memory compared with
messaging latency to remote nodes. Cluster OpenMP currently runs on
Performance Considerations for
Itanium®-based platforms and on systems based on processors that support
Cluster OpenMP
Intel® EM64T running Linux*.
Some memory operations are much more
expensive than others. To achieve good Figure 1 shows the number of processor This white paper can be
performance with Cluster OpenMP, the cycles required for an access to different downloaded in its entirety from
number of accesses to unprotected pages levels of cache and the latency to access http://assets.devx.com/goparallel/19403.pdf
must be as high as possible, relative to a value in the memory of a remote node. and shows performance results for a set
the number of accesses to protected This shows that access to the memory of of applications, showing that most were
pages. This means that once a page is a remote node is approximately 100 times able to achieve greater than 70 percent
brought up-to-date on a given node, a large slower than access to the local memory, of the performance of OpenMP programs
number of accesses should be made to it and thousands of times slower than access run on a shared memory machine.
before the next synchronization. In order to a value in cache. This comparison should
to accomplish this, a program should have drive home the point that local, rather than
as little synchronization as possible, and remote, memory should be used as much
re-use the data on a given page as much as possible.
as possible. This translates to avoiding
Cluster OpenMP provides shared
fine-grained synchronization, such as
memory across a cluster for an OpenMP
atomic constructs or locks, and having Buy Intel Software Products at:
program. It takes advantage of the
high data locality. www.dell.com/intelsoftware
relaxed memory model of OpenMP to
optimize the memory accesses in an Learn more at:
OpenMP program. www.intel.com/software/products/

Intel, the Intel logo, Intel Leap ahead and Intel Leap ahead logo, Pentium, Intel Core, and Itanium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.
*Other names and brands may be claimed as the property of others.
Copyright ©2006, Intel Corporation. All rights reserved.