2 Basic Concepts On

1
Basic Concepts in Parallelization

Ruud van der Pas
Senior Staff Engineer Oracle Solaris Studio Oracle Menlo Park, CA, USA
IWOMP 2010 CCS, University of Tsukuba Tsukuba, Japan June 14-16, 2010
RvdP/V1
Tutorial IWOMP 2010 CCS Un. of Tsukuba, June 14, 2010
Outline

Introduction Parallel Architectures Parallel Programming Models Data Races Summary
RvdP/V1
Introduction
RvdP/V1
Why Parallelization ?
Parallelization is another optimization technique The goal is to reduce the execution time To this end, multiple processors, or cores, are used 1 core Time parallelization 4 cores
Using 4 cores, the execution time is 1/4 of the single core time
RvdP/V1
What Is Parallelization ?
"Something" is parallel if there is a certain level of independence in the order of operations In other words, it doesn't matter in what order those operations are performed
granularity
A A
sequence of machine instructions collection of program statements algorithm problem you're trying to solve
An
The
RvdP/V1
What is a Thread ?
Loosely said, a thread consists of a series of instructions with it's own program counter (PC) and state A parallel program executes threads in parallel These threads are then scheduled onto processors
Thread 0 Thread 1 Thread 2 Thread 3
PC
..... ..... ..... .....
PC
..... ..... ..... .....
PC
..... ..... ..... ..... PC
..... ..... ..... .....
RvdP/V1
Parallel overhead
The total CPU time often exceeds the serial CPU time:
The newly introduced parallel portions in your program need to be executed Threads need time sending data to each other and synchronizing (communication) Often the key contributor, spoiling all the fun Typically, things also get worse when increasing the number of threads
Eff cient parallelization is about minimizing the i communication overhead
RvdP/V1
Communication
Serial Execution Parallel - Without communication Parallel - With communication
Wallclock time
Embarrassingly parallel: 4x faster Wallclock time is of serial wallclock time
Additional communication Less than 4x faster Consumes additional resources Wallclock time is more than of serial wallclock time Total CPU time increases
Communication
RvdP/V1
Load balancing
Perfect Load Balancing Load Imbalance
Time
1 idle 2 idle 3 idle
All threads f nish in the i same amount of time No threads is idle
Different threads need a different amount of time to f nish their i task Total wall clock time increases Program does not scale well
Thread is idle
RvdP/V1
10
About scalability

Def ne the speed-up S(P) as i S(P) := T(1)/T(P) The eff ciency E(P) is def ned as i i E(P) := S(P)/P In the ideal case, S(P)=P and E(P)=P/P=1=100% Unless the application is embarrassingly parallel, S(P) eventually starts to deviate from the ideal curve Past this point Popt, the application sees less and less benef t from i adding processors Note that both metrics give no information on the actual run-time As such, they can be dangerous to use
Speed-up S(P)
Ideal
Popt
In some cases, S(P) exceeds P
This is called "superlinear" behaviour Don't count on this to happen though
RvdP/V1
11
Amdahl's Law
Assume our program has a parallel fraction f This implies the execution time T(1) := f*T(1) + (1-f)*T(1) On P processors: T(P) = (f/P)*T(1) + (1-f)*T(1) Amdahl's law:
S(P) = T(1)/T(P) = 1 / (f/P + 1-f) Comments:

This "law' describes the effect the non-parallelizable part of a program has on scalability Note that the additional overhead caused by parallelization and speed-up because of cache effects are not taken into account
RvdP/V1
12
Amdahl's Law
16 14
It is easy to scale on a small number of processors Scalable performance however requires a high degree of parallelization i.e. f is very close to 1 This implies that you need to parallelize that part of the code where the majority of the time is spent
Speed-up
12 10 8 6 4 2 0 Value for f: 1.00 0.99 0.75 0.50 0.25 0.00
3 4
Processors
9 10 11 12 13 14 15 16
RvdP/V1
13
Amdahl's Law in practice

We can estimate the parallel fraction f Recall: T(P) = (f/P)*T(1) + (1-f)*T(1) It is trivial to solve this equation for f:
f = (1 - T(P)/T(1))/(1 - (1/P))
Example: T(1) = 100 and T(4) = 37 => S(4) = T(1)/T(4) = 2.70 f = (1-37/100)/(1-(1/4)) = 0.63/0.75 = 0.84 = 84% Estimated performance on 8 processors is then: T(8) = (0.84/8)*100 + (1-0.84)*100 = 26.5 S(8) = T/T(8) = 3.78
RvdP/V1
14
Threads Are Getting Cheap .....

Elapsed time and speed up for f = 84%
100 90 80 70 60 50 40 30 20 10 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Using 8 cores: 100/26.5 = 3.8x faster
Number of cores used

= Elapsed time = Speed up
RvdP/V1 Basic Concepts in Parallelization Tutorial IWOMP 2010 CCS Un. of Tsukuba, June 14, 2010
15
Numerical results
Consider: A=B+C+D+E
Serial Processing A = B + C A = A + D A = A + E
The roundoff behaviour is different and so the numerical results may be different too This is natural for parallel programs, but it may be hard to differentiate it from an ordinary bug ....
Parallel Processing Thread 0 T1 = B + C T1 = T1 + T2 Thread 1 T2 = D + E
RvdP/V1
16
Cache Coherence
RvdP/V1
17
Typical cache based system
CPU
L1 cache
L2 cache
Memory
RvdP/V1
18
Cache line modif cations i

page
cache line
caches
cache line inconsistency !
Main Memory
RvdP/V1
19
Caches in a parallel computer

A cache line starts in memory Over time multiple copies of this line may exist
Processors
Caches
Memory
CPU 0
invalidated
Cache Coherence ("cc"):

Tracks changes in copies Makes sure correct cache line is used Different implementations possible Need hardware support to make it eff cient i
invalidated
CPU 3 CPU 1
invalidated
CPU 2
RvdP/V1
20
Parallel Architectures
RvdP/V1
21
Uniform Memory Access (UMA)
Memory
Interconnect
cache cache cache
I/O I/O

Also called "SMP" (Symmetric Multi Processor) Memory Access time is Uniform for all CPUs CPU can be multicore Interconnect is "cc":
I/O
CPU
CPU
CPU
Pro

Easy to use and to administer Eff cient use of resources i Said to be expensive Said to be non-scalable
Con
Bus Crossbar No fragmentation Memory and I/O are shared resources
RvdP/V1
22
NUMA
Interconnect
M
I/O
M
I/O
M
I/O
Also called "Distributed Memory" or NORMA (No Remote Memory Access) Memory Access time is Non-Uniform Hence the name "NUMA" Interconnect is not "cc":
cache
cache
cache
CPU
CPU
CPU
Pro

Said to be cheap Said to be scalable Diff cult to use and administer i In-eff cient use of resources i
Ethernet, Inf niband, i etc, ...... Runs 'N' copies of the OS
Con

Memory and I/O are distributed resources
RvdP/V1
23
The Hybrid Architecture

Second-level Interconnect SMP/MC node SMP/MC node
Second-level interconnect is not cache coherent
Ethernet, Inf niband, etc, .... i Hybrid Architecture with all Pros and Cons:

UMA within one SMP/Multicore node NUMA across nodes

Basic Concepts in Parallelization Tutorial IWOMP 2010 CCS Un. of Tsukuba, June 14, 2010
RvdP/V1
24
cc-NUMA
2-nd level
Interconnect
Two-level interconnect:
UMA/SMP within one system NUMA between the systems Both interconnects support cache coherence i.e. the system is fully cache coherent

Has all the advantages ('look and feel') of an SMP Downside is the Non-Uniform Memory Access time
RvdP/V1
25
Parallel Programming Models
RvdP/V1
26
How To Program A Parallel Computer?

There are numerous parallel programming models The ones most well-known are:
Cl An Distributed Memory si an us y ng d/ te Sockets (standardized, low level) le or r SM PVM - Parallel Virtual Machine (obsolete) P
MPI
- Message Passing Interface (de-facto std)
Shared Memory
Posix
Threads (standardized, low level) (de-facto standard)
OpenMP
SM on P ly
Automatic
Parallelization (compiler does it for you)
RvdP/V1
27
Parallel Programming Models Distributed Memory - MPI
RvdP/V1
28
What is MPI?

MPI stands for the Message Passing Interface MPI is a very extensive de-facto parallel programming API for distributed memory systems (i.e. a cluster)
An MPI program can however also be executed on a shared memory system First specif cation: 1994 i
The current MPI-2 specif cation was released in 1997 i
Major enhancements Remote memory management Parallel I/O Dynamic process management
RvdP/V1
29
More about MPI
MPI has its own data types (e.g. MPI_INT)
User def ned data types are supported as well i MPI supports C, C++ and Fortran Include f le <mpi.h> in C/C++ and mpif.h in Fortran i An MPI environment typically consists of at least: A library implementing the API A compiler and linker that support the library A run time environment to launch an MPI program Various implementations available

HPC Clustertools, MPICH, MVAPICH, LAM, Voltaire MPI, Scali MPI, HP-MPI, .....
RvdP/V1
30
The MPI Programming Model

M M 0 2 1 3 M M
4 A Cluster Of Systems
RvdP/V1
31
The MPI Execution Model

All processes start MPI_Init
MPI_Finalize All processes i f nish = communication

32
The MPI Memory Model P

private
P
private
All threads/processes have access to their own, private, memory only Data transfer and most synchronization has to be programmed explicitly All data is private Data is shared explicitly by exchanging buffers
Transfer Mechanism
P
private private
P
private
RvdP/V1
33
Example - Send N Integers

#include <mpi.h> you = 0; him = 1; MPI_Init(&argc, &argv); if ( me == 0 ) {
include f le i initialize MPI environment get process ID
MPI_Comm_rank(MPI_COMM_WORLD, &me);
process 0 sends
him, 1957, MPI_COMM_WORLD);
error_code = MPI_Send(&data_buffer, N, MPI_INT, } else if ( me == 1 ) {
process 1 receives
you, 1957, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
error_code = MPI_Recv(&data_buffer, N, MPI_INT,
} MPI_Finalize(); leave the MPI environment
RvdP/V1
34
Run time Behavior

Process 0 you = 1 him = 0 me = 0 MPI_Send Process 1 you = 1 him = 0 me = 1 MPI_Recv
N integers sender = him =0 label = 1957
N integers destination = you = 1 label = 1957
Yes ! Connection established
RvdP/V1
35
The Pros and Cons of MPI
Advantages of MPI:
Flexibility - Can use any cluster of any size Straightforward - Just plug in the MPI calls Widely available - Several implementations out there Widely used - Very popular programming model Disadvantages of MPI:

Redesign of application - Could be a lot of work Easy to make mistakes - Many details to handle Hard to debug - Need to dig into underlying system More resources - Typically, more memory is needed Special care - Input/Output
RvdP/V1
36
Parallel Programming Models Shared Memory - Automatic Parallelization
RvdP/V1
37
Automatic Parallelization (-xautopar)

Compiler performs the parallelization (loop based) Different iterations of the loop executed in parallel Same binary used for any number of threads
for (i=0; i<1000; i++) a[i] = b[i] + c[i];
OMP_NUM_THREADS=4
Thread 0
0-249
Thread 1
250-499
Thread 2
500-749
Thread 3
750-999
RvdP/V1
38
Parallel Programming Models Shared Memory - OpenMP
RvdP/V1
39
Shared Memory
0 M
1 M
P M
http://www.openmp.org
40
Shared Memory Model T

private
T
private
Programming Model
All threads have access to the same, globally shared, memory
Shared Memory
Data can be shared or private Shared data is accessible by all threads Private data can only be accessed by the thread that owns it Data transfer is transparent to the programmer Synchronization takes place, but it is mostly implicit
T
private private
T
private
RvdP/V1
41
About data
In
a shared memory parallel program variables have a "label" attached to them:

Labelled
"Private" Visible to one thread only
made in local data, is not seen by others Example - Local variables in a function that is executed in parallel Labelled "Shared" Visible to all threads
Change
Change
made in global data, is seen by all others Example - Global data
RvdP/V1
42
Example - Matrix times vector

#pragma omp parallel for default(none) \ private(i,j,sum) shared(m,n,a,b,c) for (i=0; i<m; i++) j { sum = 0.0; for (j=0; j<n; j++) = * sum += b[i][j]*c[j]; a[i] = sum; i }
TID = 0
for (i=0,1,2,3,4) i = 0 sum = b[i=0][j]*c[j] a[0] = sum i = 1 sum = b[i=1][j]*c[j]
TID = 1
for (i=5,6,7,8,9) i = 5 sum = b[i=5][j]*c[j] a[5] = sum i = 6 sum = b[i=6][j]*c[j]
a[1] = sum
a[6] = sum
... etc ...

43
A Black and White comparison

MPI De-facto standard Endorsed by all key players Runs on any number of (cheap) systems Grid Ready High and steep learning curve You're on your own All or nothing model No data scoping (shared, private, ..) More widely used (but ....) Sequential version is not preserved Requires a library only Requires a run-time environment Easier to understand performance OpenMP De-facto standard Endorsed by all key players Limited to one (SMP) system Not (yet?) Grid Ready Easier to get started (but, ...) Assistance from compiler Mix and match model Requires data scoping Increasingly popular (CMT !) Preserves sequential code Need a compiler No special environment Performance issues implicit
RvdP/V1
44
The Hybrid Parallel Programming Model
RvdP/V1
45
The Hybrid Programming Model

Distributed Memory
Shared Memory
Shared Memory
0 M
1 M
P M
0 M
1 M
P M
RvdP/V1
46
Data Races
RvdP/V1
47
About Parallelism
Parallelism Independence No Fixed Ordering
"Something" that does not obey this rule, is not parallel (at that level ...)
RvdP/V1
48
Shared Memory Programming T

private
T
private
Y
Shared Memory
T
private private
T
private
Threads communicate via shared memory

49
What is a Data Race?

Two different threads in a multi-threaded shared memory program Access the same (=shared) memory location

Asynchronously Without holding any common exclusive locks At least one of the accesses is a write/store
and and
RvdP/V1
50
Example of a data race

#pragma omp parallel shared(n)
{n = omp_get_thread_num();}
T
private
private
Shared Memory
RvdP/V1
51
Another example
#pragma omp parallel shared(x)
{x = x + 1;}
T
private
R/W
T
R/W
private
Shared Memory
RvdP/V1
52
About Data Races

Loosely described, a data race means that the update of a shared variable is not well protected A data race tends to show up in a nasty way:

Numerical results are (somewhat) different from run to run Especially with Floating-Point data diff cult to i distinguish from a numerical side-effect Changing the number of threads can cause the problem to seemingly (dis)appear May also depend on the load on the system May only show up using many threads
RvdP/V1
53
A parallel loop
for (i=0; i<8; i++) a[i] = a[i] + b[i];
Thread 1 a[0]=a[0]+b[0] a[1]=a[1]+b[1] a[2]=a[2]+b[2] a[3]=a[3]+b[3] Every iteration in this loop is independent of the other iterations
Thread 2 a[4]=a[4]+b[4] a[5]=a[5]+b[5] a[6]=a[6]+b[6] a[7]=a[7]+b[7] Time
RvdP/V1
54
Not a parallel loop

for (i=0; i<8; i++) a[i] = a[i+1] + b[i];
Thread 1 a[0]=a[1]+b[0] a[1]=a[2]+b[1] a[2]=a[3]+b[2] a[3]=a[4]+b[3] The result is not deterministic when run in parallel !
Thread 2 a[4]=a[5]+b[4] a[5]=a[6]+b[5] a[6]=a[7]+b[6] a[7]=a[8]+b[7] Time
RvdP/V1
55
About the experiment
We manually parallelized the previous loop
The compiler detects the data dependence and does not parallelize the loop Vectors a and b are of type integer
We use the checksum of a as a measure for correctness:
checksum += a[i] for i = 0, 1, 2, ...., n-2 The correct, sequential, checksum result is computed as a reference
We ran the program using 1, 2, 4, 32 and 48 threads
Each of these experiments was repeated 4 times
RvdP/V1
56
Numerical results
threads: threads: threads: threads: threads: threads: threads: threads: threads: threads: threads: threads: threads: threads: threads: threads: threads: threads: threads: threads: 1 1 1 1 2 2 2 2 4 4 4 4 32 32 32 32 48 48 48 48 checksum checksum checksum checksum checksum checksum checksum checksum checksum checksum checksum checksum checksum checksum checksum checksum checksum checksum checksum checksum 1953 1953 1953 1953 1953 1953 1953 1953 1905 1905 1953 1937 1525 1473 1489 1513 936 1007 887 822 correct correct correct correct correct correct correct correct correct correct correct correct correct correct correct correct correct correct correct correct value value value value value value value value value value value value value value value value value value value value 1953 1953 1953 1953 1953 1953 1953 1953 1953 1953 1953 1953 1953 1953 1953 1953 1953 1953 1953 1953
Data Race In Action !
RvdP/V1
57
Summary
RvdP/V1
58
Parallelism Is Everywhere
Multiple levels of parallelism:
Granularity Instruction Level Chip Level System Level Grid Level Technology Superscalar Multicore SMP/cc-NUMA Cluster Programming Model Compiler Compiler, OpenMP, MPI Compiler, OpenMP, MPI MPI
Threads Are Getting Cheap
RvdP/V1

2 Basic Concepts On

Încărcat de

Informații document

Descriere originală:

Titlu original

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

2 Basic Concepts On

Încărcat de

Drepturi de autor:

Formate disponibile

1

Basic Concepts in Parallelization

Basic Concepts in Parallelization

Tutorial IWOMP 2010 CCS Un. of Tsukuba, June 14, 2010

Introduction Parallel Architectures Parallel Programming Models Data Races Summary

Basic Concepts in Parallelization

Tutorial IWOMP 2010 CCS Un. of Tsukuba, June 14, 2010

Basic Concepts in Parallelization

Tutorial IWOMP 2010 CCS Un. of Tsukuba, June 14, 2010

Basic Concepts in Parallelization

Tutorial IWOMP 2010 CCS Un. of Tsukuba, June 14, 2010

Basic Concepts in Parallelization

Tutorial IWOMP 2010 CCS Un. of Tsukuba, June 14, 2010

..... ..... ..... .....

..... ..... ..... .....

..... ..... ..... ..... PC

..... ..... ..... .....

Basic Concepts in Parallelization

Tutorial IWOMP 2010 CCS Un. of Tsukuba, June 14, 2010

Eff cient parallelization is about minimizing the i communication overhead

Basic Concepts in Parallelization

Tutorial IWOMP 2010 CCS Un. of Tsukuba, June 14, 2010

Embarrassingly parallel: 4x faster Wallclock time is of serial wallclock time

Basic Concepts in Parallelization

Tutorial IWOMP 2010 CCS Un. of Tsukuba, June 14, 2010

1 idle 2 idle 3 idle

All threads f nish in the i same amount of time No threads is idle

Basic Concepts in Parallelization

Tutorial IWOMP 2010 CCS Un. of Tsukuba, June 14, 2010

This is called "superlinear" behaviour Don't count on this to happen though

Basic Concepts in Parallelization

Tutorial IWOMP 2010 CCS Un. of Tsukuba, June 14, 2010

S(P) = T(1)/T(P) = 1 / (f/P + 1-f) Comments:

Basic Concepts in Parallelization

Tutorial IWOMP 2010 CCS Un. of Tsukuba, June 14, 2010

12 10 8 6 4 2 0 Value for f: 1.00 0.99 0.75 0.50 0.25 0.00

Basic Concepts in Parallelization

Tutorial IWOMP 2010 CCS Un. of Tsukuba, June 14, 2010

Amdahl's Law in practice

Basic Concepts in Parallelization

Tutorial IWOMP 2010 CCS Un. of Tsukuba, June 14, 2010

Threads Are Getting Cheap .....

Using 8 cores: 100/26.5 = 3.8x faster

Number of cores used

Parallel Processing Thread 0 T1 = B + C T1 = T1 + T2 Thread 1 T2 = D + E

Basic Concepts in Parallelization

Tutorial IWOMP 2010 CCS Un. of Tsukuba, June 14, 2010

Basic Concepts in Parallelization

Tutorial IWOMP 2010 CCS Un. of Tsukuba, June 14, 2010

Typical cache based system

Basic Concepts in Parallelization

Tutorial IWOMP 2010 CCS Un. of Tsukuba, June 14, 2010

Cache line modif cations i

cache line inconsistency !

Basic Concepts in Parallelization

Tutorial IWOMP 2010 CCS Un. of Tsukuba, June 14, 2010

Caches in a parallel computer

Cache Coherence ("cc"):

Basic Concepts in Parallelization

Tutorial IWOMP 2010 CCS Un. of Tsukuba, June 14, 2010

Basic Concepts in Parallelization

Tutorial IWOMP 2010 CCS Un. of Tsukuba, June 14, 2010