Arhitecturi Paralele de Calcul

MPI

Prep. Ing. Adrian Muraru

Agenda

1. Message Passing
2. SPMD/MPMD
3. MPI, clase de functii
4. Supliment 
 Message Passing
Model de programare paralela in sisteme cu memorie distribuita
(Distributed memory )

NUMA,MPP

S1, S2 – sect. de program seriale

P1-P4 – sect. de program paralele

Pe fiecare din noduri ruleaza cate un

proces. In timpul executiei sectiunilor
paralele (P1-P4) aceste procese comunica
intre ele.
 SPMD/MPMD
Modele de calcul paralel in sisteme cu memorie distribuita

SPMD (Single Program Multiple Data) (a)

MPMD (Multiple Program Multiple Data) (b,c)
Modelul SPMD
1. Read array a[] from input
2. Get my rank
3. If rank==0 then is=1,ie=2
If rank==0 then is=1,ie=2
If rank==0 then is=1,ie=2
4. Process from a(is) to a(ie)
5. Gather the results to
process 0
6) If rank==0 then write
array a() to the output file
1. Read array a[] from input
2. Get my rank
3. If rank==0 then is=1,ie=2
If rank==0 then is=1,ie=2
If rank==0 then is=1,ie=2
4. Process from a(is) to a(ie)
5. Gather the results to
process 0
6) If rank==0 then write
array a() to the output file
1. Read array a[] from input
2. Get my rank
3. If rank==0 then is=1,ie=2
If rank==0 then is=1,ie=2
If rank==0 then is=1,ie=2
4. Process from a(is) to a(ie)
5. Gather the results to
process 0
6) If rank==0 then write
array a() to the output file
 MPI. Clase de functii
• Functii de management mediu

• Functii de comunicatie punct-la-punct

• Operatii colective

• Grupuri de procese/Comunicatori

• Topologii (virtuale) de procese

 Functii de management mediu
• initializare, terminare, interogare mediu

•MPI_Init – initializare mediu

MPI_Init (&argc,&argv)
MPI_INIT (ierr)

•MPI_Comm_size – determina numarul de procese din grupul asociat unui com.

MPI_Comm_size (comm,&size)
MPI_COMM_SIZE (comm,size,ierr)

•MPI_Comm_rank – determina rangul procesului apelant in cadrul unui com.

MPI_Comm_rank (comm,&rank)
MPI_COMM_RANK (comm,rank,ierr)

•MPI_Abort –opreste toate procesele asociate unui comunicator

MPI_Abort (comm,errorcode)
MPI_ABORT (comm,errorcode,ierr)

•MPI_Finalize -finalizare mediu MPI

MPI_Finalize ()
MPI_FINALIZE (ierr)
 Exemplu
• initializare, terminare, interogare mediu

#include "mpi.h"
#include <stdio.h>
int main(argc,argv)
int argc;
char *argv[]; {

int numtasks, rank, rc;

rc = MPI_Init(&argc,&argv);
if (rc != MPI_SUCCESS) {
printf ("Error starting MPI program. Terminating.\n");
MPI_Abort(MPI_COMM_WORLD, rc);
}
MPI_Comm_size(MPI_COMM_WORLD,&numtasks); MPI_Comm_rank
(MPI_COMM_WORLD,&rank);
printf ("Number of tasks= %d My rank= %d\n", numtasks,rank);
/******* do some work *******/
MPI_Finalize();
}
 Comunicatie punct-la-punct
Transferul de mesaje intre 2 taskuri MPI distincte intr-un anumit sens.

• Tipuri de operatii punct-la-punct

Exista diferite semantici pentru operatiile de send/receive :

•Synchronous send
•Blocking send / blocking receive
•Non-blocking send / non-blocking receive
•Buffered send
•Combined send/receive
•"Ready" send

•o rutina send poate fi utilizata cu orice alt tip de rutina receive

•rutine MPI asociate (wait,probe)

 Comunicatie punct-la-punct
Buffering

•rezolva probleme practice,

nespecificate in standard dar
tratate in implementare:

realizarea unei comunicatii

asincrone (buffere sistem,
buffere aplicatie)
 Comunicatie punct-la-punct
Operatii blocate vs ne-blocante

Operatii blocante

•O operatie de send blocanta va returna doar atunci cand zona de

date ce a fost trimisa poate fi reutilizata, fara sa afecteze datele
primite de destinatar.
O operatie de send blocanta poate fi :
• sincrona :va returna doar atunci cand datele au ajuns efectiv la
destinatar
• asincrona : se utilizeaza o zona de tampon din sistem pentru salvarea
datelor ce urmeaza a fi trimise

•O operatie receive blocanta va “returna” doar dupa ce datele au fost

primite si pot fi folosite in program.

Operatii blocante

Returneaza controlul imediat, notifica libraria care se va ocupa de

transfer. Exista functii speciale de asteptare/interogare a statusului
transferului
 Comunicatie punct-la-punct
Operatii blocate vs ne-blocante (cont)

Blocking send MPI_Send(buffer,count,type,dest,tag,comm)

Blocking receive MPI_Recv(buffer,count,type,source,tag,comm, status)

Blocking MPI_Probe (source,tag,comm,&status)

Probe

Non-blocking MPI_Isend(buffer,count,type,dest,tag,comm, request)

send
Non-blocking MPI_Irecv(buffer,count,type,source,tag,comm,
receive request)
Wait MPI_Wait (&request,&status)
Test MPI_Test (&request,&flag,&status)

Non-blocking MPI_Iprobe (source,tag,comm,&flag,&status)

probe
 Exemplu operatii blocante
#include "mpi.h"
#include <stdio.h>
int main(argc,argv)
int argc;
char *argv[]; {
int numtasks, rank, dest, source, rc, count, tag=1;
char inmsg, outmsg='x';
MPI_Status Stat;
MPI_Init(&argc,&argv);
MPI_Comm_size(MPI_COMM_WORLD, &numtasks); MPI_Comm_rank
(MPI_COMM_WORLD, &rank);
if (rank == 0) {
dest = source = 1;
rc = MPI_Send(&outmsg, 1, MPI_CHAR, dest, tag,
MPI_COMM_WORLD);
rc = MPI_Recv(&inmsg, 1, MPI_CHAR, source, tag,
MPI_COMM_WORLD, &Stat);
}
 Exemplu operatii blocante (cont)

else if (rank == 1){

dest = source = 0;
rc = MPI_Recv(&inmsg, 1, MPI_CHAR, source, tag,
MPI_COMM_WORLD, &Stat);
rc = MPI_Send(&outmsg, 1, MPI_CHAR, dest, tag,
MPI_COMM_WORLD);
}
rc = MPI_Get_count(&Stat, MPI_CHAR, &count);
printf("Task %d: Received %d char(s) from task %d with tag %d
\n", rank, count, Stat.MPI_SOURCE, Stat.MPI_TAG);
MPI_Finalize();
}
 Exemplu operatii neblocante

int main(int argc, char *argv[]{

int numtasks, rank, next, prev, buf[2], tag1=1, tag2=2;
MPI_Request reqs[4];
MPI_Status stats[4];
MPI_Init(&argc,&argv);
MPI_Comm_size(MPI_COMM_WORLD, &numtasks); MPI_Comm_rank
(MPI_COMM_WORLD, &rank);
prev = rank-1; next = rank+1;
if (rank == 0) prev = numtasks - 1;
if (rank == (numtasks - 1)) next = 0;
MPI_Irecv(&buf[0],1,MPI_INT,prev,tag1,MPI_COMM_WORLD,&reqs[0]);
MPI_Irecv(&buf[1],1,MPI_INT,next,tag2,MPI_COMM_WORLD,&reqs[1]);
MPI_Isend(&rank,1,MPI_INT,prev,tag2,MPI_COMM_WORLD,&reqs[2]);
MPI_Isend(&rank,1,MPI_INT,next,tag1,MPI_COMM_WORLD,&reqs[3]);
// do some work
MPI_Waitall(4, reqs, stats);
MPI_Finalize();
}
 Operatii colective
• Operatiile colective implica toata procesele din cadrul
unui comunicator. Toate procesele sunt membre ale
comunicatorului initial, predefinit MPI_COMM_WORLD.

Tipuri de operatii colective:

•Sincronizare: procesele asteapta toti membrii grupului sa

ajunga in punctul de jonctiune.

•Transfer de date - broadcast, scatter/gather, all to all.

•Calcule colective (reductions) – un membru al grupului

colecteaza datele de la toti ceilalti membrii si realizeaza o
operatie asupra acestora (min, max, adunare, inmultire, etc.)

Observatie:
Toate operatiile colective sunt blocante
 Operatii colective

MPI_Barrier
MPI_Barrier (comm)
MPI_BARRIER (comm,ierr)

Fiecare task se va bloca in acest apel pana ce toti

membrii din grup au ajuns in acest punct
Operatii colective

Transfer de date
Operatii colective
Operatii colective
Operatii colective
Operatii colective
 Grupuri de procese/Comunicatori

Groupuri vs. Comunicatori

• Un grup reprezinta o multime ordonata de procese.
In cadrul unui grup, fiecare proces are asociat un identificator
unic (rang) cuprins intre 0 si N-1, unde N este numarul de
procese din grupul respectiv.

• Un comunicator “imbraca” un grup de procese in vederea

comunicarii intre acestea. Toate operatiile de trimitire/primire
mesaje trebuie sa specifice un comunicator. Comunicatorul
poate fi privit si ca o eticheta(tag) suplimentara asociata unui
mesaj ce este transmis in MPI.

• Din perspectiva programatorului notiunile de grup si

comunicator se “contopesc”: rutinele de grup specifica ce
procese vor fi utilizate pentru a construi un comunicator
 Grupuri de procese/Comunicatori

•Grupurile/comunicatori
sunt dinamice (pot fi
create/distruse in timpul
executiei)

•Un proces poate face parte

din mai multe
grupuri/comunicatori
simultan(cu ID unic in
fiecare)
 Grupuri de procese/Comunicatori
#include "mpi.h"
#include <stdio.h>
#define NPROCS 12
int main(int argc, char *argv[]) {
int rank, new_rank, sendbuf, recvbuf, numtasks;
Int ranks1[6]={0,1,2,3,4,5}, ranks2[4]={6,7,8,9,10,11};
MPI_Group orig_group, new_group;
MPI_Comm new_comm;

MPI_Init(&argc,&argv);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &numtasks);
sendbuf = rank;
/* Extract the original group handle */
MPI_Comm_group(MPI_COMM_WORLD, &orig_group);
/* Divide tasks into two distinct groups based upon rank */
if (rank < NPROCS/2) {
MPI_Group_incl(orig_group, NPROCS/2, ranks1, &new_group);
} else {
MPI_Group_incl(orig_group, NPROCS/2, ranks2, &new_group); }
/* Create new new communicator and then perform collective communications */
MPI_Comm_create(MPI_COMM_WORLD, new_group, &new_comm);
MPI_Allreduce (&sendbuf, &recvbuf, 1, MPI_INT, MPI_SUM, new_comm);
MPI_Group_rank (new_group, &new_rank);
printf("rank= %d newrank= %d recvbuf= %d\n",rank,new_rank,recvbuf); MPI_Finalize
();
}
 Grupuri de procese/Comunicatori
Comunicatie intra- si inter-grup

Comm_A Comm_B
 Topologii virtuale

Ce mai sunt si astea?

• In termeni MPI, o topologie virtuala descrie o
mapare/ordonare/organizare a proceselor MPI intr-o
“forma” geometrica
• Doua tipuri:
– Cartesian (Grid)
– Graf.
• Topologiile MPI sunt virtuale – nu exista (in general) o
relatie intre aceste topologii si structura fizica a
sistemului paralel
• Topologiile virtuale sunt bazate pe grupuri si
comunicatori
• Sunt “programate” de dezvoltatorul de aplicatii
 Performanta programelor paralele

• In general, un program paralel prezinta trei

“puncte fierbinti” (factori ce determina
performanta)

– Overhead-ul de comunicatie
– Punctele de sincronizare
– Diferenta de balansare intre taskurile executate
pe elemente de procesare distincte.

ref: [2]&[3]
 Mandelbrot paralel

process group file: config

skirtos 0 /home/pmandel
navigator 1 /home/pmandel
navigator 1 /home/pmandel
navigator 1 /home/pmandel
navigator 1 /home/pmandel
navigator 1 /home/pmandel
skirtos 1 /home/pmandel
navigator 1 /home/pmandel
navigator 1 /home/pmandel

Skirtos:$ export DISPLAY=navigator:0

Skirtos:$ mpirun –p4pg config /home/pmandel
 Referinte

• Tutorial MPI [1]

http://www.llnl.gov/computing/tutorials/mpi/

• Parallel Computing [2]

http://www.llnl.gov/computing/tutorials/parallel_comp/

• Practical MPI Programming, IBM Redbooks [3]

http://www.redbooks.ibm.com/abstracts/sg245380.html
Multumesc !

Intrebari? Observatii?

• Prezentarea este accesibila la :

http://cs.pub.ro/~apc/
• Adrian MURARU
• [adim@cs.pub.ro]