3/29/2016 IntroductiontotheMessagePassingInterface(MPI)usingC

Anintroductiontothe
MessagePassingInterface(MPI)
usingC
ThisisashortintroductiontotheMessagePassingInterface(MPI)designedtoconveythefundamental
operationanduseoftheinterface.Thisintroductionisdesignedforreaderswithsomebackground
programmingC,andshoulddeliverenoughinformationtoallowreaderstowriteandruntheirown(very
simple)parallelCprogramsusingMPI.

ThereexistsaversionofthistutorialforFortranprogramerscalledIntroductionthetheMessagePassing
Interface(MPI)usingFortran.

WhatisMPI?
MPIisalibraryofroutinesthatcanbeusedtocreateparallelprogramsinCorFortran77.StandardC
andFortranincludenoconstructssupportingparallelismsovendorshavedevelopedavarietyof
extensionstoallowusersofthoselanguagestobuildparallelapplications.Theresulthasbeenaspateof
nonportableapplications,andaneedtoretrainprogrammersforeachplatformuponwhichtheywork.

TheMPIstandardwasdevelopedtoamelioratetheseproblems.ItisalibrarythatrunswithstandardCor
Fortranprograms,usingcommonlyavailableoperatingsystemservicestocreateparallelprocessesand
exchangeinformationamongtheseprocesses.

MPIisdesignedtoallowuserstocreateprogramsthatcanrunefficientlyonmostparallelarchitectures.
Thedesignprocessincludedvendors(suchasIBM,Intel,TMC,Cray,Convex,etc.),parallellibrary
authors(involvedinthedevelopmentofPVM,Linda,etc.),andapplicationsspecialists.Thefinalversion
forthedraftstandardbecameavailableinMayof1994.

MPIcanalsosupportdistributedprogramexecutiononheterogenoushardware.Thatis,youmayruna
programthatstartsprocessesonmultiplecomputersystemstoworkonthesameproblem.Thisisuseful
withaworkstationfarm.

Helloworld
HereisthebasicHelloworldprograminCusingMPI:
#include<stdio.h>
#include<mpi.h>

main(intargc,char**argv)
{
intierr;

ierr=MPI_Init(&argc,&argv);
printf("Helloworld\n");

ierr=MPI_Finalize();
}
http://condor.cc.ku.edu/~grobe/docs/introMPIC.shtml 1/18
3/29/2016 IntroductiontotheMessagePassingInterface(MPI)usingC

Ifyoucompilehello.cwithacommandlike
mpicchello.cohello

youwillcreateanexecutablefilecalledhello,whichyoucanexecutebyusingthempiruncommandasin
thefollowingsessionsegment:

$mpirunnp4hello
Helloworld
Helloworld
Helloworld
Helloworld
$

Whentheprogramstarts,itconsistsofonlyoneprocess,sometimescalledthe"parent","root",or
"master"process.WhentheroutineMPI_Initexecuteswithintherootprocess,itcausesthecreationof3
additionalprocesses(toreachthenumberofprocesses(np)specifiedonthempiruncommandline),
sometimescalled"child"processes.

Eachoftheprocessesthencontinuesexecutingseparateversionsofthehelloworldprogram.Thenext
statementineveryprogramistheprintfstatement,andeachprocessprints"Helloworld"asdirected.
Sinceterminaloutputfromeveryprogramwillbedirectedtothesameterminal,weseefourlinessaying
"Helloworld".

Identifyingtheseparateprocesses
Aswritten,wecannottellwhich"Helloworld"linewasprintedbywhichprocess.Toidentifyaprocess
weneedsomesortofprocessIDandaroutinethatletsaprocessfinditsownprocessID.MPIassignsan
integertoeachprocessbeginningwith0fortheparentprocessandincrementingeachtimeanewprocess
iscreated.AprocessIDisalsocalledits"rank".

MPIalsoprovidesroutinesthatlettheprocessdetermineitsprocessID,aswellasthenumberof
processesthatarehavebeencreated.

HereisanenhancedversionoftheHelloworldprogramthatidentifiestheprocessthatwriteseachline
ofoutput:

#include<stdio.h>
#include<mpi.h>

main(intargc,char**argv)
{
intierr,num_procs,my_id;

ierr=MPI_Init(&argc,&argv);

/*findoutMYprocessID,andhowmanyprocesseswerestarted.*/

ierr=MPI_Comm_rank(MPI_COMM_WORLD,&my_id);
ierr=MPI_Comm_size(MPI_COMM_WORLD,&num_procs);

printf("Helloworld!I'mprocess%ioutof%iprocesses\n",
my_id,num_procs);

ierr=MPI_Finalize();
http://condor.cc.ku.edu/~grobe/docs/introMPIC.shtml 2/18
3/29/2016 IntroductiontotheMessagePassingInterface(MPI)usingC

Whenwerunthisprogram,eachprocessidentifiesitself:
$mpicchello2.cohello2

$mpirunnp4hello2
Helloworld!I'mprocess0outof4processes.
Helloworld!I'mprocess2outof4processes.
Helloworld!I'mprocess1outof4processes.
Helloworld!I'mprocess3outof4processes.
$

Notethattheprocessnumbersarenotprintedinascendingorder.Thatisbecausetheprocessesexecute
independentlyandexecutionorderwasnotcontrolledinanyway.Theprogramsmayprinttheirresultsin
differentorderseachtimetheyarerun.

(TofindoutwhichOriginprocessorsandmemoriesareusedtorunaprogramyoucanturnonthe
MPI_DSM_VERBOSEenvironmentvariablewith"exportMPI_DSM_VERBOSE=ON",orequivalent.)

Toleteachprocessperformadifferenttask,youcanuseaprogramstructurelike:
#include<mpi.h>

main(intargc,char**argv)
{
intmy_id,root_process,ierr,num_procs;
MPI_Statusstatus;

/*Createchildprocesses,eachofwhichhasitsownvariables.
*Fromthispointon,everyprocessexecutesaseparatecopy
*ofthisprogram.EachprocesshasadifferentprocessID,
*rangingfrom0tonum_procsminus1,andCOPIESofall
*variablesdefinedintheprogram.Novariablesareshared.
**/

ierr=MPI_Init(&argc,&argv);

/*findoutMYprocessID,andhowmanyprocesseswerestarted.*/

ierr=MPI_Comm_rank(MPI_COMM_WORLD,&my_id);
ierr=MPI_Comm_size(MPI_COMM_WORLD,&num_procs);

if(my_id==0){

/*dosomeworkasprocess0*/
}
elseif(my_id==1){

/*dosomeworkasprocess1*/
}
elseif(my_id==2){

/*dosomeworkasprocess2*/
}
else{

/*dothisworkinanyremainingprocesses*/
}
http://condor.cc.ku.edu/~grobe/docs/introMPIC.shtml 3/18
3/29/2016 IntroductiontotheMessagePassingInterface(MPI)usingC

/*Stopthisprocess*/

ierr=MPI_Finalize();
}

BasicMPIcommunicationroutines
Itisimportanttorealizethatseparateprocessessharenomemoryvariables.Theyappeartobeusingthe
samevariables,buttheyarereallyusingCOPIESofanyvariabledefinedintheprogram.

Asaresult,theseprogramscannotcommunicatewitheachotherbyexchanginginformationinmemory
variables.InsteadtheymayuseanyofalargenumberofMPIcommunicationroutines.Thetwobasic
routinesare:

MPI_Send,tosendamessagetoanotherprocess,and
MPI_Recv,toreceiveamessagefromanotherprocess.

ThesyntaxofMPI_Sendis:
intMPI_Send(void*data_to_send,intsend_count,MPI_Datatypesend_type,
intdestination_ID,inttag,MPI_Commcomm);

data_to_send:variableofaCtypethatcorrespondstothesend_typesuppliedbelow
send_count:numberofdataelementstobesent(nonnegativeint)
send_type:datatypeofthedatatobesent(oneoftheMPIdatatypehandles)
destination_ID:processIDofdestination(int)
tag:messagetag(int)
comm:communicator(handle)

OnceaprogramcallsMPI_Send,itblocksuntilthedatatransferhastakenplaceandthedata_to_send
variablecanbesafelyreused.Asaresult,theseroutinesprovideasimplesynchronizationservicealong
withdataexchange.

ThesyntaxofMPI_Recvis:
intMPI_Recv(void*received_data,intreceive_count,MPI_Datatypereceive_type,
intsender_ID,inttag,MPI_Commcomm,MPI_Status*status);

received_data:variableofaCtypethatcorrespondstothereceive_typesuppliedbelow
receive_count:numberofdataelementsexpected(int)
receive_type:datatypeofthedatatobereceived(oneoftheMPIdatatypehandles)
sender_ID:processIDofthesendingprocess(int)
tag:messagetag(int)
comm:communicator(handle)
status:statusstruct(MPI_Status)

Thereceive_count,sender_ID,andtagvaluesmaybespecifiedsoastoallowmessagesofunknown
length,fromseveralsources(MPI_ANY_SOURCE),orwithvarioustagvalues(MPI_ANY_TAG).

Theamountofinformationactuallyreceivedcanthenberetrievedfromthestatusvariable,aswith:
countMPI_Get_count(&status,MPI_FLOAT,&true_received_count);
received_source=status.MPI_SOURCE;
http://condor.cc.ku.edu/~grobe/docs/introMPIC.shtml 4/18
3/29/2016 IntroductiontotheMessagePassingInterface(MPI)usingC

received_tag=status.MPI_TAG;

MPI_Recvblocksuntilthedatatransferiscompleteandthereceived_datavariableisavailableforuse.

ThebasicdatatypesrecognizedbyMPIare:

MPIdatatypehandle Cdatatype
MPI_INT int
MPI_SHORT short
MPI_LONG long
MPI_FLOAT float
MPI_DOUBLE double
MPI_CHAR char
MPI_BYTE unsignedchar
MPI_PACKED

Therealsoexistothertypeslike:MPI_UNSIGNED,MPI_UNSIGNED_LONG,and
MPI_LONG_DOUBLE.

Acommonpatternofprocessinteraction
Acommonpatternofinteractionamongparallelprocessesisforone,themaster,toallocateworktoaset
ofslaveprocessesandcollectresultsfromtheslavestosynthesizeafinalresult.

Themasterprocesswillexecuteprogramstatementslike:
/*distributeportionsofarray1toslaves.*/

for(an_id=1;an_id<num_procs;an_id++){

start_row=an_id*num_rows_per_process;

ierr=MPI_Send(&num_rows_to_send,1,MPI_INT,
an_id,send_data_tag,MPI_COMM_WORLD);

ierr=MPI_Send(&array1[start_row],num_rows_per_process,
MPI_FLOAT,an_id,send_data_tag,MPI_COMM_WORLD);
}

/*and,thencollecttheresultsfromtheslaveprocesses,
*hereinavariablecalledarray2,anddosomethingwiththem.*/

for(an_id=1an_id<num_procs;an_id++){

ierr=MPI_Recv(&array2,num_rows_returned,MPI_FLOAT,
MPI_ANY_SOURCE,MPI_ANY_TAG,MPI_COMM_WORLD,&status);

/*dosomethingwitharray2here*/

http://condor.cc.ku.edu/~grobe/docs/introMPIC.shtml 5/18
3/29/2016 IntroductiontotheMessagePassingInterface(MPI)usingC

/*andthenprintoutsomefinalresultusingthe
*informationcollectedfromtheslaves.*/

Inthisfragment,themasterprogramsendsacontiguousportionofarray1toeachslaveusingMPI_Send
andthenreceivesaresponsefromeachslaveviaMPI_Recv.Inpractice,themasterdoesnothavetosend
anarrayitcouldsendascalarorsomeotherMPIdatatype,anditcouldconstructarray1fromany
componentstowhichithasaccess.

Herethereturnedinformationisputinarray2,whichwillbewrittenovereverytimeadifferentmessage
isreceived.Therefore,itwillprobablybecopiedtosomeothervariablewithinthereceivingloop.

NotetheuseoftheMPIconstantMPI_ANY_SOURCEtoallowthisMPI_Recvcalltoreceivemessages
fromanyprocess.Insomecases,aprogramwouldneedtodetermineexactlywhichprocesssenta
messagereceivedusingMPI_ANY_SOURCE.status.MPI_SOURCEwillholdthatinformation,
immediatelyfollowingthecalltoMPI_Recv.

Theslaveprogramtoworkwiththismasterwouldresemble:

/*Receiveanarraysegment,herecalledarray2*/.

ierr=MPI_Recv(&num_rows_to_receive,1,MPI_INT,
root_process,MPI_ANY_TAG,MPI_COMM_WORLD,&status);

ierr=MPI_Recv(&array2,num_rows_to_receive,MPI_FLOAT,
root_process,MPI_ANY_TAG,MPI_COMM_WORLD,&status);

/*Dosomethingwitharray2here,placingtheresultinarray3,
*andsendarray3totherootprocess.*/

ierr=MPI_Send(&array3,num_rows_to_return,MPI_FLOAT,
root_process,return_data_tag,MPI_COMM_WORLD);

Therecouldbemanyslaveprogramsrunningatthesametime.Eachonewouldreceivedatainarray2
fromthemasterviaMPI_Recvandworkonitsowncopyofthatdata.Eachslavewouldconstructits
owncopyofarray3,whichitwouldthensendtothemasterusingMPI_Send.

Anonparallelprogramthatsumsthevaluesinanarray
Thefollowingprogramcalculatesthesumoftheelementsofaarray.Itwillbefollowedbyaparallel
versionofthesameprogramusingMPIcalls.
#include<stdio.h>
#definemax_rows10000000

intarray[max_rows];

main(intargc,char**argv)
{
inti,num_rows;
longintsum;

printf("pleaseenterthenumberofnumberstosum:");
scanf("%i",&num_rows);

if(num_rows>max_rows){
http://condor.cc.ku.edu/~grobe/docs/introMPIC.shtml 6/18
3/29/2016 IntroductiontotheMessagePassingInterface(MPI)usingC

printf("Toomanynumbers.\n");
exit(1);
}

/*initializeanarray*/

for(i=0;i<num_rows;i++){
array[i]=i;
}

/*computesum*/

sum=0;
for(i=0;i<num_rows;i++){
sum+=array[i];
}

printf("Thegrandtotalis:%i\n",sum);
}

Designforaparallelprogramtosumanarray
ThecodebelowshowsacommonFortranstructureforincludingbothmasterandslavesegmentsinthe
parallelversionoftheexampleprogramjustpresented.Itiscomposedofashortsetupsectionfollowed
byasingleif...elseloopwherethemasterprocessexecutesthestatmentsbetweenthebracketsafter
theifstatement,andtheslaveprocessesexecutethestatementsbetweenthebracketsaftertheelse
statement.
/*ThisprogramsumsallrowsinanarrayusingMPIparallelism.
*Therootprocessactsasamasterandsendsaportionofthe
*arraytoeachchildprocess.Masterandchildprocessesthen
*allcalculateapartialsumoftheportionofthearrayassigned
*tothem,andthechildprocessessendtheirpartialsumsto
*themaster,whocalculatesagrandtotal.
**/

#include<stdio.h>
#include<mpi.h>

intmy_id,root_process,ierr,num_procs,an_id;
MPI_Statusstatus;

root_process=0;

/*Nowreplicatethisprocesstocreateparallelprocesses.

ierr=MPI_Init(&argc,&argv);

/*findoutMYprocessID,andhowmanyprocesseswerestarted*/

ierr=MPI_Comm_rank(MPI_COMM_WORLD,&my_id);
ierr=MPI_Comm_size(MPI_COMM_WORLD,&num_procs);

if(my_id==root_process){
/*Imustbetherootprocess,soIwillquerytheuser
*todeterminehowmanynumberstosum.

*initializeanarray,

http://condor.cc.ku.edu/~grobe/docs/introMPIC.shtml 7/18
3/29/2016 IntroductiontotheMessagePassingInterface(MPI)usingC

*distributeaportionofthearraytoeachchildprocess,

*andcalculatethesumofthevaluesinthesegmentassigned
*totherootprocess,

*and,finally,Icollectthepartialsumsfromslaveprocesses,
*printthem,andaddthemtothegrandsum,andprintit*/
}

else{

/*Imustbeslaveprocess,soImustreceivemyarraysegment,

*calculatethesumofmyportionofthearray,

*and,finally,sendmyportionofthesumtotherootprocess.*/

/*Stopthisprocess*/

ierr=MPI_Finalize();
}

Thecompleteparallelprogramtosumaarray
HereistheexpandedparallelversionofthesameprogramusingMPIcalls.
/*ThisprogramsumsallrowsinanarrayusingMPIparallelism.
*Therootprocessactsasamasterandsendsaportionofthe
*arraytoeachchildprocess.Masterandchildprocessesthen
*allcalculateapartialsumoftheportionofthearrayassigned
*tothem,andthechildprocessessendtheirpartialsumsto
*themaster,whocalculatesagrandtotal.
**/

#include<stdio.h>
#include<mpi.h>

#definemax_rows100000
#definesend_data_tag2001
#definereturn_data_tag2002

intarray[max_rows];
intarray2[max_rows];

main(intargc,char**argv)
{
longintsum,partial_sum;
MPI_Statusstatus;
intmy_id,root_process,ierr,i,num_rows,num_procs,
an_id,num_rows_to_receive,avg_rows_per_process,
sender,num_rows_received,start_row,end_row,num_rows_to_send;

/*Nowreplictethisprocesstocreateparallelprocesses.
*Fromthispointon,everyprocessexecutesaseperatecopy
*ofthisprogram*/

http://condor.cc.ku.edu/~grobe/docs/introMPIC.shtml 8/18
3/29/2016 IntroductiontotheMessagePassingInterface(MPI)usingC

ierr=MPI_Init(&argc,&argv);

root_process=0;

/*findoutMYprocessID,andhowmanyprocesseswerestarted.*/

ierr=MPI_Comm_rank(MPI_COMM_WORLD,&my_id);
ierr=MPI_Comm_size(MPI_COMM_WORLD,&num_procs);

if(my_id==root_process){

/*Imustbetherootprocess,soIwillquerytheuser
*todeterminehowmanynumberstosum.*/

printf("pleaseenterthenumberofnumberstosum:");
scanf("%i",&num_rows);

if(num_rows>max_rows){
printf("Toomanynumbers.\n");
exit(1);
}

avg_rows_per_process=num_rows/num_procs;

/*initializeanarray*/

for(i=0;i<num_rows;i++){
array[i]=i+1;
}

/*distributeaportionofthebectortoeachchildprocess*/

for(an_id=1;an_id<num_procs;an_id++){
start_row=an_id*avg_rows_per_process+1;
end_row=(an_id+1)*avg_rows_per_process;

if((num_rowsend_row)<avg_rows_per_process)
end_row=num_rows1;

num_rows_to_send=end_rowstart_row+1;

ierr=MPI_Send(&num_rows_to_send,1,MPI_INT,
an_id,send_data_tag,MPI_COMM_WORLD);

ierr=MPI_Send(&array[start_row],num_rows_to_send,MPI_INT,
an_id,send_data_tag,MPI_COMM_WORLD);
}

/*andcalculatethesumofthevaluesinthesegmentassigned
*totherootprocess*/

sum=0;
for(i=0;i<avg_rows_per_process+1;i++){
sum+=array[i];
}

printf("sum%icalculatedbyrootprocess\n",sum);

/*and,finally,Icolletthepartialsumsfromtheslaveprocesses,
*printthem,andaddthemtothegrandsum,andprintit*/

for(an_id=1;an_id<num_procs;an_id++){
http://condor.cc.ku.edu/~grobe/docs/introMPIC.shtml 9/18
3/29/2016 IntroductiontotheMessagePassingInterface(MPI)usingC

for(an_id=1;an_id<num_procs;an_id++){

ierr=MPI_Recv(&partial_sum,1,MPI_LONG,MPI_ANY_SOURCE,
return_data_tag,MPI_COMM_WORLD,&status);

sender=status.MPI_SOURCE;

printf("Partialsum%ireturnedfromprocess%i\n",partial_sum,sender);

sum+=partial_sum;
}

printf("Thegrandtotalis:%i\n",sum);
}

else{

/*Imustbeaslaveprocess,soImustreceivemyarraysegment,
*storingitina"local"array,array1.*/

ierr=MPI_Recv(&num_rows_to_receive,1,MPI_INT,
root_process,send_data_tag,MPI_COMM_WORLD,&status);

ierr=MPI_Recv(&array2,num_rows_to_receive,MPI_INT,
root_process,send_data_tag,MPI_COMM_WORLD,&status);

num_rows_received=num_rows_to_receive;

/*Calculatethesumofmyportionofthearray*/

partial_sum=0;
for(i=0;i<num_rows_received;i++){
partial_sum+=array2[i];
}

/*andfinally,sendmypartialsumtohterootprocess*/

ierr=MPI_Send(&partial_sum,1,MPI_LONG,root_process,
return_data_tag,MPI_COMM_WORLD);
}
ierr=MPI_Finalize();
}

Thefollowingtableshowsthevaluesofseveralvariablesduringtheexecutionofsumarray_mpi.The
informationcomesfromatwoprocessorparallelrun,andthevaluesofprogramvariablesareshownin
bothprocessormemoryspaces.NotethatthereisonlyoneprocessactivepriortothecalltoMPI_Init.

Valuehistoriesofselectedvariables
withinthemasterandslaveprocesses
duringa2processexecutionofprogramsumarray_mpi
Before After After
Program Before After
MPI_Send MPI_Recv MPI_Recv
location MPI_Init MPI_Init
toslave byslave bymaster
Variable
Proc0 Proc0 Proc1 Proc0 Proc1 Proc0 Proc1 Proc0 Proc1
Name
root_process 0 0 0 0 0 0 0 0 0

http://condor.cc.ku.edu/~grobe/docs/introMPIC.shtml 10/18
3/29/2016 IntroductiontotheMessagePassingInterface(MPI)usingC

my_id . 0 1 0 1 0 1 0 1
num_procs . 2 2 2 2 2 2 2 2
num_rows . . . 6 . 6 . 6 .
avg_rows_
. . . 3 . 3 . 3 .
per_process
num_rows_
. . . . . . 3 . 3
received
array[0] . . . 1.0 . 1.0 . 1.0 .
array[1] . . . 2.0 . 2.0 . 2.0 .
array[2] . . . 3.0 . 3.0 . 3.0 .
array[3] . . . 4.0 . 4.0 . 4.0 .
array[4] . . . 5.0 . 5.0 . 5.0 .
array[5] . . . 6.0 . 6.0 . 6.0 .
array2[0] . . . . . . 4.0 . 4.0
array2[1] . . . . . . 5.0 . 5.0
array2[2] . . . . . . 6.0 . 6.0
array2[3] . . . . . . . . .
array2[4] . . . . . . . . .
array2[5] . . . . . . . . .
partial_sum . . . . . . . 6.0 15.0
sum . . . . . . . 21.0 .

LoggingandtracingMPIactivity
ItispossibletousempiruntorecordMPIactivity,byusingtheoptionsmpilogandmpitrace.Formore
informationaboutthisfacilityseemanmpirun.

Collectiveoperations
MPI_SendandMPI_Recvare"pointtopoint"communicationsfunctions.Thatis,theyinvolveone
senderandonereceiver.MPIincludesalargenumberofsubroutinesforperforming"collective"
operations.CollectiveoperationareperformedbyMPIroutinesthatarecalledbyeachmemberofa
groupofprocessesthatwantsomeoperationtobeperformedforthemasagroup.Acollectivefunction
mayspecifyonetomany,manytoone,ormanytomanymessagetransmission.

MPIsupportsthreeclassesofcollectiveoperations:

synchronization,
datamovement,and
collectivecomputation

Theseclassesarenotmutuallyexclusive,ofcourse,sinceblockingdatamovementfunctionsalsoserve
tosynchronizeprocessactivity,andsomeMPIroutinesperformbothdatamovementandcomputation.
http://condor.cc.ku.edu/~grobe/docs/introMPIC.shtml 11/18
3/29/2016 IntroductiontotheMessagePassingInterface(MPI)usingC

Synchronization
TheMPI_Barrierfunctioncanbeusedtosynchronizeagroupofprocesses.Tosynchronizeagroupof
processes,eachonemustcallMPI_Barrierwhenithasreachedapointwhereitcangonofurtheruntilit
knowsthatallitscohortshavereachedthesamepoint.OnceaprocesshascalledMPI_Barrier,itwillbe
blockeduntilallprocessesinthegrouphavealsocalledMPI_Barrier.

Collectivedatamovement
Thereareseveralroutinesforperformingcollectivedatadistributiontasks:

MPI_Bcast
Broadcastdatatootherprocesses
MPI_Gather,MPI_Gatherv
Gatherdatafromparticipatingprocessesintoasinglestructure
MPI_Scatter,MPI_Scatter
Breakastructureintoportionsanddistributethoseportionstootherprocesses
MPI_Allgather,MPI_Allgatherv
Gatherdatafromdifferentprocessesintoasinglestructurethatisthensenttoallparticipants
(Gathertoall)
MPI_Alltoall,MPI_Alltoallv
Gatherdataandthenscatterittoallparticipants(Alltoallscatter/gather)

Theroutineswith"V"suffixesmovevariablesizedblocksofdata.

ThesubroutineMPI_Bcastsendsamessagefromoneprocesstoallprocessesinacommunicator.

intMPI_Bcast(void*data_to_be_sent,intsend_count,MPI_Datatypesend_type,
intbroadcasting_process_ID,MPI_Commcomm);

Whenprocessesarereadytoshareinformationwithotherprocessesaspartofabroadcast,ALLofthem
mustexecuteacalltoMPI_BCAST.ThereisnoseparateMPIcalltoreceiveabroadcast.

MPI_Bcastcouldhavebeenusedintheprogramsumarray_mpipresentedearlier,inplaceofthe
MPI_Sendloopthatdistributeddatatoeachprocess.Doingsowouldhaveresultedinexcessivedata
movement,ofcourse.AbettersolutionwouldbeMPI_ScatterorMPI_Scatterv.

ThesubroutinesMPI_ScatterandMPI_Scattervtakeaninputarray,breaktheinputdataintoseparate
portionsandsendaportiontoeachoneoftheprocessesinacommunicatinggroup.

intMPI_Scatter(void*send_data,intsend_count,MPI_Datatypesend_type,
void*receive_data,intreceive_count,MPI_Datatypereceive_type,
intsending_process_ID,MPI_Commcomm);
or

intMPI_Scatterv(void*send_data,int*send_count_array,int*send_start_array,
MPI_Datatypesend_type,void*receive_data,intreceive_count,
MPI_Datatypereceive_type,intsender_process_ID,MPI_Commcomm);

data_to_send:variableofaCtypethatcorrespondstotheMPIsend_typesuppliedbelow
send_count:numberofdataelementstosend(int)
send_count_array:arraywithanentryforeachparticipatingprocesscontainingthenumberofdata
http://condor.cc.ku.edu/~grobe/docs/introMPIC.shtml 12/18
3/29/2016 IntroductiontotheMessagePassingInterface(MPI)usingC

elementstosendtothatprocess(int)
send_start_array:arraywithanentryforeachparticipatingprocesscontainingthedisplacement
relativetothestartofdata_to_sendforeachdatasegmenttosend(int)
send_type:datatypeofelementstosend(oneoftheMPIdatatypehandles)

receive_data:variableofaCtypethatcorrespondstotheMPIreceive_typesuppliedbelow
receive_count:numberofdataelementstoreceive(int)
receive_type:datatypeofelementstoreceive(oneoftheMPIdatatypehandles)
sender_ID:processIDofthesender(int)
receive_tag:receivetag(int)
comm:communicator(handle)
status:statusobject(MPI_Status)

TheroutineMPI_Scattervcouldhavebeenusedintheprogramsumarray_mpipresentedearlier,inplace
oftheMPI_Sendloopthatdistributeddatatoeachprocess.

MPI_Bcast,MPI_Scatter,andothercollectiveroutinesbuildacommunicationtreeamongthe
participatingprocessestominimizemessagetraffic.IfthereareNprocessesinvolved,therewould
normallybeN1transmissionsduringabroadcastoperation,butifatreeisbuiltsothatthebroadcasting
processsendsthebroadcastto2processes,andtheyeachsenditonto2otherprocesses,thetotalnumber
ofmessagestransferredisonlyO(lnN).

Collectivecomputationroutines
Collectivecomputationissimilartocollectivedatamovementwiththeadditionalfeaturethatdatamay
bemodifiedasitismoved.Thefollowingroutinescanbeusedforcollectivecomputation.

MPI_Reduce
Performareductionoperation.Thatis,applysomeoperationtosomeoperandinevery
participatingprocess.Forexample,addanintegerresidingineveryprocesstogetherandputthe
resultinaprocessspecifiedintheMPI_Reduceargumentlist.
MPI_Allreduce
Performareductionleavingtheresultinallparticipatingprocesses
MPI_Reduce_scatter
Performareductionandthenscattertheresult
MPI_Scan
Performareductionleavingpartialresults(computeduptothepointofaprocess'sinvolvementin
thereductiontreetraversal)ineachparticipatingprocess.(parallelprefix)

ThesubroutineMPI_Reducecombinesdatafromallprocessesinacommunicatorusingoneofseveral
reductionoperationstoproduceasingleresultthatappearsinaspecifiedtargetprocess.
intMPI_Reduce(void*data_to_be_sent,void*result_to_be_received_by_target,
intsend_count,MPI_Datatypesend_type,MPI_Opoperation,
inttarget_process_ID,MPI_Commcomm);

Whenprocessesarereadytoshareinformationwithotherprocessesaspartofadatareduction,allofthe
participatingprocessesexecuteacalltoMPI_Reduce,whichuseslocaldatatocalculateeachprocess's
portionofthereductionoperationandcommunicatesthelocalresulttootherprocessesasnecessary.
Onlythetarget_process_IDreceivesthefinalresult.

http://condor.cc.ku.edu/~grobe/docs/introMPIC.shtml 13/18
3/29/2016 IntroductiontotheMessagePassingInterface(MPI)usingC

MPI_Reducecouldhavebeenusedintheprogramsumarray_mpipresentedearlier,inplaceofthe
MPI_Recvloopthatcollectedpartialsumsfromeachprocess.

Collectivecomputationbuiltinoperations
ManyoftheMPIcollectivecomputationroutinestakebothbuiltinanduserdefinedcombination
functions.Thebuiltinfunctionsare:

Operationhandle Operation
MPI_MAX Maximum
MPI_MIN Minimum
MPI_PROD Product
MPI_SUM Sum
MPI_LAND LogicalAND
MPI_LOR LogicalOR
MPI_LXOR LogicalExclusiveOR
MPI_BAND BitwiseAND
MPI_BOR BitwiseOR
MPI_BXOR BitwiseExclusiveOR
MPI_MAXLOC Maximumvalueandlocation
MPI_MINLOC Minimumvalueandlocation

Acollectiveoperationexample
Thefollowingprogramintegratesthefunctionsin(X)overtherange0to2pi.Itwillbefollowedbya
parallelversionofthesameprogramthatusestheMPIlibrary.
/*programtointegratesin(x)between0andpibycomputing
*theareaofanumberofrectangleschosensoastoapproximate
*theshapeunderthecurveofthefunction.
*
*1)asktheusertochoosethenumberofintervals,
*2)computetheintervalwidth(rect_width),
*3)foreachinterval:
*
*a)findthemiddleoftheinterval(x_middle),
*b)computetheheightoftherectangle,sin(x_middle),
*c)findtheareaoftherectangleastheproductof
*theintervalwidthanditsheightsin(x_middle),and
*d)incrementarunningtotal.
**/

#include<stdio.h>
#include<math.h>

#definePI3.1415926535

main(intargc,char**argv)
{
http://condor.cc.ku.edu/~grobe/docs/introMPIC.shtml 14/18
3/29/2016 IntroductiontotheMessagePassingInterface(MPI)usingC

inti,num_intervals;
doublerect_width,area,sum,x_middle;

printf("Pleaseenterthenumberofintervalstointerpolate:");
scanf("%i",&num_intervals);

rect_width=PI/num_intervals;

sum=0;
for(i=1;i<num_intervals+1;i++){

/*findthemiddleoftheintervalontheXaxis.*/

x_middle=(i0.5)*rect_width;
area=sin(x_middle)*rect_width;
sum=sum+area;
}

printf("Thetotalareais:%f\n",(float)sum);
}

ThenextprogramisanMPIversionoftheprogramabove.ItusesMPI_Bcasttosendinformationto
eachparticipatingprocessandMPI_Reducetogetagrandtotaloftheareascomputedbyeach
participatingprocess.

/*Thisprogramintegratessin(x)between0andpibycomputing
*theareaofanumberofrectangleschosensoastoapproximate
*theshapeunderthecurveofthefunctionusingMPI.
*
*Therootprocessactsasamastertoagroupofchildprocess
*thatactasslaves.Themasterpromptsforthenumberof
*interpolationsandbroadcaststhatvaluetoeachslave.
*
*Therearenum_procsprocessesalltogether,andaprocess
*computestheareadefinedbyeverynum_procsthinterval,
*collectsapartialsumofthoseareas,andsendsitspartial
*sumtotherootprocess.
**/

#include<stdio.h>
#include<math.h>
#include<mpi.h>

#definePI3.1415926535

main(intargc,char**argv)
{
intmy_id,root_process,num_procs,ierr,num_intervals,i;
doublerect_width,area,sum,x_middle,partial_sum;
MPI_Statusstatus;

/*Letprocess0betherootprocess.*/

root_process=0;

/*Nowreplicatethisprocesstocreateparallelprocesses.*/

ierr=MPI_Init(&argc,&argv);

/*FindoutMYprocessID,andhowmanyprocesseswerestarted.*/
http://condor.cc.ku.edu/~grobe/docs/introMPIC.shtml 15/18
3/29/2016 IntroductiontotheMessagePassingInterface(MPI)usingC

ierr=MPI_Comm_rank(MPI_COMM_WORLD,&my_id);
ierr=MPI_Comm_size(MPI_COMM_WORLD,&num_procs);

if(my_id==root_process){

/*Imustbetherootprocess,soIwillquerytheuser
todeterminehowmanyinterpolationintervalstouse.*/

printf("Pleaseenterthenumberofintervalstointerpolate:");
scanf("%i",&num_intervals);
}

/*Then...nomatterwhichprocessIam:
*
*Iengageinabroadcastsothatthenumberofintervalsis
*sentfromtherootprocesstotheotherprocesses,and...
**/
ierr=MPI_Bcast(&num_intervals,1,MPI_INT,root_process,
MPI_COMM_WORLD);

/*calculatethewidthofarectangle,and*/

rect_width=PI/num_intervals;

/*thencalculatethesumoftheareasoftherectanglesfor
*whichIamresponsible.Startwiththe(my_id+1)th
*intervalandprocesseverynum_procsthintervalthereafter.
**/
partial_sum=0;
for(i=my_id+1;i<num_intervals+1;i+=num_procs){

/*FindthemiddleoftheintervalontheXaxis.*/
x_middle=(i0.5)*rect_width;
area=sin(x_middle)*rect_width;
partial_sum=partial_sum+area;
}
printf("proc%icomputes:%f\n",my_id,(float)partial_sum);

/*andfinally,engageinareductioninwhichallpartialsums
*arecombined,andthegrandsumappearsinvariable"sum"in
*therootprocess,
**/
ierr=MPI_Reduce(&partial_sum,&sum,1,MPI_DOUBLE,
MPI_SUM,root_process,MPI_COMM_WORLD);

/*and,ifIamtherootprocess,printtheresult.*/

if(my_id==root_process){
printf("Theintegralis%f\n",(float)sum);

/*(yes,wecouldhavesummedjusttheheights,and
*postponedthemultiplicationbyrect_widthtilnow.)*/
}

/*Closedownthisprocesses.*/

ierr=MPI_Finalize();
}

http://condor.cc.ku.edu/~grobe/docs/introMPIC.shtml 16/18
3/29/2016 IntroductiontotheMessagePassingInterface(MPI)usingC

Simultaneoussendandreceive
ThesubroutineMPI_Sendrecvexchangesmessageswithanotherprocess.Asendreceiveoperationis
usefulforavoidingsomekindsofunsafeinteractionpatternsandforimplementingremoteprocedure
calls.

AmessagesentbyasendreceiveoperationcanbereceivedbyMPI_Recvandasendreceiveoperation
canreceiveamessagesentbyanMPI_Send.
MPI_Sendrecv(&data_to_send,send_count,send_type,destination_ID,send_tag,
&received_data,receive_count,receive_type,sender_ID,receive_tag,
comm,&status)

data_to_send:variableofaCtypethatcorrespondstotheMPIsend_typesuppliedbelow
send_count:numberofdataelementstosend(int)
send_type:datatypeofelementstosend(oneoftheMPIdatatypehandles)
destination_ID:processIDofthedestination(int)
send_tag:sendtag(int)
received_data:variableofaCtypethatcorrespondstotheMPIreceive_typesuppliedbelow
receive_count:numberofdataelementstoreceive(int)
receive_type:datatypeofelementstoreceive(oneoftheMPIdatatypehandles)
sender_ID:processIDofthesender(int)
receive_tag:receivetag(int)
comm:communicator(handle)
status:statusobject(MPI_Status)

MPItags
MPI_SendandMPI_Recv,aswellasotherMPIroutines,allowtheusertospecifyatagvaluewitheach
transmission.Thesetagvaluesmaybeusedtospecifythemessagetype,or"context,"inasituation
whereaprogrammayreceivemessagesofseveraltypesduringthesameprogram.Thereceiversimply
checksthetagvaluetodecidewhatkindofmessageithasreceived.

MPIcommunicators
EveryMPIcommunicationoperationinvolvesa"communicator."Communicatorsidentifythegroupof
processesinvolvedinacommunicationoperationand/orthecontextinwhichitoccurs.Thesourceand
destinationprocessesspecifiedinpointtopointroutineslikeMPI_SendandMPI_Recvmustbe
membersofthespecifiedcommunicatorandthetwocallsmustreferencethesamecommunicator.

Collectiveoperationsincludejustthoseprocessesidentifiedbythecommunicatorspecifiedinthecalls.

ThecommunicatorMPI_COMM_WORLDisdefinedbydefaultforallMPIruns,andincludesall
processesdefinedbyMPI_Initduringthatrun.Additionalcommunicatorscanbedefinedthatincludeall
orpartofthoseprocesses.Forexample,supposeagroupofprocessesneedstoengageintwodifferent
reductionsinvolvingdisjointsetsofprocesses.Acommunicatorcanbedefinedforeachsubsetof
MPI_COMM_WORLDandspecifiedinthetworeductioncallstomanagemessagetransmission.

MPI_Comm_splitcanbeusedtocreateanewcommunicatorcomposedofasubsetofanother
communicator.MPI_Comm_dupcanbeusedtocreateanewcommunicatorcomposedofallofthe
http://condor.cc.ku.edu/~grobe/docs/introMPIC.shtml 17/18
3/29/2016 IntroductiontotheMessagePassingInterface(MPI)usingC

membersofanothercommunicator.Thismaybeusefulformanaginginteractionswithinasetof
processesinplaceofmessagetags.

Moreinformation
ThisshortintroductionomitsmanyMPItopicsandroutines,andcoversmosttopicsonlylightly.In
particular,itomitsdiscussionsoftopologies,unsafecommunicationandnonblockingcommunication.

Foradditionalinformationconcerningtheseandothertopicspleaseconsult:

themajorMPIWebsite,whereyouwillfindversionsofthestandards:
http://www.mcs.anl.gov/mpi
thebooks:
Gropp,Lusk,andSkjellum,UsingMPI:PortableParallelProgrammingwiththeMessage
PassingInterface,MITPress,1994
Foster,Ian,DesigningandBuildingParallelPrograms,availableinbothhardcopy
(AddisonWesleyPublishingCo.,1994)andonlineversions,
Pacheco,Peter,AUser'sGuidetoMPI,whichgivesatutorialintroductionextendedtocover
derivedtypes,communicatorsandtopologies,or
thenewsgroupcomp.parallel.mpi

Exercises
HerearesomeexercisesforcontinuingyourinvestigationofMPI:

Convertthehelloworldprogramtoprintitsmessagesinrankorder.
Converttheexampleprogramsumarray_mpitouseMPI_Scatterand/orMPI_Reduce.
Writeaprogramtofindallpositiveprimesuptosomemaximumvalue,usingMPI_Recvto
receiverequestsforintegerstotest.Themasterwillloopfrom2tothemaximumvalueon
1.issueMPI_Recvandwaitforamessagefromanyslave(MPI_ANY_SOURCE),
2.ifthemessageiszero,theprocessisjuststarting,
ifthemessageisnegative,itisanonprime,
ifthemessageispositive,itisaprime.
3.useMPI_Sendtosendanumbertotest.
andeachslavewillsendarequestforanumbertothemaster,receiveanintegertotest,testit,and
returnthatintegerifitisprime,butitsnegativevalueifitisnotprime.
Writeaprogramtosendatokenfromprocessortoprocessorinaloop.

Documentpreparedby:

DanielThomassetand
MichaelGrobe
AcademicComputingServices
TheUniversityofKansas

withassistanceandoverheadsprovidedby
TheNationalComputationalScienceAlliance(NCSA)at
TheUniversityofIllinois

http://condor.cc.ku.edu/~grobe/docs/introMPIC.shtml 18/18