4/1/12

Maximum entropy probability distribution - Wikipedia, the free encyclopedia - http://en

Maximumentropyprobabilitydistribution
FromWikipedia,thefreeencyclopedia

Instatisticsandinformationtheory,amaximumentropyprobabilitydistributionisaprobability distributionwhoseentropyisatleastasgreatasthatofallothermembersofaspecifiedclassofdistributions. Accordingtotheprincipleofmaximumentropy,ifnothingisknownaboutadistributionexceptthatit belongstoacertainclass,thenthedistributionwiththelargestentropyshouldbechosenasthedefault.The motivationistwofold:first,maximizingentropyminimizestheamountofpriorinformationbuiltintothe distributionsecond,manyphysicalsystemstendtomovetowardsmaximalentropyconfigurationsovertime.

Contents
1Definitionofentropy 2Examplesofmaximumentropydistributions 2.1Givenmeanandstandarddeviation:thenormaldistribution 2.2Uniformandpiecewiseuniformdistributions 2.3Positiveandgivenmean:theexponentialdistribution 2.4Discretedistributionswithgivenmean 2.5Circularrandomvariables 3AtheorembyBoltzmann 3.1Continuousversion 3.2Discreteversion 3.3Caveats 4Seealso 5Notes 6References

Definitionofentropy
Furtherinformation:Entropy(informationtheory) IfXisadiscreterandomvariablewithdistributiongivenby

thentheentropyofXisdefinedas

IfXisacontinuousrandomvariablewithprobabilitydensityp(x),thentheentropyofXissometimesdefined

en.wikipedia.org/wiki/Maximum_entropy_probability_distribution#A_theorem_by_Boltz

1/6

4/1/12

Maximum entropy probability distribution - Wikipedia, the free encyclopedia - http://en

as[1][2][3]

wherep(x)logp(x)isunderstoodtobezerowheneverp(x)=0.Inconnectionwithmaximumentropy distributions,thisformofdefinitionisoftentheonlyonegiven,oratleastitistakenasthestandardform. However,itisrecognisableasthespecialcasem=1ofthemoregeneraldefinition

whichisdiscussedinthearticlesEntropy(informationtheory)andPrincipleofmaximumentropy. Thebaseofthelogarithmisnotimportantaslongasthesameoneisusedconsistently:changeofbasemerely resultsinarescalingoftheentropy.Informationtheoreticiansmayprefertousebase2inordertoexpressthe entropyinbitsmathematiciansandphysicistswilloftenpreferthenaturallogarithm,resultinginaunitofnats ornepersfortheentropy.

Examplesofmaximumentropydistributions
AtableofexamplesofmaximumentropydistributionsisgiveninPark&Bera(2009)[4]

Givenmeanandstandarddeviation:thenormaldistribution
ThenormaldistributionN(,2)hasmaximumentropyamongallrealvalueddistributionswithspecified meanandstandarddeviation.Therefore,theassumptionofnormalityimposestheminimalpriorstructural constraintbeyondthesemoments.(Seethedifferentialentropyarticleforaderivation.)

Uniformandpiecewiseuniformdistributions
Theuniformdistributionontheinterval[a,b]isthemaximumentropydistributionamongallcontinuous distributionswhicharesupportedintheinterval[a,b](whichmeansthattheprobabilitydensityis0outsideof theinterval). Moregenerally,ifwe'regivenasubdivisiona=a0<a1<...<ak=boftheinterval[a,b]andprobabilities p1,...,pkwhichadduptoone,thenwecanconsidertheclassofallcontinuousdistributionssuchthat

Thedensityofthemaximumentropydistributionforthisclassisconstantoneachoftheintervals[aj1,aj)it lookssomewhatlikeahistogram. Theuniformdistributiononthefiniteset{x1,...,xn}(whichassignsaprobabilityof1/ntoeachofthese values)isthemaximumentropydistributionamongalldiscretedistributionssupportedonthisset.

en.wikipedia.org/wiki/Maximum_entropy_probability_distribution#A_theorem_by_Boltz

2/6

4/1/12

Maximum entropy probability distribution - Wikipedia, the free encyclopedia - http://en

Positiveandgivenmean:theexponentialdistribution
Theexponentialdistributionwithmean1/isthemaximumentropydistributionamongallcontinuous distributionssupportedin[0,)thathaveameanof1/. Inphysics,thisoccurswhengravityactsonagasthatiskeptatconstantpressureandtemperature:ifX describestheheightofamolecule,thenthevariableXisexponentiallydistributed(whichalsomeansthatthe densityofthegasdependsonheightproportionaltotheexponentialdistribution).Thereason:Xisclearly positiveanditsmean,whichcorrespondstotheaveragepotentialenergy,isfixed.Overtime,thesystemwill attainitsmaximumentropyconfiguration,accordingtothesecondlawofthermodynamics.

Discretedistributionswithgivenmean
Amongallthediscretedistributionssupportedontheset{x1,...,xn}withmean,themaximumentropy distributionhasthefollowingshape:

wherethepositiveconstantsCandrcanbedeterminedbytherequirementsthatthesumofallthe probabilitiesmustbe1andtheexpectedvaluemustbe. Forexample,ifalargenumberNofdiceisthrown,andyouaretoldthatthesumofalltheshownnumbersis S.Basedonthisinformationalone,whatwouldbeareasonableassumptionforthenumberofdiceshowing1, 2,...,6?Thisisaninstanceofthesituationconsideredabove,with{x1,...,x6}={1,...,6}and=S/N. Finally,amongallthediscretedistributionssupportedontheinfiniteset{x1,x2,...}withmean,themaximum entropydistributionhastheshape:

whereagaintheconstantsCandrweredeterminedbytherequirementsthatthesumofalltheprobabilities mustbe1andtheexpectedvaluemustbe.Forexample,inthecasethatxk=k,thisgives

Circularrandomvariables
Foracontinuousrandomvariable distributedabouttheunitcircle,theVonMisesdistributionmaximizes theentropywhengiventherealandimaginarypartsofthefirstcircularmoment[5]or,equivalently,the circularmeanandcircularvariance. Whengiventhemeanandvarianceoftheangles modulo theentropy.[5] ,thewrappednormaldistributionmaximizes

en.wikipedia.org/wiki/Maximum_entropy_probability_distribution#A_theorem_by_Boltz

3/6

4/1/12

Maximum entropy probability distribution - Wikipedia, the free encyclopedia - http://en

AtheorembyBoltzmann
AlltheaboveexamplesareconsequencesofthefollowingtheorembyLudwigBoltzmann.

Continuousversion
SupposeSisaclosedsubsetoftherealnumbersRandwe'regivennmeasurablefunctionsf1,...,fnandn numbersa1,...,an.WeconsidertheclassCofallcontinuousrandomvariableswhicharesupportedonS(i.e. whosedensityfunctioniszerooutsideofS)andwhichsatisfythenexpectedvalueconditions

IfthereisamemberinCwhosedensityfunctionispositiveeverywhereinS,andifthereexistsamaximal entropydistributionforC,thenitsprobabilitydensityp(x)hasthefollowingshape:

wheretheconstantscandjhavetobedeterminedsothattheintegralofp(x)overSis1andtheabove conditionsfortheexpectedvaluesaresatisfied. Conversely,ifconstantscandjlikethiscanbefound,thenp(x)isindeedthedensityofthe(unique) maximumentropydistributionforourclassC. ThistheoremisprovedwiththecalculusofvariationsandLagrangemultipliers.

Discreteversion
SupposeS={x1,x2,...}isa(finiteorinfinite)discretesubsetoftherealsandwe'regivennfunctionsf1,...,fn andnnumbersa1,...,an.WeconsidertheclassCofalldiscreterandomvariablesXwhicharesupportedonS andwhichsatisfythenconditions

IfthereexistsamemberofCwhichassignspositiveprobabilitytoallmembersofSandifthereexistsa maximumentropydistributionforC,thenthisdistributionhasthefollowingshape:

wheretheconstantscandjhavetobedeterminedsothatthesumoftheprobabilitiesis1andtheabove conditionsfortheexpectedvaluesaresatisfied.

en.wikipedia.org/wiki/Maximum_entropy_probability_distribution#A_theorem_by_Boltz

4/6

4/1/12

Maximum entropy probability distribution - Wikipedia, the free encyclopedia - http://en

Conversely,ifconstantscandjlikethiscanbefound,thentheabovedistributionisindeedthemaximum entropydistributionforourclassC. ThisversionofthetheoremcanbeprovedwiththetoolsofordinarycalculusandLagrangemultipliers.

Caveats
Notethatnotallclassesofdistributionscontainamaximumentropydistribution.Itispossiblethataclass containdistributionsofarbitrarilylargeentropy(e.g.theclassofallcontinuousdistributionsonRwithmean0 butarbitrarystandarddeviation),orthattheentropiesareboundedabovebutthereisnodistributionwhich attainsthemaximalentropy(e.g.theclassofallcontinuousdistributionsXonRwithE(X)=0andE(X2)= E(X3)=1[citationneeded]). ItisalsopossiblethattheexpectedvaluerestrictionsfortheclassCforcetheprobabilitydistributiontobezero incertainsubsetsofS.Inthatcaseourtheoremdoesn'tapply,butonecanworkaroundthisbyshrinkingthe setS.

Seealso
Exponentialfamily Gibbsmeasure Partitionfunction(mathematics)

Notes
1. ^Williams,D.(2001)WeighingtheOddsCambridgeUPISBN052100618x(pages197199) 2. ^Bernardo,J.M.,Smith,A.F.M.(2000)BayesianTheory'.'Wiley.ISBN047149464x(pages209,366) 3. ^O'Hagan,A.(1994)Kendall'sAdvancedTheoryofstatistics,Vol2B,BayesianInference,EdwardArnold. ISBN0340529229(Section5.40) 4. ^Park,SungY.Bera,AnilK.(2009)."Maximumentropyautoregressiveconditionalheteroskedasticitymodel" (http://www.wise.xmu.edu.cn/Master/Download/..%5C..%5CUploadFiles%5Cpaper masterdownload%5C2009519932327055475115776.pdf).JournalofEconometrics(Elsevier):219230. http://www.wise.xmu.edu.cn/Master/Download/..%5C..%5CUploadFiles%5Cpaper masterdownload%5C2009519932327055475115776.pdf.Retrieved20110602. 5. ^abJammalamadaka,S.RaoSenGupta,A.(2001).Topicsincircularstatistics (http://books.google.com/books? id=sKqWMGqQXQkC&printsec=frontcover&dq=Jammalamadaka+Topics+in+circular&hl=en&ei=iJ3QTe77NK L00gGdyqHoDQ&sa=X&oi=book_result&ct=result&resnum=1&ved=0CDcQ6AEwAA#v=onepage&q&f=false). NewJersey:WorldScientific.ISBN9810237782.http://books.google.com/books? id=sKqWMGqQXQkC&printsec=frontcover&dq=Jammalamadaka+Topics+in+circular&hl=en&ei=iJ3QTe77NK L00gGdyqHoDQ&sa=X&oi=book_result&ct=result&resnum=1&ved=0CDcQ6AEwAA#v=onepage&q&f=false. Retrieved20110515.

References
T.M.CoverandJ.A.Thomas,ElementsofInformationTheory,1991.Chapter11.

en.wikipedia.org/wiki/Maximum_entropy_probability_distribution#A_theorem_by_Boltz

5/6

4/1/12

Maximum entropy probability distribution - Wikipedia, the free encyclopedia - http://en

I.J.Taneja,GeneralizedInformationMeasuresandTheirApplications (http://www.mtm.ufsc.br/~taneja/book/book.html)2001.Chapter1 (http://www.mtm.ufsc.br/~taneja/book/node14.html) Retrievedfrom"http://en.wikipedia.org/w/index.php? title=Maximum_entropy_probability_distribution&oldid=481601486" Categories: Entropyandinformation Continuousdistributions Discretedistributions Typesofprobabilitydistributions

Particlestatistics

Thispagewaslastmodifiedon13March2012at00:09. TextisavailableundertheCreativeCommonsAttributionShareAlikeLicenseadditionaltermsmay apply.SeeTermsofusefordetails. WikipediaisaregisteredtrademarkoftheWikimediaFoundation,Inc.,anonprofitorganization.

en.wikipedia.org/wiki/Maximum_entropy_probability_distribution#A_theorem_by_Boltz

6/6