SourceCoding

Whatissourcecoding?
Communicationsystemsaredesignedtotransmitthe
informationgeneratedbyasourcetosome
destination
Adigitalcommunicationsystemisdesignedto
transmitinformationindigitalform
Sourcecoding:theconversionofthesourceoutput
toadigitalform,performedbyasourceencoder,
whichproducesasequenceofbinarydigits
Designgoal:representasourcewiththefewestbits
suchthatbestrecoveryofthesourcefromthe
compresseddataispossible
ELEC5360
2
ELEC5360
3
Source
Encoder
Information
Source
Channel
Encoder
Modulator
Channel
Noise
Source
Decoder
Received
Information
Channel
Decoder
Demodulator
Binaryinterface
Differentinformationsources
Discretesources
Asequenceofsymbolsfromaknowndiscretealphabet
Thealphabetcouldbe,e.g.,binarydigits,Englishletters
Analogsequencesources
Asequenceofreal/complexnumbers
Analogwaveformsources
Ananalogwaveform,e.g.,voicesignal,videowaveform
ELEC5360
4
Differentinformationsources
Sourcecodingforadiscretesource
Theinformationsourcecanbeuniquelyretrievedfromthe
encodedstringofbits
Uniquelydecodable
Losslesscoding
Sourcecodingforanalogsources
Someofquantizationisnecessary
Itintroducesdistortion
Lossycompression
Atradeoffbetweenthebitrateandthedistortion
ELEC5360
5
AGeneralDiagramforSourceCoding
Sampler
Input
waveform
Quantizer
Discrete
encoder
Binary
Channel
Analogfilter
Output
waveform
Tablelookup
Discrete
decoder
Binary
interface
Analog
sequence
Symbol
sequence
ELEC5360
6
Lossless
coding
Lossy
compression
APracticalExample:PCM
Pulsecodemodulation(PCM)
Digitalrepresentationofananalogsignal
ThestandardformfordigitalaudioandvariousBluray,DVDandCD
formats
Operation
Samplethemagnitudeoftheanalogsignalregularlyatuniform
intervals
Eachsampleisthenquantizedtothenearestvaluewithinarangeof
digitalsteps
ThePCMprocessiscommonlyimplementedonasingle
integratedcircuitgenerallyreferredtoasananalogtodigital
converter (ADC).
ELEC5360
7
PCM
Anexample:
Intelephony,astandardaudiosignalforasinglephonecallis
encodedas8,000analogsamplespersecond,of8bitseach,
givinga64kbit/sdigitalsignal
ELEC5360
8
Sampler
Input
waveform
Quantizer
Discrete
encoder
Binary
Channel
Analogfilter
Output
waveform
Tablelookup
Discrete
decoder
Binary
interface
Analog
sequence
Symbol
sequence
ELEC5360
9
Lossless
coding
DiscreteMemoryless Source(DMS)
Consideradiscreteinformationsourcewithanalphabet
ofM possibleletters,
1 2 M
Notnecessarilynumericalvalues,e.g.,{sunny,cloudy,rainy}
Probabilisticmodel
Discretememoryless source(DMS)
Theoutputsequenceisstatisticallyindependent
i.e.,thecurrentoutputletterisindependentofallpastand
futureoutputs
Essentially,itisasequenceofiid randomvariablesX

ELEC5360
10
P X = x
k
= p
k
, 1 k H
where p
k
M
k=1
= 1
InformationMeasureandCodewordLength
Entropy
AmeasureofuncertaintyorambiguityinX
AmeasureofinformationthatisacquiredbyknowledgeofX
Theaveragecodewordlength
wherel
]
isthelengthofthejth codeword
ELEC5360
11
E X = -p
x
log
2
p
x
xA
I

= l
]
p
]
M
]=1
Symbolbysymbolencoding
x
]
c
]
= c
],1
, c
],2
, c
],I
]
VariablelengthSourceCoding
Uniquelydecodable
Instantaneouscodes withoutanydecodingdelay
Prefixfreecodes nocodewordisaprefixofanyothercodeword
Bothuniquelyandinstantaneouslydecodable
ELEC5360
12
Letter P CodeI CodeII CodeIII CodeIV
x
1
1/2 00 1 0 0
x
2
1/4 01 00 10 01
x
3
1/8 10 01 110 011
x
4
1/8 11 10 111 111
Notuniquely
decodable
Prefixfree
code
PrefixfreeCodes
Ifauniquelydecodablecodeexistswithacertainset
ofcodewordlengths,thenaprefixfreecodecan
easilybeconstructedwiththesamesetoflengths
Thedecodercandecodeeachcodewordofaprefix
freecodeimmediatelyonthearrivalofthelastbitin
thatcodeword,i.e.,instantaneouslydecodable
Givenaprobabilitydistributiononthesource
symbols,itiseasytoconstructaprefixfreecodeof
minimumexpectedlength
ELEC5360
13
Kraftinequalityforprefixfreecodes
Theorem:Everyprefixfreecodeforanalphabet
1 2 M
withcodewordlength
]
satisfiesthe
following
Conversely,ifthisconditionissatisfied,thenaprefix
freecodewithlength
]
exists.
Note:justbecauseacodehaslengthsthatsatisfythis
condition,itdoesnotfollowthatthecodeisprefixfree,or
evenuniquelydecodable
ELEC5360
14
2
-I
]
M
]=1
1
ProofofKraftInequality
Theproofisbasedonbase2expansions
Base2expansion. y
1
y
2
y
I
= y
m
2
-m I
m=1
e.g.,.011represents1/4+1/8
Thebase2expansion. y
1
y
2
y
I
coverstheinterval
y
m
2
-m I
m=1
, y
m
2
-m I
m=1
+ 2
-I
. Thisintervalhassize2
-I
and
includesallnumberswhosebase2expansionsstartwith. y
1
y
2
y
I
Anycodewordoflengthl isrepresentedbyarationalnumberinthe
interval[0,1)andcoversanintervalofsize2
-I
Theprooffollowsfromthisrepresentation.
ELEC5360
15
Minimum forPrefixfreeCodes
Wewanttominimizetheexpectedlengthsubjectto
theKraftinequality
Theexpectedcodewordlengthisrelatedtotheoutput
rateoftheencoder
Entropyboundsforprefixfreecodes
LetI

mn
betheminimumexpectedcodewordlengthover
alltheprefixfreecodesforagiveDMS.Then
I

mn
= E X ifandonlyiseachprobabilityp
]
isaninteger
powerof2.
ELEC5360
16
E X I

mn
< E X + 1 bit/symbol
LagrangeMultiplierSolutionfor
Theproblemoffinding
mn
canbeformulatedas
Wecangetanonintegersolutionofl
]
byusingaLagrange
multiplier
Settingthederivativewithrespecttoeachl
]
equalto0
Wherel
]
= log
2
p
]
I

mn
nonintcgcr isalowerboundofI

mn
l
]
isnotaninteger
ELEC5360
17
I

mn
= min

2
-l
]
M
]=1
<1

l
]
p
]
M
]=1
l
]
p
]
M
]=1
+ 2
-I
]
M
]=1
I

mn
nonintcgcr = -p
]
log
2
p
]
= E X
Upperboundof
Choosethecodeword lengthstobel
]
= -log
2
p
]
Then
Fromthelefthandside,theKraftinequalityissatisfied
Fromtherighthandside
So
ELEC5360
18
HuffmansAlgorithm
Theexpectedcodewordlengthminimizationproblemisan
integeroptimizationproblem,whichingeneralisquite
difficult
Surprisingly,DavidHuffmanproposedaverysimpleand
straightforwardalgorithmforconstructingoptimalprefixfree
codes
Huffmandevelopedthealgorithmin1950asatermpaperinRobert
Fanos informationtheoryclassatMIT
isoptimalinthesensethatthecodewords satisfytheprefixcondition
andtheaverageblocklengthisaminimum.
Applications:JPEG,MP3,etc
ELEC5360
19
HuffmansAlgorithm
Propertiesofoptimalcodes
1. Optimalcodeshavethepropertythatifp

> p
]
,thenl

l
]
2. Optimalprefixfreecodeshavethepropertythattheassociatedcode
treeisfull
3. Optimalprefixfreecodeshavethepropertythat,foreachofthe
longestcodewords inthecode,thesiblingofthatcodeword is
anotherlongestcodeword
4. LetXbearandomsymbolwithapmf satisfyingp
1
p
2
p
M
.
ThereisanoptimalprefixfreecodeforXinwhichthecodewords for
M1andMaresiblingsandhavemaximallengthwithinthecode
ELEC5360
20
Example1
ELEC5360
21
p
1
p
2
p
7
Example2
ELEC5360
22
LempelZivUniversalDataCompression
Huffmancoding
Weneedtoknowthesourcestatistics
Itisnotoptimalwhenthepmf areunknown,not
identicallydistributed,ornotindependent
LempelZivalgorithm
Belongstotheclassofuniversalsourcecodingalgorithms
Doesnotneedtoknowthesourcestatistics
LZ77,usesstringmatchingonaslidingwindow
LZ78,usesanadaptivedictionary
Applications:UNIXcompress,GIF,TIFF,PDF,etc
ELEC5360
23
LZ78
Dictionarybasedcompression
Codeword output
Anindexreferringtothelongestmatchingdictionaryentry
+Thefirstnonmatchingsymbol
Alsoaddthenewstringtothedictionary
Foranewsymbolnotinthedictionary,thecodeword is0+thissymbol
ELEC5360
24
Example:
FixedtovariablelengthSourceCoding
Previously,wediscussedsymbolbysymbolcoding
Wecandobetterbyencodingblocksofsourcesymbols,i.e.,
encodingasupersymbol
Theorem [Prefixfreesourcecodingtheorem]
Foranydiscretememoryless sourcewithentropyH[X],and
anyintegern 1, thereexistsaprefixfreeencodingof
sourcentuplesforwhichtheexpectedcodewordlengthper
sourcesymbol
Furthermore,noprefixfreeencodingoffixedlengthsource
blocksofanylengthnresultsinanexpectedcodewordlength
ELEC5360
25
I

E X + 1n
I

< E X
FixedlengthtofixedlengthSourceCoding
Onemaindisadvantageoffixedtovariablelength
codesisthatbitsleavetheencoderatavariablerate
Iftheincomingsymbolshaveafixedrate,theencodedbits
mustbequeuedandthereisapositiveprobabilityforthe
queuetooverflow
Analternativepointofviewistoconsiderfixed
lengthtofixedlength codes
ELEC5360
26
AsymptoticEquipartition Property(AEP)
AEP:ifX
1
, X
2
, areiid ~p(x),then
Thetypicalsetforanye > u isdefinedas
Alltypicalsequenceshaveroughlythesameprobability
2
-nH(X)
Fornsufficientlylarge,P x, x
2
, , x
n
I
s
> 1 - e
Itcanbeprovedthat
Soroughlythenumberoftypicalsequencesis2
nH(X)
ELEC5360
27
-
1
n
log p X
1
, X
2
, , X
n
E(X) inprobability
I
s
= x, x
2
, , x
n
: -
1
n
log p X
1
, X
2
, , X
n
- E(X) < e
1 -o 2
n H X -s
< I
s
< 2
n H X +s
ProveinHW ProveinHW
AsymptoticEquipartition Property(AEP)
Roughlyspeaking,theAEPsays
Givenaverylongstringofn iid discreterandomsymbols
X
1
,,X
n
,thereexistsatypicalsetofsamplestrings(x
1
,,
x
n
)whoseaggregateprobabilityisalmost1.
Thereareroughly2
nH|X]
typicalstringsoflengthn,and
eachhasaprobabilityroughlyequalto2
-nH|X]
ELEC5360
28
SourceCodingTheorem
Wecanprovideacodewordonlyforeachtypicalsequence
Ifthesequenceisnotatypicalsequence,thenasource
codingfailureisdeclared
Thisprobabilitycanbemadearbitrarilysmallbychoosingn
largeenough
LosslessSourceCodingTheorem
LetX denoteaDMSwithentropyH(X).Thereexistsalossless
sourcecodeforthissourceatanyrateR ifR>H(X).There
existsnolosslesscodeforthissourceatrateslessthanH(X).
ELEC5360
29
Sampler
Input
waveform
Quantizer
Discrete
encoder
Binary
Channel
Analogfilter
Output
waveform
Tablelookup
Discrete
decoder
Binary
interface
Analog
sequence
Symbol
sequence
ELEC5360
30
Lossy
quantization
LossyDataCompression
Forcontinuousamplitudeanalogsequences
Losslesscompressionisimpossible
Lossycompressionthroughscalarorvectorquantization
Quantization
Usesafinitenumberoflevelstorepresenteach
continuousamplitudesymbol
Introducesdistortion,alossofsignalfidelity
ThenwemayuseHuffmancodingtoimprovethe
efficiency
Tradeoff betweenbitrateanddistortion!
Minimizebitrateforagivendistortion
Minimizedistortionforagivenrate
ELEC5360
31
DistortionMeasure
Definedistortionassomedistancemetricbetweentheactual
signalsamples x
k
andthequantizedvalues x
k
,denoted
byJ x
k
, x
k
Acommonlyuseddistortionmeasureisthesquarederrorfunction
Thedistortionbetweenasequenceofn samplesx
n
andthe
correspondingn quantizedvaluesx
n
istheaverageoverthen
sourceoutputsamples
Itsexpectedvalueis
ELEC5360
32
J x
k
, x
k
= x
k
-x
k
2
J x
n
, x
n
=
1
n
J x
k
, x
k
n
k=1
= E J X
n
, X

n
=
1
n
E J X
k
, X
`
k
n
k=1
= E J X, X
`
RateDistortionFunction
RatedistortionfunctionR(D)
I X; X
`
isthemutualinformationbetweenX and X
`
R(D) istheminimumratethatisrequiredtorepresentthe
outputX withadistortionlessthanorequaltoD
RateR(D) decreasesasD increases
Sourcecodingwithafidelitycriterion
Amemoryless sourceX canbeencodedatrateRfora
distortionnotexceedingD ifR>R(D).Conversely,foranycode
withrateR<R(D) thedistortionexceedsD.
ELEC5360
33
R = min
L d X,X
`
<
I X; X
`
GaussianSourcewithSquaredErrorDistortion
ForcontinuousamplitudeGaussiansource(with
variance
2
),theratedistortionfunctionisknown
Noinformationneedbetransmittedwhen > o
2
Gaussianisupperboundtoallothersources thatis
Gaussianrequiresmaximumrateamongallothersources
Thedistortionratefunction
Themeansquareerrordistortiondecreasesattherateof
6dB/bit
ELEC5360
34
R
g
= _
1
2
log
o
2

, u o
2
u, > o
2

g
R = 2
-2R
o
2
BinarySourcewithHammingDistortion
Considerabinarysourcewithp = P X = 1 = 1 -P X = u ,
withentropydenotedasE
b
p
Fromlosslesssourcecodingtheorem,itcanbecompressedatanyrate
R thatsatisfiesR > E X = E
b
p ,andcanberecoveredperfectly
IfR < E
b
p ,errorswilloccur
Hammingdistortion
TheaveragedistortionisE J X, X
`
= P X = X
`
= P
c
,i.e.,the
averageofHammingdistortionistheerrorprobabilityin
reconstructionofthesource
Theratedistortionfunction
ELEC5360
35
J x, x = _
1, x = x
u, x = x
R = _
E
b
p - E
b
, u min p, 1 -p
u, otbcrwisc
I
1
, I
2
,,withlosslessdiscrete
sourceencoding
Encodedsequence,I
1
, I
2
,
WithanalphabetofsizeM
Quantization
Scalarquantization:eachanalogRVisquantizedindependently
Vectorquantization:theanalogsequenceissegmentedinto
blocksofnRVseach,theneachntupleisquantizedasaunit
ELEC5360
36
Inputsequence,u
1
, u
2
,
AnalogRVs,withpdf
0
u
Quantizer
Discrete
encoder
Binary
Channel
Outputsequence,u

1
, u

2
, Tablelookup
Discrete
decoder
Binary
interface
Example1
ELEC5360
37
Example2
ELEC5360
38
ScalarQuantization
Ascalarquantizer partitionsthesetofrealnumbersintoM
subsetsR
1
, , R
M
,calledquantizationregions
EachregionR
]
isrepresentedbyarepresentationpointo
]
Whenthesourceproducesanumberu R
]
,thatnumberis
quantizedintothepointo
]
Q:givenM,chooseregionsandrepresentationpointsto
minimizethemeansquarederror
ELEC5360
39
ScalarQuantizer
Choiceofintervalsforgivenrepresentationpoints
Givenanyu,thesquarederrortoo
]
is u -o
]
2
Itisminimizedbyrepresentingubytheclosesto
]
Thustheboundaryb
]
lieshalfwaybetweeno
]
ando
]+1
Choiceofrepresentationpointsforgivenintervals
Chooseo
]
tominimize
o
]
mustbetheconditionalmeanofUconditionalonu R
]
ELEC5360
40
HSE = E u -I(u)
2
= _
0
(u) u - o
]
2
Ju
R
]
M
]=1
TheLloydMaxAlgorithm
Remark
TheMSEdecreasesforeachexecutionofstep(2)and(3)
MSEisnonnegative,itapproachessomelimit
However,thealgorithmmightreachalocalminimumofMSE
Thealgorithmalsoworksforvectorquantization
ELEC5360
41
Sampler
Input
waveform
Quantizer
Discrete
encoder
Binary
Channel
Analogfilter
Output
waveform
Tablelookup
Discrete
decoder
Binary
interface
Analog
sequence
Symbol
sequence
ELEC5360
42
Sampling
AnalogDataCompression
Thegeneralapproach
Expandthewaveformintoanorthogonalexpansion
Quantizethecoefficientsinthatexpansion
Usediscretesourcecodingonthequantizer output
Example:PCM
Soquantizationanddiscretesourcecodingserveasouter
layers
Example
Instandardtelephony,thevoiceisfilteredto4kHzandthensampled
at8000samplespersecond.Eachsampleisthenquantizedtooneof
256possiblelevels,representedby8bits.Thusthevoicesignalis
representedasa64kilobit/second(kb/s)sequence.
ELEC5360
43
FiniteEnergySignals
Weassumefiniteenergysignals,i.e.,L
2
functions
Finiteenergysignalsareappropriateformodelingboth
sourcewaveformsandchannelwaveforms
Theyenjoysimplicityandgeneralityofmathematical
properties
E.g.,everyL
2
timelimitedwaveformhasaFourierseries,everyL
2
waveformhasaFourierintegral
canbetreatedessentiallyasvectors
ELEC5360
44
s t s
1
, s
2
,
OrthogonalExpansion
Orthogonalexpansionforafiniteenergysignals t
where
and
1
t ,
2
t , formsanorthonormalbasisofL
2
and
e.g.,Fourierseries
ELEC5360
45
s t = s
k

k
t

k=1
s
k
= _ s t
k
t

-
Jt
_
k
t
]
t

-
Jt = _
u, k = ]
1, k = ]
_ s t
2
Jt = s
k
2

k=1
FourierSeries
Consideratimelimitedcomplexfunctionu(t)in[T/2,T/2]
ItsFourierseriesisgivenby
where
Thus,awaveformcanberegardedasavector
Itcanbeusedforsamplinginthefrequencydomain
ELEC5360
46
SamplingTheorem
Considerafrequencylimitedcomplexfunctionu(t)in[W,W]
Samplingtheorem
where
Thus,awaveformcanberegardedasavector
Itcanbeusedforsamplinginthetimedomain
ELEC5360
47
SubspaceApproximation
Wemaytakealimitednumberofsamples
Willthisbeagoodapproximation?
Givenanorthonormalbasis
1
t ,
2
t ,
ForanarbitraryL
2
functionf(t),theexpression
hasaminimum,for
andtheminimumequals
Moreover
where
ELEC5360
48
FiniteEnergySignalsandMSE
Fromorthogonalexpansion
Supposethesecoefficientss
k
arequantizedass
k
,andthese
quantizedcoefficientsareusedtorecoverthesourcesignal,
denotedass t ,thenwehave
Thedistortion,measuredastheenergyinthedifference
betweenthesourcewaveformandthereconstructed
waveform,isproportionaltothesquaredquantizationerrorin
thequantizedcoefficients
ThisexplainsthewidespreaduseoftheMSE
ELEC5360
49
_ s t
2
Jt = s
k
2

k=1
_ s t -s t
2
Jt = s
k
- s
k
2

k=1
Segmentinginthetimedomain
Frequencydomainsampling
SegmentthewaveformintosegmentsofdurationT,andthen
expandeachsegmentinaFourierseries
Considerasegmentcenteredaroundsomearbitrarytime,
i.e.,expandfrom -I2 to +I2,theFourierseriesis
where
IfthebandwidthislimitedtobeB,thenweroughlyneed2BT real
numberstorepresentthissegment
Mostvoicecompressionalgorithmsusesuchanapproach,
usuallybreakingthevoicewaveforminto20mssegments.
ELEC5360
50
s t = s
k
c
2n]kt1
k
, - I2 t + I2
s
k
=
1
I
_ s t
+12
-12
c
-2n]kt1
Jt
Segmentinginthefrequencydomain
Timedomainsampling
SegmentthewaveforminfrequencyofbandwidthB and
samplingeachfrequencyband
Considerasegmentcenteredaroundsomearbitrary
frequency,i.e.,expandfrom -B to +B,withthe
samplingintervalI = 12B,followingthesamplingtheorem
IfthetimeintervalislimitedtobeT,thenweroughlyneed2BT real
numberstorepresentthissegment
Bothc
2n]kt1
iect
t-
1
andsinc
t
1
-k c
2n] t-k1
are
orthogonalexpansions
ELEC5360
51
s t = s kI sinc
t
I
- k c
2n] t-k1
k
DegreesofFreedom
Frompreviousdiscussion,weseethatthosedifferent
expansionsrequireroughly2BT realnumbersforthe
approximatespecificationofawaveformessentiallylimitedto
timeT andfrequencybandB
Thisnumber,2BT,iscalleddegreesoffreedom
Thisisanimportantruleofthumbusedbycommunication
engineers
Forsourcecoding,thisrepresentsthenumberofcoefficientsweneed
tospecifyinordertorecoverthesourcewaveform
Forchannelcodingthatwewilldiscusslater,itmeansthiswaveform
cancarrythisamountofsymbols
Note:ourdiscussionisbasedonsomevagueidea.Amore
precisewayisthroughtheprolate spheroidalwaveforms
expansion
ELEC5360
52
Summary
Discretesourcecoding
Losslesscompression
Lossycompression
Quantization
Sampling
Readingassignment
Ch24ofGallager
Section6.16.4ofProakis
ELEC5360
53