Documente Academic
Documente Profesional
Documente Cultură
296.3
Page 1
Page 2
Compression Outline
Multimedia
Images: gif (LZW), jbig (context),
jpeg-ls (residual), jpeg (transform+RL+arithmetic)
TV: HDTV (mpeg-4)
Sound: mp3
An example
Other structures
Indexes: google, lycos
Meshes (for graphics): edgebreaker
Graphs
Databases:
296.3
296.3
Page 3
Introduction:
Lossless vs. lossy
Model and coder
Benchmarks
Information Theory: Entropy, etc.
Probability Coding: Huffman + Arithmetic Coding
Applications of Probability Coding: PPM + others
Lempel-Ziv Algorithms: LZ77, gzip, compress, ...
Other Lossless Algorithms: Burrows-Wheeler
Lossy algorithms for images: JPEG, MPEG, ...
Compressing graphs and meshes: BBK
296.3
Page 4
Encoding/Decoding
Input
Message
Output
Message
Page 5
296.3
Page 6
Model
Probs.
Coder
Bits
Example models:
Simple: Character counts, repeated strings
Complex: Models of a human face
296.3
Page 7
296.3
Page 8
Quality of Compression
Comparison of Algorithms
296.3
Program
RK
BOA
PPMD
IMP
BZIP
GZIP
LZ77
Page 9
Compression Outline
296.3
Score
430
407
265
254
273
318
?
Page 10
Information Theory
Page 11
296.3
Page 12
Entropy Example
i ( s) = log
p( S ) = {.25,.25,.25,.25}
H ( S ) = 4 .25 log 4 = 2
1
= log p( s)
p( s)
H ( S ) = p( s) log
sS
1
p( s)
296.3
p( S ) = {.5,.25,.125,.125}
296.3
Conditional Entropy
Page 14
i ( s| c) = log p( s| c)
The conditional information can be either more or less
than the unconditional information.
The conditional entropy is the weighted average of the
conditional self information
H ( S | C ) = p(c ) p ( s | c ) log
p ( s | c)
cC
sS
296.3
Page 15
p(w|w)
.9
b
p(w|b)
.2
296.3
p(b|b)
.8
Page 16
296.3
Page 17
Compression Outline
# of guesses 1 2 3 4 5 > 5
Probability .79 .08 .03 .02 .02 .05
From the experiment he predicted
H(English) = .6-1.3
296.3
Page 18
Shannons experiment
296.3
Page 20
Discrete or Blended
message:
0001
011
message:
1,2,3, and 4
296.3
Page 21
Prefix Codes
296.3
Page 22
1
1 0
a
0
a = 0, b = 110, c = 111, d = 10
296.3
Page 23
296.3
Page 24
Binary Unary
..001
0
..010
10
..011
110
..100
1110
..101
11110
..110 111110
Average Length
For a code C with associated probabilities p(c) the
average length is defined as
Gamma
0|
10|0
10|1
110|00
110|01
110|10
l a (C ) =
p ( s )l (c )
( s ,c )C
Page 25
Relationship to Entropy
Page 26
2 l ( c ) 1
H ( S ) la ( C )
cC
la (C ) H ( S ) + 1
296.3
296.3
Page 27
2 l 1
l L
296.3
Page 28
l ( s)
sS
2 (
2 (
p ( s)
sS
la ( S ) =
log 1/ p ( s ) )
sS
sS
= 1
p(s)l (c)
p(s) log (1 / p(s ))
p(s) (1 + log(1 / p( s)))
1 + p ( s) log(1 / p( s))
( s , c )S
sS
sS
sS
= 1 + H (S )
And we are done.
296.3
Page 29
296.3
Page 30
Huffman Codes
Invented by Huffman as a class assignment in 1950.
Used in many, if not most, compression algorithms
gzip, bzip, jpeg (as option), fax compression,
Properties:
Generates optimal prefix codes
Cheap to generate codes
Cheap to encode and decode
la = H if probabilities are powers of 2
Page 31
296.3
Page 32
Huffman Codes
Example
Huffman Algorithm:
Start with a forest of trees each consisting of a
single vertex corresponding to a message s and
with weight p(s)
Repeat until one tree left:
Select two trees with minimum weight roots p1
and p2
Join into single tree by adding root with weight
p1 + p2
b(.2)
c(.2)
(.5)
d(.5)
(1.0)
1
(.5) d(.5)
a(.1) b(.2)
(.3)
c(.2)
1
0
Step 1
(.3)
c(.2)
a(.1) b(.2)
0 1
Step 2
a(.1) b(.2)
Step 3
0
Page 33
(1.0)
1
d(.5)
(.5)
1
0
(.3)
c(.2)
0 1
a(.1) b(.2)
296.3
Page 34
296.3
Page 35
296.3
Page 36
296.3
l < 2 + si
i =1
Page 37
296.3
j =1
a = .2
0.55
0.2
c = .3
0.27
b = .5
0.3
a = .2
0.0
0.3
c = .3
b = .5
296.3
0.7
1.0
b = .5
0.2
0.0
Page 38
b = .5
0.21
a = .2
0.2
a = .2
0.2
Page 40
10
li = li 1 + si 1 fi
s1 = p1
si = si 1 pi
bottom of interval
size of interval
Warning
Three types of interval:
message interval : interval for a single message
sequence interval : composition of message
intervals
code interval : interval for a specific code used
to represent a sequence interval (discussed
later)
sn = pi
i =1
296.3
Page 41
296.3
Page 42
1.0
c = .3
0.7
0.49
0.55
0.49
b = .5
0.2
0.0
0.55
c = .3
0.2
c = .3
b = .5
b = .5
0.35
0.3
a = .2
0.49
0.475
a = .2
a = .2
0.3
Page 43
296.3
Page 44
11
Representing Fractions
Representing an Interval
.75
= .11
1/ 3
= .0101
11 / 16 = .1011
So how about just using the smallest binary
fractional representation in the sequence interval.
e.g. [0,.33) = .0 [.33,.66) = .1 [.66,1) = .11
But what if you receive a 1?
Should we wait for another 1?
296.3
.110
.1010
.111 [.75,1.0)
.1011 [.625,.75)
296.3
Page 46
.0
Sequence Interval
0
Note that if code intervals overlap then one code is
a prefix of the other.
Lemma: If a set of code intervals do not overlap
then the corresponding codes form a prefix code.
296.3
interval
1
.1
max
Page 45
.11
.101
min
Page 47
.75
Code Interval (.101)
.61
.625
296.3
Page 48
12
.110
.100
.001
Page 49
Bound on Length
296.3
Page 50
2 + si bits.
i =1
Proof:
1 + log s
=
=
=
<
296.3
n
1 + log pi
i =1
1 + log pi
i =1
n
1 + si
i =1
n
2 + si
i =1
Page 51
296.3
Page 52
13
R=256
si
= ui 1 li 1 + 1
ui
= li 1 + si f ( mi + 1) / T 1
li
= li 1 + si f ( mi ) / T
c = 30
l
b=7
a = 11
296.3
Page 53
296.3
Page 54
296.3
Page 55
14