Documente Academic
Documente Profesional
Documente Cultură
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 1 / 52
Outline
3 Huffman Code
5 Lempel-Ziv Code
6 Arithmetic Code
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 2 / 52
Source Coding
Source Channel
x(t) y(t) Destination
or or
x[n] y[n]
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 3 / 52
Source Coding
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 4 / 52
Source Coding
x(t)
sampling at rate
2W
x[n]
No loss of information
=⇒ intermediate values redundant!
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 5 / 52
Source Coding
x(t)
sampling at rate
2W
x[n]
No loss of information
=⇒ intermediate values redundant!
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 5 / 52
Source Coding
The channel may not have the “capacity” to communicate at the source
information rate =⇒ need to represent source at a lower rate with some loss
of information
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 6 / 52
Source Coding
The channel may not have the “capacity” to communicate at the source
information rate =⇒ need to represent source at a lower rate with some loss
of information
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 6 / 52
Source Coding
The channel may not have the “capacity” to communicate at the source
information rate =⇒ need to represent source at a lower rate with some loss
of information
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 6 / 52
Source Coding
The channel may not have the “capacity” to communicate at the source
information rate =⇒ need to represent source at a lower rate with some loss
of information
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 6 / 52
Source Coding
The channel may not have the “capacity” to communicate at the source
information rate =⇒ need to represent source at a lower rate with some loss
of information
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 6 / 52
Types of Lossy Source Coding
Lossless
I no loss of information
I used by various compression schemes for file storage (e.g. compress in Linux)
Lossy
I with loss of information, e.g. quantization, vector quantization, sub-band
coding
I used by standards like JPEG, MPEG etc.
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 7 / 52
Lossless Source Coding
for
Discrete Memoryless Sources
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 8 / 52
Discrete Memoryless source
Source X0 , X1 , X2 ...........
i.i.d.
Xi ∈ {x0 , x1 , · · · , xM−1 }
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 9 / 52
If M ≤ 2b , then we can represent the symbols by binary fixed length codes as:
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 10 / 52
Something better possible!
Example:
Symbol: x0 x1 x2 x3
Probability: 0.5 0.25 0.125 0.125
Fixed length: 00 01 10 11
Variable length: 0 10 110 111
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 11 / 52
Desired properties of source codes
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 12 / 52
Desired properties of source codes
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 12 / 52
Desired properties of source codes
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 13 / 52
Desired properties of source codes
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 13 / 52
Example : Classes of codes
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 14 / 52
Huffman code
Code Design
Arrange the alphabet in decreasing order of their probability
Assign 0 and 1 as the last bit of the last two symbols
Combine the last two symbols as one and assign the sum of their probabilities
to the new symbol
Repeat this process on the new set of symbol again and again. While
assigning a bit to a derived symbol, the bit is pre-pended to the codewords of
all the contributing symbols.
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 15 / 52
Example
X Codeword Probability
1 0.20
2 0.15
3 0.25
4 0.25
5 0.15
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 16 / 52
Example
X Codeword Probability
3 0.25
4 0.25
1 0.20
2 0.15
5 0.15
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 16 / 52
Example
X Codeword Probability
3 0.25 0.30
4 0.25 0.25
1 0.20 0.25
2 0 0.15 0.20
5 1 0.15
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 16 / 52
Example
X Codeword Probability
2 0 0.15 0.20
5 1 0.15
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 16 / 52
Example
X Codeword Probability
2 0 0 0.15 0.20
5 0 1 0.15
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 16 / 52
Example
X Codeword Probability
2 0 0 0 0.15 0.20
5 0 0 1 0.15
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 16 / 52
Kraft Inequality:
2−li ≤ 1
P
i
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 17 / 52
Kraft Inequality: Proof
0
0 1
0 0
1
1
0 Necessity:
0 1 0
1
0
1
1
0
1 0 code 01
1
0 0
1
1
0 code 101
1 0 1
0
1
1
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 18 / 52
Kraft Inequality: Proof
0
0 1
0 0
1
1 Necessity:
0
0 1 0 Codewords disqualified by 01:
1
0 010,011,0100,0101, 0110,0111
1
1
0
1 0 code 01
1
0 0
1
1
0
1 0 1
0
1
1
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 18 / 52
Kraft Inequality: Proof
0
0 1
0 0
1
1 Necessity:
0
0 1 0 Codewords disqualified by 01:
1
0 010,011,0100,0101, 0110,0111,0
1
1
0
1 0 code 01
1
0 0
1
1
0
1 0 1
0
1
1
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 18 / 52
Kraft Inequality: Proof
0
0 1
0 0
1 Necessity:
1
0 Codewords disqualified by 01:
0 1 0
1 010,011,0100,0101, 0110,0111,0
0
1 Consider codeword C (xi ), length li
1
0 and let lmax = maxi {li }
1 0 code 01
1
0 0
1
1
0
1 0 1
0
1
1
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 18 / 52
Kraft Inequality: Proof
0
0 1
0 0 Necessity:
1
1
0 Codewords disqualified by 01:
0 1 0 010,011,0100,0101, 0110,0111,0
1
1
0 Consider codeword C (xi ), length li
1 and let lmax = maxi {li }
0
1 0 code 01 No. of descendants of C (i) at level
1
0 lmax = 2lmax −li
0 1
1
0
1 0 1
0
1
1
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 18 / 52
Kraft Inequality: Proof
0
0 1
0 Necessity:
0 1
1 Codewords disqualified by 01:
0 010,011,0100,0101, 0110,0111,0
0 1 0
1 Consider codeword C (xi ), length li
0
1 and let lmax = maxi {li }
1
0 No. of descendants of C (i) at level
0 code 01
1 1 lmax = 2lmax −li
0 0
1 Descendants are disjoint for
1
0 different i
1 0 1
0
1
1
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 18 / 52
Kraft Inequality: Proof
0
0 1 code 001 Necessity:
0 0
1 Codewords disqualified by 01:
1
0 010,011,0100,0101, 0110,0111,0
0 1 0 Consider codeword C (xi ), length li
1
1
0 and let lmax = maxi {li }
1 No. of descendants of C (i) at level
0
1 0 code 01 lmax = 2lmax −li
1
0 0 Descendants are disjoint for
1
1 different i
0
1 0 1 Considering
P lmax −li alllmax
codewords,
0 i 2 ≤ 2
1
1
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 18 / 52
Kraft Inequality: Proof
0
0 Necessity:
1 code 001
0 0 Codewords disqualified by 01:
1
1 010,011,0100,0101, 0110,0111,0
0
1 0 Consider codeword C (xi ), length li
0 1
0 and let lmax = maxi {li }
1
1 No. of descendants of C (i) at level
0 lmax = 2lmax −li
1 0 code 01
1 Descendants are disjoint for
0 0
1 different i
1
0 Considering
P lmax −li alllmax
codewords,
1 0 1 2 ≤ 2
i
0
1 ⇒
P −li
≤1
1 i 2
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 18 / 52
Kraft Inequality: Proof
0
0 1 code 001
0 0
1
1
0 Sufficiency:
0 1 0
1 l1 , l2 , . . . , lM - Given in increasing
0
1 order.
1
0 To construct a prefix code
1 0 code 01
1 For i = 1, 2, · · · , M,
0 0
1 Take the first remaining node at
1
0 level li as C (xi ).
1 0 1
0
1
1
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 19 / 52
Kraft Inequality: Proof
0
0 1 code 001
0 0
1
1
0 Sufficiency:
0 1 0
1 l1 , l2 , . . . , lM - Given in increasing
0
1 order.
1
0 To construct a prefix code
1 0 code 01
1 For i = 1, 2, · · · , M,
0 0
1 Take the first remaining node at
1
0 level li as C (xi ).
1 0 1
0
1
1
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 19 / 52
Kraft Inequality: Proof
0
0 1 code 001
0 0
1
1
0 Sufficiency:
0 1 0
1 l1 , l2 , . . . , lM - Given in increasing
0
1 order.
1
0 To construct a prefix code
1 0 code 01
1 For i = 1, 2, · · · , M,
0 0
1 Take the first remaining node at
1
0 level li as C (xi ).
1 0 1
0
1
1
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 19 / 52
Kraft Inequality: Proof
0
0 1 code 001
0 0
1
1
0 Sufficiency:
0 1 0
1 l1 , l2 , . . . , lM - Given in increasing
0
1 order.
1
0 To construct a prefix code
1 0 code 01
1 For i = 1, 2, · · · , M,
0 0
1 Take the first remaining node at
1
0 level li as C (xi ).
1 0 1
0
1
1
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 19 / 52
Kraft Inequality: Comments
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 20 / 52
Kraft Inequality: Comments
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 20 / 52
Optimal code
Theorem
H(x) ≤ L < H(x) + 1
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 21 / 52
Optimal code
Theorem
H(x) ≤ L < H(x) + 1
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 21 / 52
Part I: For any uniquely decodable code, the expected
length L(C , X ) ≥ H(X )
equality iff pi = qi ∀i
X X 1
L(C , X ) = pi li = pi log − log(z)
qi
i i
X 1
≥ pi log − log(z) ≥ H(x)
pi
i
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 22 / 52
Part I: For any uniquely decodable code, the expected
length L(C , X ) ≥ H(X )
equality iff pi = qi ∀i
X X 1
L(C , X ) = pi li = pi log − log(z)
qi
i i
X 1
≥ pi log − log(z) ≥ H(x)
pi
i
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 22 / 52
Part I: For any uniquely decodable code, the expected
length L(C , X ) ≥ H(X )
equality iff pi = qi ∀i
X X 1
L(C , X ) = pi li = pi log − log(z)
qi
i i
X 1
≥ pi log − log(z) ≥ H(x)
pi
i
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 22 / 52
Part I: For any uniquely decodable code, the expected
length L(C , X ) ≥ H(X )
2 li = log p1i
These conditions suggest good code design philosophy.
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 23 / 52
Part II : There exists code with L(C , X ) < H(X ) + 1
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 24 / 52
Part II : There exists code with L(C , X ) < H(X ) + 1
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 24 / 52
Part II : There exists code with L(C , X ) < H(X ) + 1
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 24 / 52
Part II : There exists code with L(C , X ) < H(X ) + 1
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 24 / 52
Part II : There exists code with L(C , X ) < H(X ) + 1
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 24 / 52
Need to combine many source symbols for source coding
symbols probability best symbol-wise code
1 0.9 1
0 0.1 0
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 25 / 52
How L → H(X ) can be achieved for any DMS:
The average code length L = Lnn per actual source symbol satisfies
H(x) ≤ L < H(x) + n1
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 26 / 52
Huffman Code Revisited
Properties
Huffman code is a prefix code.
Huffman code is optimum - no code better than Huffman code for any alphabet.
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 27 / 52
Huffman Code Revisited
Disadvantages
Need to know/ calculate the symbol probabilities explicitly ⇒ to combine n
symbols of an M-ary source, need to compute M n probabilities.
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 28 / 52
Lempel-Ziv Coding
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 29 / 52
LZ Code
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 30 / 52
LZ Code: How!
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 31 / 52
Encoding: Example
Input:
0
Output:
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 32 / 52
Encoding: Example
Input: 1
0 1
1
Output: 1
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 32 / 52
Encoding: Example
Input: 1 0
00 01 10
1 0
Output: 1 0,0
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 32 / 52
Encoding: Example
Input: 1 0 11
00 01 10 11
1 0 11
Output: 1 0,0 01,1
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 32 / 52
Encoding: Example
Input: 1 0 11 01
000 001 010 011 100
1 0 11 01
Output: 1 0,0 01,1 10,1
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 32 / 52
Encoding: Example
Input: 1 0 11 01 010
000 001 010 011 100 101
1 0 11 01 010
Output: 1 0,0 01,1 100,0
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 32 / 52
Encoding: Example
Input: 1 0 11 01 010 00
000 001 010 011 100 101 110
1 0 11 01 010 00
Output: 1 0,0 01,1 100,0 010,0
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 32 / 52
Encoding: Example
Input: 1 0 11 01 010 00 10
000 001 010 011 100 101 110 111
1 0 11 01 010 00 10
Output: 1 0,0 01,1 100,0 010,0 001,0
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 32 / 52
Encoding: Example
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 32 / 52
Decoding: Example
Input:
0
Output:
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 33 / 52
Decoding: Example
Input: 1
0 1
1
Output: 1
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 33 / 52
Decoding: Example
Input: 1 00
00 01 10
0 0
Output: 1 0
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 33 / 52
Decoding: Example
Input: 1 00 011
00 01 10 11
0 0 11
Output: 1 0 11
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 33 / 52
Decoding: Example
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 33 / 52
Decoding: Example
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 33 / 52
Decoding: Example
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 33 / 52
Decoding: Example
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 33 / 52
Few Improvements
1 For any string S, if both S 0 and S 1 are in the list, then S can be removed
from the list.
2 If S and S 0 are in the list and input is S 1, then the bit 1 need not be
transmitted because it can not be 0.
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 34 / 52
Few Improvements
1 For any string S, if both S 0 and S 1 are in the list, then S can be removed
from the list.
2 If S and S 0 are in the list and input is S 1, then the bit 1 need not be
transmitted because it can not be 0.
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 34 / 52
Shannon-Fano-Elias Code
or
Arithmetic Code
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 35 / 52
Shannon-Fano-Elias Code
1
p(a)
ax
F(x) =
0
0 1 6 7
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 36 / 52
Shannon-Fano-Elias Code
1
p(a)
ax
F(x) =
p(x)
0 x
0 1 6 7
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 36 / 52
Shannon-Fano-Elias Code
1
p(a)
x
F(x)
a
F(x) =
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 36 / 52
Shannon-Fano-Elias Code
l(x) = log 1 +1
p(x)
1
p(a)
ax
F(x) =
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 36 / 52
Shannon-Fano-Elias Code
code
x
for x
a
0.011001 (say)
F(x) =
p(x) ||
F(x) − 1 p(x)
2 l(x)
0 x
0 1 6 7
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 36 / 52
Shannon-Fano-Elias Code
l(x) = log 1 +1
p(x)
1
p(a)
1 p(x)
l(x) 2
2
ax
F(x) =
p(x)
F(x) − 1 p(x)
2 l(x)
0 x
0 1 6 7
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 36 / 52
Shannon-Fano-Elias Code
l(x) = log 1 +1
p(x)
1
p(a)
1 p(x)
l(x) 2
2
ax
F(x) =
p(x)
F(x) − 1 p(x)
2 l(x)
0 x
0 1 6 7
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 36 / 52
Shannon-Fano-Elias Code
1 p(x)
l(x) code 2
2
x
for x
a
0.011001 (say)
F(x) =
p(x) ||
F(x) − 1 p(x)
2 l(x)
0 x
0 1 6 7
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 36 / 52
Shannon-Fano-Elias Code
1 p(x)
l(x) 2
2
x
0.011010
a
F(x) =
p(x)
0.011001 (say)
0 x
0 1 6 7
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 36 / 52
Performance of SFA code
1
P P
1. L = i P(x )l
i i = i P(x i ) dlog P(xi ) e + 1
1
P
= i P(xi )dlog P(xi ) e + 1
1
P
< i P(xi )log P(x i)
+2
= H(X ) + 2 ⇒ suboptimal
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 37 / 52
Arithmetic Code
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 38 / 52
Arithmetic code: What is it!
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 39 / 52
Block-wise SFA
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 40 / 52
Computing Fi (X1 , X2 , · · · , Xi ) iteratively
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 41 / 52
Block-wise SFA
1 Need not compute all probability masses or the complete joint cumulative
distribution function.
2 Arithmetic coding can tackle source with correlation and thus gives better
compression for source with correlation.
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 42 / 52
Encoding
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 43 / 52
Computing Fi (X1 , X2 , · · · , Xi ) iteratively
Fi (x(i) ) = F(i−1) (x(i−1) − 1) + P(x(i−1) )F (xi |x(i−1) )
Subscript in F (·) is omitted for simplicity
F(x (i−1) )
P(x (i−1))
F(x (i−1)− 1)
0
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 44 / 52
Computing Fi (X1 , X2 , · · · , Xi ) iteratively
Fi (x(i) ) = F(i−1) (x(i−1) − 1) + P(x(i−1) )F (xi |x(i−1) )
Subscript in F (·) is omitted for simplicity
1 1
F(x (i−1) )
F(x i x (i−1))
P(x (i−1))
P(x i x (i−1))
F(x (i−1)− 1) F(x i−1 x (i−1))
0 0
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 44 / 52
Computing Fi (X1 , X2 , · · · , Xi ) iteratively
Fi (x(i) ) = F(i−1) (x(i−1) − 1) + P(x(i−1) )F (xi |x(i−1) )
Subscript in F (·) is omitted for simplicity
1 F(x (i−1)) 1
F(x (i−1) )
F(x (i) ) F(x i x (i−1))
P(x (i−1))
P(x (i) ) P(x i x (i−1))
F(x (i−1)− 1) F(x (i) −1 ) F(x i−1 x (i−1))
0 F(x (i−1)− 1) 0
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 44 / 52
Iterative encoding
2 If the binary representation of Fi (x(i) ) and Fi (x(i) − 1) have the first t bits
common, then the final code-string will start with those t bits.
3 As i increases, the interval (Fi (x(i) − 1), Fi (x(i) )) will be narrower and
narrower. More and more bits will be common to Fi (x(i) ) and Fi (x(i) − 1).
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 45 / 52
Iterative encoding
2 If the binary representation of Fi (x(i) ) and Fi (x(i) − 1) have the first t bits
common, then the final code-string will start with those t bits.
3 As i increases, the interval (Fi (x(i) − 1), Fi (x(i) )) will be narrower and
narrower. More and more bits will be common to Fi (x(i) ) and Fi (x(i) − 1).
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 45 / 52
Iterative encoding
2 If the binary representation of Fi (x(i) ) and Fi (x(i) − 1) have the first t bits
common, then the final code-string will start with those t bits.
3 As i increases, the interval (Fi (x(i) − 1), Fi (x(i) )) will be narrower and
narrower. More and more bits will be common to Fi (x(i) ) and Fi (x(i) − 1).
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 45 / 52
Iterative encoding
2 If the binary representation of Fi (x(i) ) and Fi (x(i) − 1) have the first t bits
common, then the final code-string will start with those t bits.
3 As i increases, the interval (Fi (x(i) − 1), Fi (x(i) )) will be narrower and
narrower. More and more bits will be common to Fi (x(i) ) and Fi (x(i) − 1).
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 45 / 52
Computing P(xi |x(i−1) )
Special Cases
1 I.I.D. source: P(xi |x(i−1) ) = P(xi )
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 46 / 52
Computing P(xi |x(i−1) )
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 47 / 52
Computing P(xi |x(i−1) )
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 47 / 52
Example: File Compression
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 48 / 52
Probability model
1 P(|x(i−1) ) = 0.15
Fa +1
2 P(xi = a|x(i−1) ) = 0.85 × Fa +Fb +2
Fb +1
3 P(xi = b|x(i−1) ) = 0.85 × Fa +Fb +2
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 49 / 52
Arithmetic encoding: Example
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 50 / 52
File compression
Input: b
0 1...
b
a 0 0...
0
Output:
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 51 / 52
File compression
Input: b b
1
b
0 1...
bb
ba
a 0 0...
0
Output: 1
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 51 / 52
File compression
Input: b b
1
b 0 11...
bb 0 10...
ba
a 0 0...
0
Output: 1
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 51 / 52
File compression
Input: b b
1
0 11...
b 0 11...
bb 0 10...
ba bb
0 10...
a 0 0...
0
Output: 1
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 51 / 52
File compression
Input: b b b
1
bb 0 11...
b 0 11...
bb 0 10...
ba bbb
0 10...
a 0 0...
bba
0
Output: 1
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 51 / 52
File compression
Input: b b b a
1
bb 0 11...
b 0 11...
bbb
bb 0 10... 0 101...
bbbb
ba
bbba
a 0 0... 0 100...
bba
0
Output: 1
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 51 / 52
File compression
Input: b b b a
1
bb 0 11...
b 0 11...
bbb
bb 0 10... 0 101...
bbbb
ba
bbba 0 1001...
a 0 0...
bba 0 1000...
0
Output: 1 001
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 51 / 52
File compression
Input: b b b a
1
bb 0 11... 0 10011111
bbb bbba
0 10011110
0 101... 0 10011101
bbbb
bbbab 0 10011100
0 10011011
bbba 0 1001... 0 10011010
bbbaa 0 10011001
bba 0 1000...
0 10011000
0
Output: 1 001 11101
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 51 / 52
File compression: Decoding
Input: 1
0 1...
b
a 0 0...
0
Output:
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 52 / 52
File compression: Decoding
Input: 1 0
1
b 0 11...
bb 0 10...
ba
a 0 0...
0
Output: b
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 52 / 52
File compression: Decoding
Input: 1 0 0
1
b 0 11...
bb 0 101...
0 100...
ba
a 0 0...
0
Output: b
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 52 / 52
File compression: Decoding
Input: 1 0 0 1
1
bb 0 11...
b 0 11...
bb 0 101...
0 101...
0 100... bbb
ba
0 1001...
a 0 0...
bba 0 1000...
0
Output: b b
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 52 / 52
File compression: Decoding
Input: 1 0 0 1 1
bb 0 11...
bbb
bbbb 0 101...
bbba 0 10011...
0 10010...
bba 0 1000...
Output: b b
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 52 / 52
File compression: Decoding
Input: 1 0 0 1 1 1
0 10011111
bbba
0 10011110
0 10011101
bbbab 0 10011100
0 10011011
0 10011010
bbbaa 0 10011001
0 10011000
Output: b b b
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 52 / 52
File compression: Decoding
Input: 1 0 0 1 1 1 1
0 10011111
bbba
0 10011110
0 10011101
bbbab 0 10011100
0 10011011
0 10011010
bbbaa 0 10011001
0 10011000
Output: b b b
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 52 / 52
File compression: Decoding
Input: 1 0 0 1 1 1 1 0
0 10011111
bbba
0 10011110
0 10011101
bbbab 0 10011100
0 10011011
0 10011010
bbbaa 0 10011001
0 10011000
Output: b b b a
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 52 / 52
File compression: Decoding
Input: 1 0 0 1 1 1 1 0 1
0 10011111
bbba 0 100111101
0 100111100
0 10011101
bbbab 0 10011100
0 10011011
0 10011010
bbbaa 0 10011001
0 10011000
Output: b b b a
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 52 / 52
Summary
3 Huffman Code
5 Lempel-Ziv Code
6 Arithmetic Code
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 53 / 52
Thank You
Prof. B. K. Dey (IIT Bombay) Lossless Source Coding January 24, 2019 54 / 52