Arithmetic Coding

Presented by: Einat & Kim

Objectives
The Arithmetic Coding idea Huffman vs. Arithmetic Coding Underflow Incremental coding Integer Arithmetic & Overflow Advantages vs. Disadvantages Binary Arithmetic coding Compressed tree
2

Model Based Approach

The Model a way of calculating, in any given context, the distribution of probabilities for the next input symbol. The decoder must have access to the same model & be able to regenerate the same input string from the encoded string. Model types: - fixed model - adaptive model
3

The Arithmetic Coding Idea

The Basic Algorithm

1. We begin with a current interval" [L; H) initialized to [0; 1). 2. For each symbol, we perform two steps : (a) We subdivide the current interval into subintervals, one for each possible alphabet symbol. The size of a symbol's subinterval is proportional to the estimated probability that the symbol will be the next symbol in the file, according to the model of the input. (b) We select the subinterval corresponding to the symbol that actually occurs next in the file, and make it the new current interval. 3. We output enough bits to distinguish the final current interval from all other possible final intervals.

The Basic Algorithm (cont.)

Computation of the subinterval corresponding to the ith symbol that occurs:

[ L Pc( H L), L PN ( H L))

Pc Pk
k 1 i 1

PN Pk
k 1
6

Encoding
The message is represented by an interval of real numbers between 0 & 1. As the message becomes longer, its interval becomes smaller the no. of bits needed to specify that interval grows. The reduction of the intervals size is according to the symbols probability (generated by the model).

A more common symbol will reduce the range by less,

add fewer bits to the message..
7

A simple example : encoding the message eaii!

The mod el

Symbol
a

Probability
.2

Range
[0,0.2)

e
i o u

.3
.1 .2 .1

[0.2, 0.5)
[0.5, 0.6) [0.6, 0.8) [0.8, 0.9)

.1

[0.9, 1)
8

After seeing 1
0.9

e
0.5

a
! u o i
0.26

i
! u o i
0.236

i
! u o i
0.2336

!
! u o i
0.2336

0.8

! u o

! u

o
i

0.6 0.5

e
0.2

a
0.2 0.2

a
0.23

a
0.233

a
0.23354

 The size of the final range is

0.2336 -0.23354 = 0.00006,

that is also exactly the multiplication of the probabilities of the five symbols in the message eaii! :
(0.3)* (0.2)* (0.1)* (0.1)* (0.1) = 0.00006.

it takes 5 decimal digits to encode the message.

According to Shannon :

The best compression code is the output length contains a contribution of -logp bits from the encoding of each symbol whose probability of occurrence is p.

The entropy of eaii! is :

-log 0.3 -log 0.2 -log 0.1 -log 0.1 -log 0.1 =-log 0.00006 = 4.22

10

A few remarks..
5 decimal digits seems a lot to encode a message comprising 4 vowels! Our example ended up by expanding rather than compressing. Different models will give different entropies

The best single-character model of the message eaii! is the set of symbol frequencies : { e(O.2) , a(0.2) , i(O.4) , !(0.2) } - which gives an entropy of 2.89 decimal digits.

The encoding would be only 3 digits long!

11

Decoding
The decoder receives the final range or a value from that range & finds symbol such that:
symbol k 1

(value L) /( H L)

symbol1 k 1

The decoder deduces the message from the first symbol to the last, according to the symbols probability of which the interval belongs to. Each message ends with EOF to avoid ambiguity: i.e. a single no. 0.0 could represent any of a,aa,aaa
12

Back to the example

The decoder gets the final range : [0.23354,0.2336). The range lies entirely within the space the model first character was e !!! allocate for e
Now it can simulate the operation of the encoder:

e a

Initially [0,1) [0.2,0.5)

[0.2,0.26)

[0.23354,0.2336).
13

 Huffmans method takes a set of probabilities and calculates, for

each symbol a code word that unambiguously represents that symbol. It is known to give the best possible representation when all of the symbols must be assigned discrete code words, each an integral number of bits long,

all symbol probabilities are approximated by powers of -2 Huffmans worst case : A symbol with probability approaching 1 , - i.e. p=0.9 log(1/p) = 0.15 conveys negligible information on average, but requires at least 1 bit to transmit.
14

On the other hand..

In an arithmetic coder: - the exact symbol probabilities are preserved. - compression effectiveness is better. - Using exact probabilities means that it is not possible to maintain a discrete code word for each symbol; Instead, an overall code for the whole message must be calculated.

15

Two major difficulties

The shrinking current interval requires use of high precision arithmetic. No output is produced until the entire message has been encoded.

16

Solution incremental coding

As the code range narrows, the top bits of L and H become the same. Any bits that are the same can be transmitted immediately, since they cannot be affected by future narrowing.

17

Solution (cont.) Example

Suppose the final interval is: . [0.8125;0.825)

In binary (approximately) : . [0.11010 00000, 0.11010 01100)

We can uniquely identify this interval by outputting 110100.

18

Solution incremental coding(cont.)

The idea:

To output each leading bit as soon as it is known, To double the length of the current interval so that it reflects only the unknown part of the final interval.
This solves both problems
19

Underflow!
Reminder: we scale the cumulative probabilities into the interval [L,H) for each character transmitted.. Suppose L and H are so close together, that this scaling operation maps some different symbols of the model onto the same integer in the interval

20

Underflow (cont.)
If such a symbol actually occurred it would not be possible to continue encoding
The encoder must guarantee that the interval [L,H) is always large enough to prevent this !
Therefore

the interval is at least as large as

21

Underflow (cont.)
When does this condition appear?
- L and H can only become close together when they straddle Half . < L<
0

1/2

<H<
3/4 0.1.. 1

1/4
0.0..

We dont yet know the next output bit, but we do know that the following bit will have the opposite value , 01 or 10 .

22

Underflow (cont.)
we keep track of that fact, and expand the current interval symmetrically about .
1

0 0

0 0

Output: 01

Output: 10
23

Underflow (cont.)
What if after this operation its still < L< <H< ?
Illustration:
1 Output: 011 0 0 0
24

Underflow (cont.)
1 Output: 100 0 0 0 1 1

We need only count the no. of expansions & follow the next bit by that no. of opposites!
25

Incremental Coding
.. After the selection of the subinterval corresponding to an input symbol. Repeat the following steps: 1. If the new subinterval is not entirely within one of the intervals [0; ), [ ; ), or [ ; 1), we stop iterating and return.
26

Incremental Coding (cont.)

2. If the new subinterval lies entirely within [0; ), we output 0 and any 1s left over from previous symbols; then we double the size of the interval [0; ), expanding up.
1
1

0 0
27

Incremental Coding (cont.)

3. If the new subinterval lies entirely within [ ; 1), we output 1 and any 0s left over from previous symbols; then we double the size of the interval [;1), expanding down.
1
1

0 0
28

Incremental Coding (cont.)

4. If the new subinterval lies entirely within [ ; ), we keep track of this fact for future output; then we double the size of the interval [ ; ), expanding in both directions away from the midpoint.
1
1

0 0
29

Example (encoding the message bbb)

After seeing

b
1
0.9 0.85
EOF

b
1
Output 1 follow 0.7 0.65
EOF

b
1
0.8 0.75
EOF

EOF

1
0.9

EOF

1
Output 10

Subdivide

b a

0.6 0.4 0.4

Expand [;1)

b
0.4

Expand [;)

b
0.5

Expand [;1)

0.6 0.5

a a
0.3

0.2

0
a b EOF

0.4

0.5

0.1

30

1
Output 1

1
0.8

0.6 0.5

Expand [;1)
Output 0 0.5 0.4 Output 0 0.5

0.2

Expand [0;)

Expand [0;)
0

Output: 110100
31

Integer Arithmetic.
When a whole floating number is passed to the decompressor , no rounding can be performed .. The highest precision todays fpu offers is 80 bits. So we CANT work with the whole number! Instead we redefine the range [0;1) to [0;N), i.e. [0000h;FFFFh). We also reduce the probabilities so we only need 16 bit. In the subdivision process we select non-overlapping intervals (of length at least 1) with lengths approximately proportional to the probabilities.

32

Integer Arithmetic Example

Lets see theyre the same..
0.000
0000h

0.250
4000h

0.500
8000h

0.750
C000h

1.000
FFFFh

33

Integer Arithmetic Example

Well divide the numbers by the maximum:

0000h: 4000h: 8000h: C000h: FFFFh:

0/65535 = 0.0 16384/65535 = 32768/65535 = 49152/65535 = 65535/65535 =

0.25 0.5 0.75 1.0

34

Overflow!
Now we consider the possibility of overflow in the integer multiplications. In every step of the encoding the interval is expanded:

L (( H L 1] *

symbol1 k 1

P ) / N)
k

Constraints are made on the range & the maximum frequency to ensure that arithmetic is done to 32-bit precision.
35

Advantages vs. Disadvantages

Advantages
Flexibility - conjunction with

Disadvantages
Slow Doesnt produce a prefix code (minor) needs to indicate EOF (minor) poor error resistance

models that can provide a sequence of event probabilities (i.e. adaptive models).

Optimality

36

Binary Arithmetic Coding

Till now we discussed about a multi-symbol alphabet It applies to a binary alphabet as well

Why distinguish the two cases? - both the coder and the interface to the model are simpler for a binary alphabet.
The coding of bilevel images often produces probabilities close to 1, indicating the use of arithmetic coding to obtain good compression.
000100!
010010111 ?
37

Binary Arithmetic Coding (cont.)

Historically..

Much of the arithmetic coding research by Rissanen, Langdon, and others at IBM has focused on bilevel images.

In most other text and image compression applications, a multi-symbol alphabet is more natural..
38

Binary Arithmetic Coding (cont.)

But..

We can map the possible symbols to the leaves of a binary tree & encode an event by traversing the tree & encoding a decision at each internal node.

This way : - the model no longer has to maintain and produce cumulative probabilities - a single probability suffices to encode each decision.

- calculating the new current interval is also simplified, since just one endpoint changes after each decision.

39

Binary Arithmetic Coding (cont.)

On the other hand..

we now usually have to encode more than one event for each input symbol. we have a new data structure problem, maintaining the coding trees efficiently without using excessive space.
40

Binary Arithmetic Coding (cont.)

In Huffman tree the average path length is minimal. Therefore, the smallest average number of events per input symbol occurs when the tree is a Huffman tree. However, maintaining such trees dynamically is complicated and slow ..

41

Compressed Trees
An efficient data structure is needed to map each of n symbols to a sequence of binary choices..

Solution

- the compressed tree, suitable for binary encoding of multi-symbol alphabets.

42

Compressed Trees (cont.)

The compressed tree :
A space-efficient data structure based on the complete binary tree.
We are free to represent the n-symbol alphabet by a complete binary tree, because arithmetic coding has nearly optimal compression..

43

Compressed Trees (cont.)

The tree can be flattened (linearized) by breadth-first traversal We can save space by storing only one probability at each internal node.
- This probability can be stored to sufficient precision in just one byte!
44

Compressed Trees example

Suppose we have the following probability distribution for an 8-symbol alphabet:
Symbol

a
0

b
0

c
1 8

d
1 4

e
1 8

f
0

g
1 8

h
3 8

Probability

45

Compressed Trees example (cont.)

We can represent this distribution by the tree, rounding probabilities & expressing them as multiples of 0.01.
38 | 62 0 | 100 - | 33 | 67 100 | 0 20 | 80 25 | 75

46

Compressed Trees example (cont.)

The linear representation: 38 0 20 33 100 25

The compressed tree :

38

20

33

100

25

47

The End!

48