Thomas Wiegand Heiko Schwarz

Video Coding State-of-the-Art High-Efficiency Video Coding

Motion representation Motion compensation Transform coding Intra coding Entropy Coding

It has been customary in the past to transmit successive complete images of the transmitted picture. [...] In accordance with this invention, this difficulty is avoided by transmitting only the difference between successive images of the object. Thomas Wiegand: Recent Advances in Video Coding Wiegand:

slide 3

1989: Digital TV Digital Broadcast, DVD 1999: TML-1 of H.264/AVC Every HDTV Receiver Every Blu-Ray Player Most Internet Video Countless Mobile Video
Today: more than 1 Billion devices with H.264/AVC
Thomas Wiegand: Recent Advances in Video Coding Wiegand:
slide 4

Source: http://techcrunch.com/2010/05/01/h-26466-percent-web-video/

Thomas Wiegand: Recent Advances in Video Coding Wiegand:

slide 5

Input Video Signal

Coder Control Transform/ Scal./Quant. Decoder Scaling & Inv. Transform

Control Data Quant. Transf. coeffs

Split into Macroblocks 16x16 pixels

Entropy Coding De-blocking Filter Output Video Signal Motion Data Motion Estimation

Intra-frame Prediction MotionIntra/Inter Compensation

Thomas Wiegand: Recent Advances in Video Coding Wiegand:

slide 6

Efficiency of Motion Compensation

Intraframe coding: only spatial correlation exploited
Complexity DCT [Ahmed, Natarajan, Rao 1974], JPEG [1992] increase

Motion compensation (MC): integer-pel accuracy, 16x16

H.261 [1991]

Half-pel accurate MC
MPEG-1 [1993], MPEG-2/H.262 [1994]

16x16 & 8x8 block size MC, quarter-pel

H.263 [1996], MPEG-4 [1999] Quarter-pel & 16x16 - 4x4 block size MC and multiple reference pictures H.264/AVC [2003]
slide 7

64x64 4x4 block sizes and new filters for MC HEVC [start 2010]
Thomas Wiegand: Recent Advances in Video Coding Wiegand:

Development of Video Coding

PSNR [dB] 40 38 36 35 34 32 30 28 0 100
Integer-pel motion compensation (H.261, 1991) Intra frame DCT coding (JPEG, 1990) Variable block size (16x16 4x4) + quarter-pel + multi-frame motion compensation (H.264/AVC, 2003) Variable block size (16x16 8x8) (H.263, 1996) + quarter-pel motion compensation (MPEG-4, 1998) Half-pel motion compensation (MPEG-1 1993 MPEG-2 1994)

Bit-rate Reduction: 75%

Foreman 10 Hz, QCIF 100 frames

200

300

Thomas Wiegand: Recent Advances in Video Coding Wiegand:

slide 8

Rate [kbit/s]

Development of Video Coding

PSNR [dB] 40 38 36 34 32 30 28 0 100
Visual Comparison
Integer-pel motion compensation (H.261, 1991) Intra frame DCT coding (JPEG, 1990) Variable block size (16x16 4x4) + quarter-pel + multi-frame motion compensation (H.264/AVC, 2003) Variable block size (16x16 8x8) (H.263, 1996) + quarter-pel motion compensation (MPEG-4, 1998) Half-pel motion compensation (MPEG-1 1993 MPEG-2 1994)

Foreman 10 Hz, QCIF 100 frames

200

300

Thomas Wiegand: Recent Advances in Video Coding Wiegand:

slide 9

Rate [kbit/s]

JPEG

Foreman, QCIF, 10 Hz, 100 kbit/s H.264/AVC

Thomas Wiegand: Recent Advances in Video Coding Wiegand:

slide 10

Video encoder needs to decide between many options denoted with parameter vector p Constrained Problem:
D - Distortion R - Rate RT - Target Rate p - Parameter Vector

min D(p) s.t. R( p) R T

p

Unconstrained Lagrangian Formulation:

min D(p) + R( p)
p

with controlling the rate-distortion trade-off

Minimization tests the various modes in video coding

[Wiegand, et al., 1996] Thomas Wiegand: Recent Advances in Video Coding Wiegand:

slide 11

[Shoham & Gersho, 1989] Thomas Wiegand: Recent Advances in Video Coding Wiegand:

slide 12

Next Steps in Video Coding

PSNR [dB] 40 38 36 34 32 30 28 Rate [kbit/s] 0 100 200 300
slide 13

H.264/AVC (2003)

Thomas Wiegand: Recent Advances in Video Coding Wiegand:

New Joint Collaborative Team on Video Coding (JCT-VC) of ISO/IEC MPEG and ITU-T VCEG April 2010: Call for Proposals evaluated by JCT-VC
27 proposals received Significant video quality improvements relative to H.264/AVC High Profile for high and low delay JCT-VC decided to create Test Model under Consideration (TMuC)

October 2010: Creation of first Test Model Early 2013: intended finalization of standard
slide 15

Thomas Wiegand: Recent Advances in Video Coding Wiegand:

Input Video Signal

Coder Control Transform/ Scal./Quant. Decoder Scaling & Inv. Transform

Control Data Quant. Transf. coeffs

Split into Macroblocks 16x16 pixels

Entropy Coding De-blocking Filter 16x16 MB Types 0 8x8 Motion Estimation 8x8 Types 0 Output 16x8 Video Signal 0 1 Motion 8x4 Data 0 1 8x16 0 1 8x8 0 1 2 3
slide 16

Intra-frame Prediction MotionIntra/Inter Compensation

4x8 0 1

4x4 0 1 2 3

Thomas Wiegand: Recent Advances in Video Coding Wiegand:

Division of a picture into square blocks Blocks are assigned to quadtrees Maximum block size is signalled (e.g. 64x64) Quadtree-based subdivision of tree block into prediction and transform blocks Rectangular shapes Geometric partitions
slide 17

Thomas Wiegand: Recent Advances in Video Coding Wiegand:

=1

1. Extend motion vector by reference picture index 2. Provide reference pictures at decoder side

=4

=2 Current Picture

4 Prior Decoded Pictures as Reference

3. In case of bi-predictive coding (B pictures): decode 2 sets of motion parameters

[Wiegand, et al., 1999]

Thomas Wiegand: Recent Advances in Video Coding Wiegand:

slide 18

Flexible buffering of reference pictures generalizes B pictures: B pictures can be reference pictures (decoupling of concepts) Reference picture for a B picture can be any prior decoded picture

Motion-Compensated Interpolation
Longer Interpolation Filters
In H.264/AVC: 6-tap filters + bi-linear In HM: 12-tap interpolation filters

Generalized Interpolation
MOMS basis functions IIR pre-filtering of reference image (2-tap) Short FIR filters (4-taps)

Adaptive Interpolation filters

Transmission of filter coefficients in slice header Transmission of an index into a list of predefined filters

Directional interpolation filters

slide 19

Low-complexity interpolation

Thomas Wiegand: Recent Advances in Video Coding Wiegand:

Motion Representation
Quadtree-based splitting of blocks Motion Partition Merging

R
T L X
slide 20

Thomas Wiegand: Recent Advances in Video Coding Wiegand:

Motion Vector Coding

Adaptive Motion Vector Coding
Signaling of motion vector predictor

top left

top

top right

left

current block

Thomas Wiegand: Recent Advances in Video Coding Wiegand:

slide 21

bottom left

Transform Coding
Input Video Signal Coder Control Transform/ Scal./Quant. Decoder Scaling & Inv. Transform Entropy Coding De-blocking Filter Output Video Signal Motion Data Motion Estimation Control Data Quant. Transf. coeffs

Split into Macroblocks 16x16 pixels

Intra-frame Prediction MotionIntra/Inter Compensation

Thomas Wiegand: Recent Advances in Video Coding Wiegand:

slide 22

1. Partitioning into prediction blocks (intra or inter prediction) 2. Partitioning of prediction blocks into transform blocks (specifying the transform sizes)

Partitioning of a prediction block into transform blocks

Thomas Wiegand: Recent Advances in Video Coding Wiegand:

slide 23

Partitioning of a tree block into prediction blocks

Variable Transform Block Sizes

Transform sizes range from 4x4 to 32x32 Requirement on fast algorithms Additional new directional transforms proposed

Thomas Wiegand: Recent Advances in Video Coding Wiegand:

slide 24

Intra Prediction
Input Video Signal Coder Control Transform/ Scal./Quant. Decoder

H.264/AVC directional prediction

Q I J K L A a e i m B b f j n C c g k o D E F G H d h l p
4 Entropy1 5 6 Coding 0 7 2 8 3

Split into Macroblocks 16x16 pixels

Scaling & Inv. Transform

New directional predictions and De-blocking transforms

Intra-frame Prediction MotionIntra/Inter Compensation Filter Output Video Signal Motion Data Motion Estimation

Thomas Wiegand: Recent Advances in Video Coding Wiegand:

slide 25

Entropy Coding
Input Video Signal Coder Control Transform/ Scal./Quant. Decoder Scaling & Inv. Transform Entropy Coding De-blocking Filter Output Video Signal Motion Data Motion Estimation Control Data Quant. Transf. coeffs

Split into Macroblocks 16x16 pixels

Intra-frame Prediction MotionIntra/Inter Compensation

Thomas Wiegand: Recent Advances in Video Coding Wiegand:

slide 26

Entropy coding: reversible mapping from one data representation to another, more compact representation Entropy is lower bound on transmission rate [Shannon, 1948]

Problem complicated in video coding Video encoder generates sequence of symbols Sequence of symbols contains a mix of data types: control data, motion vectors, transform coefficients
slide 27

Thomas Wiegand: Recent Advances in Video Coding Wiegand:

Symbols in video codec correspond to a mix of data type: control data, motion vectors, transform coefficients Huffman coding: create a code for each data type Arithmetic coding: binarize data and arithmetically encode each binary symbol Example

{s1 , s2 ,..., s N } {b1 ,b2 ,b3 ,b4 ,...,bB }

Symbol 0 1 2 Binary Symbol 0 10
slide 28

11

Thomas Wiegand: Recent Advances in Video Coding Wiegand:

Simple construction method for entropy codes [Huffman, 1951] Minimizes average codeword length with constraint

Bounds on Rate Example

Redundancy 17.8 %
Thomas Wiegand: Recent Advances in Video Coding Wiegand:

slide 29

Alternative construction method for entropy codes

[Rissanen, 1976], [Pasco, 1976]

Achieves entropy bound asymptotically for

In video coding: context-adaptive binary arithmetic coding(CABAC), [Marpe, Schwarz, Wiegand, 2003]
update probability estimation

Context Modeling

Binarization

Probability Estimation

Coding Engine
slide 30

Bottleneck !

Adaptive binary arithmetic coder

Thomas Wiegand: Recent Advances in Video Coding Wiegand:

Arithmetic coding represents probability of binary symbols New concept: PIPE coding [Marpe, Schwarz, Wiegand 2010]

Thomas Wiegand: Recent Advances in Video Coding Wiegand:

slide 31

expected rate per bin [bit]

Uniform probability distribution of binary probabilities With slope of binary entropy function

1,0

entropy limit approximation by interval discretization interval representative

0,8

0,6 probability interval

0,4

0,2

0,0

Rate increase

0,1

0,2 0,3 probability p

0,4

0,5

Average Rate Increase (relative to entropy limit) f( p ) uniform K=1 12.47% K=2 3.67% K=4 1.01% K=8 0.27% K=12 0.12% K=16 0.07%
slide 32

Thomas Wiegand: Recent Advances in Video Coding Wiegand:

Assume fixed probability for each interval Create combined events of binary symbols Probability of symbol p Leaf node probability is

p = 0.4 q = 1 p q

prob. q3 q2 q2p qp p pq p2 0.216 0.144 0.240 0.240 0.160

codewords 11 001 01 10 000

Thomas Wiegand: Recent Advances in Video Coding Wiegand:

slide 33

bin sequence 000 001 01 10 11

codeword 11 001 01 10 000

Coding efficiency of arithmetic coding Complexity of Huffman coding

average codeword length increase [bits] 0.006 optimal code assumption (theoretical) 0.005 real V2V codes (with up to 65 codewords) 0.004 0.003 0.002 0.001 0.000 0 0.1 0.2 0.3 0.4 0.5 probability p
slide 34

0.12 % 0.24 %

Thomas Wiegand: Recent Advances in Video Coding Wiegand:

In-Loop Filtering
Input Video Signal Coder Control Transform/ Scal./Quant. Decoder Scaling & Inv. Transform Entropy Coding De-blocking Filter Output Video Signal Motion Data Motion Estimation Control Data Quant. Transf. coeffs

Split into Macroblocks 16x16 pixels

Intra-frame Prediction MotionIntra/Inter Compensation

Thomas Wiegand: Recent Advances in Video Coding Wiegand:

slide 35

In-Loop Filtering
Deblocking filter Adaptive Loop Filter
Wiener filter Transmission of filter coefficients and indication for regions

Thomas Wiegand: Recent Advances in Video Coding Wiegand:

slide 36

Coding Efficiency

Thomas Wiegand: Recent Advances in Video Coding Wiegand:

slide 37

Coding Efficiency
On average, current gains seem to be around 30% bit rate reduction Goal: 50% bit rate reduction against H.264/AVC HP at same quality Finalization: January 2013

Thomas Wiegand: Recent Advances in Video Coding Wiegand:

slide 38

H.264/AVC: everywhere today High-Efficiency Video Coding New project of MPEG and VCEG Started in April 2010 First Test Model in Oct. 2010 Currently roughly 30% bit rate reduction Goal: 50% bit rate reduction vs. H.264/AVC HP
slide 39

Finalization intended for early 2013

Thomas Wiegand: Recent Advances in Video Coding Wiegand:

HHI & TUB members and research associates D. Marpe H. Schwarz K. Mller P. Kauff R. Schfer other members of the HHI/TUB team

Thomas Wiegand: Recent Advances in Video Coding Wiegand:

slide 40