Information Theory

KIE 2008
COMMUNICATION SYSTEMS

Dr Effariza Hanafi
Department of Electrical Engineering
Email: effarizahanafi@um.edu.my

KIE 2008 1
•

KIE 2008 2
•

KIE 2008 . 3
•
∞
• 𝑋𝑖 𝑖=−∞
• 𝑋𝑖

KIE 2008 4
•
•
• 𝑋𝑖

• 𝒜
𝑝𝑖 𝑁
𝑖=1
• 𝒜 = {𝑎1 , 𝑎2 , … 𝑎𝑁 } 𝑋
𝑋 𝑝𝑖 = 𝑃(𝑋 =
𝑎𝑖 )
• 𝒜 = {0, 1} p = 0.5

KIE 2008 5
•
• 𝑎1 𝑎𝑁
𝑎1 𝑎𝑁
•
𝑎𝑁
•

KIE 2008 6
• 𝑎𝑗
𝑎𝑗1 𝑎𝑗2

𝑋𝑗 = (𝑋𝑗1 , 𝑋𝑗2 )

𝑎𝑗 = {𝑎𝑗1 , 𝑎𝑗2 }

𝑃 𝑋 = 𝑎𝑗 = 𝑃(𝑋𝑗1 = 𝑎𝑗1 )𝑃(𝑋𝑗2 = 𝑎𝑗2 )

•
• 𝑎𝑗
𝑎𝑗1 𝑎𝑗2

KIE 2008 7
• 𝑎𝑗 𝑝𝑗

𝑎𝑗 𝑎𝑗
𝑎𝑗 I(𝑝𝑗 )
ii. I(∙)
iii. I(∙)
𝑝𝑗 𝑝𝑗1 𝑝𝑗2 I 𝑝𝑗 = I 𝑝𝑗1 + I(𝑝𝑗2 )

•
I 𝑥 = −log(𝑥)
•

•
KIE 2008 8
•

𝑁 𝑁 1
H 𝑋 = − ෍ 𝑝𝑖 log 𝑝𝑖 = ෍ 𝑝𝑖 log
𝑖=1 𝑖=1 𝑝𝑖

• 0 log 0 = 0 [note: log(0)=]

• log 𝑥 log 2 𝑥

KIE 2008 9
•

• 𝑁

𝑁
1 1
H 𝑋 = −෍ log = log 𝑁
𝑖=1 𝑁 𝑁

• 0 ≤ H 𝑋 ≤ log 𝑁
H 𝑋 = −𝑝 log 𝑝 − 1 − 𝑝 log(1 − 𝑝)
KIE 2008 10
 𝒜=
1 1 1 1 1
{−2, −1,0, 1, 2} { , , , , }
2 4 8 16 16

KIE 2008 11
•
• 𝑿=
(𝑋1 , 𝑋2 , … , 𝑋𝑛 )
H 𝑿 = − σ𝑥1 ,𝑥2 ,…,𝑥𝑛 𝑝(𝑥1 , 𝑥2 , … , 𝑥𝑛 ) log 𝑝(𝑥1 , 𝑥2 , … , 𝑥𝑛 )
•

• 𝑋𝑛
𝑋1 𝑋𝑛−1
H 𝑋𝑛 |𝑋1 , … , 𝑋𝑛−1 = − σ𝑥1 ,…,𝑥𝑛 𝑝(𝑥1 , … , 𝑥𝑛 ) log 𝑝(𝑥𝑛 |𝑥1 , … , 𝑥𝑛−1 )

•
𝐻 𝑋, 𝑌 = 𝐻 𝑌 + 𝐻(𝑋|𝑌)
•
H 𝐗 = H 𝑋1 + H 𝑋2 |𝑋1 + ⋯ + H 𝑋𝑛 |𝑋1 , 𝑋2 , … , 𝑋𝑛−1
KIE 2008 12
 H 𝐗 = H 𝑋1 + H 𝑋2 |𝑋1 + ⋯ + H 𝑋𝑛 |𝑋1 , 𝑋2 , … , 𝑋𝑛−1

• (𝑋1 , 𝑋2 , … , 𝑋𝑛 )
𝑛

H 𝑿 = − ෍ 𝐻(𝑋𝑖 )
𝑖=1

KIE 2008 13
•
1
𝐻 = lim 𝐻(𝑋𝑛 |𝑋1 , 𝑋2 , … , 𝑋𝑛−1 ) = lim 𝐻(𝑋1 , 𝑋2 , … , 𝑋𝑛 )
𝑛→∞ 𝑛→∞ 𝑛

• Mutual Information between two discrete random variables 𝑋 𝑌:

𝐼 𝑋; 𝑌 = 𝐻 𝑋 − 𝐻 𝑋 𝑌
𝑝(𝑥|𝑦)
=෍ ෍ 𝑝 𝑥, 𝑦 log
𝑥∈𝒳 𝑦∈𝒴 𝑝 𝑥
𝑝(𝑥, 𝑦)
=෍ ෍ 𝑝 𝑥, 𝑦 log
𝑥∈𝒳 𝑦∈𝒴 𝑝 𝑥 𝑝(𝑦)

𝐼 𝑋; 𝑌 = 𝐼 𝑌; 𝑋 = 𝐻 𝑋 + 𝐻(𝑌) − 𝐻(𝑋, 𝑌)
KIE 2008 14
•
•

• 𝑋
𝑓𝑋 𝑥
∞
ℎ 𝑋 = − න 𝑓𝑋 𝑥 log 𝑓𝑋 𝑥 𝑑𝑥
−∞
0 log 0 = 0
∞ ∞
ℎ 𝑋, 𝑌 = − න න 𝑓 𝑥, 𝑦 log 𝑓 𝑥, 𝑦 𝑑𝑥𝑑𝑦 ℎ 𝑋|𝑌 = ℎ 𝑋, 𝑌 − ℎ(𝑌)
−∞ −∞
𝑋 𝑌:
𝐼 𝑋; 𝑌 = ℎ 𝑌 − ℎ 𝑌 𝑋 = ℎ 𝑋 − ℎ 𝑋 𝑌 = 𝐼(𝑌; 𝑋)
KIE 2008 15
KIE 2008 16
•

• 𝑛
𝑛𝑝1 𝑛𝑝2 𝑛𝑝𝑁
• 𝑛 𝑛→∞

KIE 2008 17
•
•
ς𝑁 𝑛𝑝𝑖
ς𝑁 𝑛𝑝𝑖 log 𝑝𝑖 𝑛 σ𝑁
𝑖=1 𝑝𝑖 log 𝑝𝑖
𝑃 𝑿=𝒙 ≈ 𝑝
𝑖=1 𝑖 = 𝑖=1 2 =2
−𝑛𝐻(𝑋)
=2

•
•
2−𝑛𝐻(𝑋)
2𝑛𝐻(𝑋)

KIE 2008 18
• 𝑁
𝑁𝑛 𝑛 2𝑛𝐻(𝑋)
•

•

𝑛
•

KIE 2008 19
• 2𝑛𝐻(𝑋) 𝑛𝐻(𝑋)

• 𝐻(𝑋)
•
• 𝐻 𝑋 = log 𝑁 2𝑛𝐻(𝑋) = 2𝑛 log 𝑁 = 𝑁 𝑛

KIE 2008 20
•

𝑛 2𝑛𝐻
•

KIE 2008 21
• 𝐻

•
•

KIE 2008 22
•

•
•

KIE 2008 23
 𝑎1 , 𝑎2 , 𝑎3 , 𝑎4 , 𝑎5

•
•
•
•
•
•

KIE 2008 24
•

• 𝑝𝑖
log(1/ 𝑝𝑖 )
1
• σ𝑁
𝑖=1 𝑝𝑖 log =
𝑝𝑖
H(X)

KIE 2008 25
KIE 2008 26
KIE 2008 27
•

𝑅ത = 𝐸 𝐿 = ෍ 𝑝 𝑥 𝑙(𝑥)
𝑥∈𝒳
𝑙 𝑥 : 𝑥
• H X ≤ 𝑅ത ≤ 𝐻 𝑋 + 1
H X
η=
• 𝑅ത
• 𝑛 𝑛
1
H 𝑿𝑛 ≤ 𝑅𝑛 ≤ 𝐻 𝑿𝑛 + 1, so 𝑅ത = 𝑅𝑛
𝑛
𝑅𝑛 :
1
• H 𝑿𝑛 = 𝑛H x H X ≤ 𝑅ത ≤ 𝐻 𝑋 +
𝑛

KIE 2008 28
•

KIE 2008 29
 𝑎1 , 𝑎2 , 𝑎3 .

KIE 2008 30
•
.
• .
•

KIE 2008 31
•

KIE 2008 32
 

KIE 2008 33
•

KIE 2008 34
•

KIE 2008 35
KIE 2008 36
•

•
•

KIE 2008 37
•

•
•
• 𝒳
• 𝒴
• 𝑝(𝒚|𝒙)
•
• 𝑦𝑖
𝑥𝑖 ,

KIE 2008 38
•
𝒚 ∈ 𝒴𝑛 𝒙 ∈ 𝒳𝑛:
𝑛
𝑝 𝒚𝒙 =ෑ 𝑃(𝑦𝑖 |𝑥𝑖 )
𝑖=1

•
•
• 𝜖 = 𝑃 1 0 = 𝑃(0|1)

KIE 2008 39
•

• 𝒳 𝒴
𝑌 =𝑋+𝑍 𝑍

• 𝑃
• 𝑛 𝑛
𝑁
1
෍ 𝑥𝑖2 ≤ 𝑃
𝑛
𝑖=1

KIE 2008 40
• 𝐻(𝑋)

•
•

KIE 2008 41
•

•
•

KIE 2008 42
•

• 𝑛
𝑛

• 𝑛 𝑛

𝑛𝜖

KIE 2008 43
• 𝑛 𝑛!
=
𝑛 𝑛𝜖 𝑛𝜖 𝑛 − 𝑛𝜖 ! 𝑛𝜖 !
•
𝑛
𝑛 −𝑛
𝑛! ≈ 𝑛 𝑒 2𝜋𝑛 ≈ 2𝑛𝐻𝑏 (𝜖)
𝑛𝜖

𝐻𝑏 𝜖 = −𝜖𝑙𝑜𝑔2 𝜖 − (1 − 𝜖)log 2 1 − 𝜖
• 2𝑛𝐻𝑏(𝜖)

•
2𝑛𝐻(𝑌)
•

2𝑛𝐻(𝑌)
𝑀 = 𝑛𝐻 (𝜖) = 2𝑛(𝐻 𝑌 −𝐻𝑏 (𝜖))
2 𝑏
KIE 2008 44
• 𝑛

log 2 M = 𝑛(𝐻 𝑌 − 𝐻𝑏 (𝜖))

• The transmission rate per channel use is

log2 M
R= = 𝐻 𝑌 − 𝐻𝑏 (𝜖)
𝑛

• Since 𝜖 cannot be controlled, to maximize the

transmission rate over the channel, we need to find
the input distribution 𝑃(𝑥) that maximizes 𝐻 𝑌

KIE 2008 45
 log2 M
R= = 𝐻 𝑌 − 𝐻𝑏 (𝜖)
𝑛

•
R = 1 − 𝐻𝑏 (𝜖)
•
• 𝐶 = 1 − 𝐻𝑏 (𝜖)

• 𝜖 𝜖

KIE 2008 46
•
𝐶 = max 𝐼(𝑋; 𝑌)
𝑃(𝑥)

𝐼 𝑋; 𝑌 :
•
•
•
•
1 1𝑛 𝑃 1 𝑃
𝐶= log 𝑀 = log(1 + ) = log(1 + )
𝑛 𝑛2 𝑃𝑁 2 𝑃𝑁
𝑃
•
1 𝑃
𝐶 = log(1 + )
2 𝑁𝑜 𝑊
𝑃
𝐶 = 𝑊 log(1KIE+ ) bits/sec 47
2008 𝑁𝑜 𝑊
KIE 2008 48
 𝑃
𝐶 = 𝑊 log(1 + 𝑁 ) bits/sec
𝑜𝑊

•
•

KIE 2008 49
•
𝑃 𝑃 𝑃
lim 𝑊 log(1 + )≈ log 𝑒 = 1.44
𝑊→∞ 𝑁𝑜 𝑊 𝑁0 𝑁0

•
𝑅<𝐶 𝑃
𝑅 < 𝑊 log 1 +
• 𝑁0 𝑊
𝑅
• 𝑟= ,
𝑊
𝑃 𝐸𝑏
r < log 1 + r < log 1 + 𝑟
𝑁0 𝑊 𝑁0

𝐸𝑏 :
𝐸𝑏
• 𝑟→ 0, = ln 2 ≈ −1.6𝑑𝐵
𝑁0
•
𝐸𝑏 𝐸𝑏 2𝑟 − 1
> ln 2 = ≈ ln 2 (𝑟 → 0)
𝑁0 𝑁0 𝑟
KIE 2008 50
 
