SGGSIE&T, Nanded

Department of E&TC
Information Theory and Coding- Problem Set 1
(sbbhosle@sggs.ac.in)

1. Identify in each of the following cases whether the 4. Consider a Discrete Memoryless Channel (DMC)
sample space is countable or uncountable. Justify your with input X and output Y . The input takes
answer. values from the alphabate of equiprobable symbols
{x1 , x2 } and the output takes values in the alphabet
a. Number of electrons moving to conduction band
{y1 , y2 , y3 , y4 }. The transition probability matrix for
from valence band in a given time interval due
the channel is shown below
to thermal or optical excitation
 
b. The amount of charge that flows into earth due 0.1 0.2 0.3 0.4
P =
to lightning 0.5 0.3 0.1 0.1

c. Tossing of infinite number of coins where, P (i, j) = Pr(Y = yj |X = xi ). Given that the
symbol received is y1 what is the probability that the
d. The waiting time in queue before getting service
symbol transmitted was x1 ? Compute the mutual in-
e. The amount of water in a dam at any given time formation I(y1 , x1 ). Determine entropies of X and Y .

2. Probability is a set function P(·) which assigns, to

any event (set) A, a number P(A) called probability
of A and which satisfies the following axioms
1. P(A) ≥ 0
5. A coin with P(H) = p is tossed until the first
head occurs. Let X denotes the number of coin flips
required. Find the entropy H(X) in bits. The follow-
ing expressions may be useful:

2. P(Ω) = 1 ∞
X 1
∞
X r
n
r = , nrn =
3. If A ∩ B = Φ, then P(A ∪ B) = P(A) + P(B) 1−r (1 − r)2
n=0 n=0

Prove that the conditional probability defined as

6. Let X be a random variable taking discrete values
P(A ∩ M )
P(A|M ) = and Y is defined as Y = 2X . Derive the relationship
P(M )
between H(X) and H(Y ).
is indeed the probability.

7. For random variables X and Y define ρ(X, Y ) =

3. Two fair dice are thrown and their outcomes are
recorded as X and Y respectively. The random vari-
ables U and V are defined as
U = max(X, Y )
V = min(X, Y )
Compute the joint distribution of U and V .
H(X|Y ) + H(Y |X). show that ρ(X, Y ) = 0, if there is
a bijection from X to Y . (Bijection is a mapping that
is both one-to-one (an injection) and onto (a surjec-
tion), i.e. a function which relates each member of a
set S (the domain) to a separate and distinct member
of another set T (the range), where each member in T
also has a corresponding member in S.)