TWOWAY COMMUNICATION
CHANNELS
CLAUDE E. SHANNON
MASSACHUSETTS INSTITUTE OF TECHNOLOGY _{C}_{A}_{M}_{B}_{R}_{I}_{D}_{G}_{E}_{,} MASSACHUSETTS
1. Introduction
A twoway communication channel is shown _{s}_{c}_{h}_{e}_{m}_{a}_{t}_{i}_{c}_{a}_{l}_{l}_{y} in _{f}_{i}_{g}_{u}_{r}_{e} 1. Here _{x}_{i}
is an input _{l}_{e}_{t}_{t}_{e}_{r} _{t}_{o} _{t}_{h}_{e} _{c}_{h}_{a}_{n}_{n}_{e}_{l} at terminal 1 and _{y}_{i} an _{o}_{u}_{t}_{p}_{u}_{t} _{w}_{h}_{i}_{l}_{e} _{x}_{2} is _{a}_{n}
xl
Y,
do^
TERMINAL I
CHANNEL
Y2
_{X}_{2}
TERMINAL 2
FIGURE 1
input at terminal 2 and Y2 the corresponding output. Once each second, _{s}_{a}_{y}_{,}
new inputs _{x}_{i} and x2 _{m}_{a}_{y} be chosen from _{c}_{o}_{r}_{r}_{e}_{s}_{p}_{o}_{n}_{d}_{i}_{n}_{g} _{i}_{n}_{p}_{u}_{t} _{a}_{l}_{p}_{h}_{a}_{b}_{e}_{t}_{s} and
put into the channel; outputs _{y}_{i} and Y2 may then be observed. These _{o}_{u}_{t}_{p}_{u}_{t}_{s}
will be related statistically to the inputs and perhaps historically to previous
inputs and outputs if the channel has _{m}_{e}_{m}_{o}_{r}_{y}_{.} The _{p}_{r}_{o}_{b}_{l}_{e}_{m} is to _{c}_{o}_{m}_{m}_{u}_{n}_{i}_{c}_{a}_{t}_{e} in both directions through the channel as effectively as possible. _{P}_{a}_{r}_{t}_{i}_{c}_{u}_{l}_{a}_{r}_{l}_{y}_{,} we
wish to determine what _{p}_{a}_{i}_{r}_{s} of _{s}_{i}_{g}_{n}_{a}_{l}_{l}_{i}_{n}_{g} rates _{R}_{1} _{a}_{n}_{d} _{R}_{2} for the two directions
can be _{a}_{p}_{p}_{r}_{o}_{a}_{c}_{h}_{e}_{d} with _{a}_{r}_{b}_{i}_{t}_{r}_{a}_{r}_{i}_{l}_{y} small error _{p}_{r}_{o}_{b}_{a}_{b}_{i}_{l}_{i}_{t}_{i}_{e}_{s}_{.}
X'
y,
^{N}^{W}
^{K} I
K 2
_{}}

d
Y2
FIGURE 2
Before _{m}_{a}_{k}_{i}_{n}_{g} these notions _{p}_{r}_{e}_{c}_{i}_{s}_{e}_{,} we _{g}_{i}_{v}_{e} some _{s}_{i}_{m}_{p}_{l}_{e} _{e}_{x}_{a}_{m}_{p}_{l}_{e}_{s}_{.} In _{f}_{i}_{g}_{u}_{r}_{e} 2
the twoway channel decomposes into two independentoneway noiseless _{b}_{i}_{n}_{a}_{r}_{y}
This work was _{s}_{u}_{p}_{p}_{o}_{r}_{t}_{e}_{d} in _{p}_{a}_{r}_{t} _{b}_{y} the U.S.
_{A}_{r}_{m}_{y} _{(}_{S}_{i}_{g}_{n}_{a}_{l} _{C}_{o}_{r}_{p}_{s}_{)}_{,} the U.b. Air Force _{(}_{O}_{f}_{f}_{i}_{c}_{e}
and the U.S. Navy (Office
_{o}_{f} _{S}_{c}_{i}_{e}_{n}_{t}_{i}_{f}_{i}_{c} _{R}_{e}_{s}_{e}_{a}_{r}_{c}_{h}_{,} Air Research and _{D}_{e}_{v}_{e}_{l}_{o}_{p}_{m}_{e}_{n}_{t} _{C}_{o}_{m}_{m}_{a}_{n}_{d}_{)}_{,}
of Naval Research).
611
_{6}_{1}_{2} _{F}_{O}_{U}_{R}_{T}_{H} _{B}_{E}_{R}_{K}_{E}_{L}_{E}_{Y} SYMPOSIUM: SHANNON
_{c}_{h}_{a}_{n}_{n}_{e}_{l}_{s} K1 and _{K}_{2}_{.} Thus _{x}_{i}_{,} x2, _{y}_{1} and Y2 are all binary variables and the opera
tion of the channel is defined by Y2 = xl and
y,
= _{x}_{2}_{.} _{W}_{e} _{c}_{a}_{n} here transmit in
each direction at rates up to one bit per second. Thus we can find codes whose
R2
R1
RI
I
FIGURE 3
_{r}_{a}_{t}_{e}_{s} _{(}_{R}_{1}_{,} R2) approximate as closely as desired any point in the square, figure 3, with arbitrarily small (in this case, zero) error probability.
In _{f}_{i}_{g}_{u}_{r}_{e} 4 all inputs and outputs are again binary and the operation is defined
_{M}_{O}_{D}_{.} 2 ADDER
Y,
FIGURE 4
Y2
by yi
_{=} _{Y}_{2} = _{x}_{l} _{+} _{X}_{2} _{(}_{m}_{o}_{d} _{2}_{)}_{.} _{H}_{e}_{r}_{e} _{a}_{g}_{a}_{i}_{n} it is _{p}_{o}_{s}_{s}_{i}_{b}_{l}_{e} to transmit one bit per
_{s}_{e}_{c}_{o}_{n}_{d} _{i}_{n} _{e}_{a}_{c}_{h} direction simultaneously, but the method is a bit more sophis
ticated. _{A}_{r}_{b}_{i}_{t}_{r}_{a}_{r}_{y} binary digits may be fed in at xi and x2 but, to decode, the
_{o}_{b}_{s}_{e}_{r}_{v}_{e}_{d} _{y} _{m}_{u}_{s}_{t} _{b}_{e} _{c}_{o}_{r}_{r}_{e}_{c}_{t}_{e}_{d} _{t}_{o} _{c}_{o}_{m}_{p}_{e}_{n}_{s}_{a}_{t}_{e} for the influence of the transmitted
_{x}_{.} Thus an observed
_{s}_{h}_{o}_{u}_{l}_{d} _{b}_{e} _{a}_{d}_{d}_{e}_{d} _{t}_{o} the _{j}_{u}_{s}_{t} transmitted _{x}_{i} _{(}_{m}_{o}_{d} _{2}_{)} to
_{d}_{e}_{t}_{e}_{r}_{m}_{i}_{n}_{e} _{t}_{h}_{e} transmitted _{x}_{2}_{.} Of course _{h}_{e}_{r}_{e}_{,} too, one may obtain lower rates
yi
_{t}_{h}_{a}_{n} the _{(}_{1}_{,} _{1}_{)} _{p}_{a}_{i}_{r} and _{a}_{g}_{a}_{i}_{n} approximate any point in the square, figure 3.
_{A} _{t}_{h}_{i}_{r}_{d} _{e}_{x}_{a}_{m}_{p}_{l}_{e} _{h}_{a}_{s} _{i}_{n}_{p}_{u}_{t}_{s} _{x}_{i} _{a}_{n}_{d} _{x}_{2} each from a _{t}_{e}_{r}_{n}_{a}_{r}_{y} alphabet and outputs
_{y}_{,} _{a}_{n}_{d} _{Y}_{2} _{e}_{a}_{c}_{h} _{f}_{r}_{o}_{m} _{a} _{b}_{i}_{n}_{a}_{r}_{y} _{a}_{l}_{p}_{h}_{a}_{b}_{e}_{t}_{.} _{S}_{u}_{p}_{p}_{o}_{s}_{e} that the probabilities of different
_{Y}_{2}_{)}_{,} conditional on various _{i}_{n}_{p}_{u}_{t} _{p}_{a}_{i}_{r}_{s} (xI, x2), are given by be seen that _{b}_{y} _{u}_{s}_{i}_{n}_{g} _{o}_{n}_{l}_{y} _{x}_{l} = 0 at terminal 1 it is possible to
output pairs _{(}_{Y}_{I}_{,} table I. It _{m}_{a}_{y}
_{s}_{e}_{n}_{d} _{o}_{n}_{e} _{b}_{i}_{t} _{p}_{e}_{r} _{s}_{e}_{c}_{o}_{n}_{d} in the 2  1 direction using only the input letters 1 _{a}_{n}_{d} _{2} _{a}_{t} _{t}_{e}_{r}_{m}_{i}_{n}_{a}_{l} _{2}_{,} _{w}_{h}_{i}_{c}_{h} _{t}_{h}_{e}_{n} result with _{c}_{e}_{r}_{t}_{a}_{i}_{n}_{t}_{y} in a and b respectively at
_{t}_{e}_{r}_{m}_{i}_{n}_{a}_{l} _{1}_{.} _{S}_{i}_{m}_{i}_{l}_{a}_{r}_{l}_{y}_{,} _{i}_{f} _{x}_{2} is held at _{0}_{,} transmission in the 1  2 direction is
_{p}_{o}_{s}_{s}_{i}_{b}_{l}_{e} at one bit _{p}_{e}_{r} second. _{B}_{y} dividing
_{i}_{n} _{t}_{h}_{e} ratio X to 1  X it is possible
_{t}_{h}_{e} time for use of these two strategies
_{t}_{o} _{t}_{r}_{a}_{n}_{s}_{m}_{i}_{t} _{i}_{n} _{t}_{h}_{e} two directions with
_{T}_{W}_{O}_{}_{W}_{A}_{Y} COMMUNICATION CHANNELS
TABLE I
_{Y}_{1}_{Y}_{}
Output Pair
61.3
aa 
ab 
ba 
bh 

_{_} 
_{_} 

00 
1/4 
1/4 
1/4 
1/4 

01 
1/2 
1/2 
0 
0 

02 
0 
0 
1/2 
1/2 

Input 
10 
1/2 
0 
1/2 
0 

Pair 
11 
1/4 
1/4 
1/4 
1/4 

12 
_{1}_{/}_{4} 
1/4 
1/4 
1/4 

_{2}_{0} 
0 
1/2 
0 
1/2 

21 
1/4 
1/4 
1/4 
1/4 

22 
1/4 
1/4 
1/4 
1/4 
_{a}_{v}_{e}_{r}_{a}_{g}_{e} rates R1 = 1X, R2 = X. Thus we can find codes approaching any _{p}_{o}_{i}_{n}_{t} in the _{t}_{r}_{i}_{a}_{n}_{g}_{u}_{l}_{a}_{r} _{r}_{e}_{g}_{i}_{o}_{n}_{,} _{f}_{i}_{g}_{u}_{r}_{e} 5. It is not difficult to see, and will follow
R2\
I
R,
FIGURE 5
from later results, that no point outside this trianigle cail be approached with codes of _{a}_{r}_{b}_{i}_{t}_{r}_{a}_{r}_{i}_{l}_{y} low error probability.
_{I}_{n} _{t}_{h}_{i}_{s} _{c}_{h}_{a}_{n}_{n}_{e}_{l}_{,} _{c}_{o}_{m}_{m}_{u}_{n}_{i}_{c}_{a}_{t}_{i}_{o}_{n} in the two directions might be called in _{c}_{o}_{m}_{p}_{a}_{t}_{i}_{b}_{l}_{e}_{.} Forward communication is _{p}_{o}_{s}_{s}_{i}_{b}_{l}_{e} only if X2 is held at zero. Other
_{w}_{i}_{s}_{e}_{,} all x1 letters are completely noisy. Conversely, backward communication
_{i}_{s} _{p}_{o}_{s}_{s}_{i}_{b}_{l}_{e} _{o}_{n}_{l}_{y} _{i}_{f} _{x}_{i} _{i}_{s} _{h}_{e}_{l}_{d} _{a}_{t} _{z}_{e}_{r}_{o}_{.} _{T}_{h}_{e} _{s}_{i}_{t}_{u}_{a}_{t}_{i}_{o}_{n} _{i}_{s} _{a} kind _{o}_{f} discrete analogue
_{t}_{o} _{a} _{c}_{o}_{m}_{m}_{o}_{n} _{p}_{h}_{y}_{s}_{i}_{c}_{a}_{l} _{t}_{w}_{o}_{}_{w}_{a}_{y} _{s}_{y}_{s}_{t}_{e}_{m}_{;} _{a} _{p}_{a}_{i}_{r} _{o}_{f} _{r}_{a}_{d}_{i}_{o} _{t}_{e}_{l}_{e}_{p}_{h}_{o}_{n}_{e} stations with
_{"}_{p}_{u}_{s}_{h}_{}_{t}_{o}_{}_{t}_{a}_{l}_{k}_{"} _{b}_{u}_{t}_{t}_{o}_{n}_{s} _{s}_{o} _{a}_{r}_{r}_{a}_{n}_{i}_{g}_{e}_{d} _{t}_{h}_{a}_{t} _{w}_{h}_{e}_{n} the button is _{p}_{u}_{s}_{h}_{e}_{d} the local
receiver is turned off. A _{f}_{o}_{u}_{r}_{t}_{h} _{s}_{i}_{m}_{p}_{l}_{e} _{e}_{x}_{a}_{m}_{p}_{l}_{e} of a twowvay channiiel, suggested by Blackwell, is the
_{b}_{i}_{n}_{a}_{r}_{y} _{m}_{u}_{l}_{t}_{i}_{p}_{l}_{y}_{i}_{n}_{g} channel. Here all inputs and outputs are binary and the _{o}_{p}_{e}_{r}_{a}_{t}_{i}_{o}_{n} is _{d}_{e}_{f}_{i}_{n}_{e}_{d} Yi = _{Y}_{2} = _{x}_{1}_{x}_{2}_{.} The _{r}_{e}_{g}_{i}_{o}_{n} of approachable rate pairs for this
_{c}_{h}_{a}_{n}_{n}_{e}_{l} _{i}_{s} _{n}_{o}_{t} _{k}_{n}_{o}_{w}_{n} _{e}_{x}_{a}_{c}_{t}_{l}_{y}_{,} but we shall later find bounds on it.
In _{t}_{h}_{i}_{s} _{p}_{a}_{p}_{e}_{r} we will _{s}_{t}_{u}_{d}_{y} the coding properties of twoway channels. In _{p}_{a}_{r}_{t}_{i}_{c}_{u}_{l}_{a}_{r}_{,} inner and outer bounds on the region of approachable rate pairs _{(}_{R}_{1}_{,} _{R}_{2}_{)} _{w}_{i}_{l}_{l} _{b}_{e} _{f}_{o}_{u}_{n}_{d}_{,} _{t}_{o}_{g}_{e}_{t}_{h}_{e}_{r} _{w}_{i}_{t}_{h} _{b}_{o}_{u}_{n}_{d}_{s} _{r}_{e}_{l}_{a}_{t}_{i}_{n}_{g} to the rate at which zero error _{p}_{r}_{o}_{b}_{a}_{b}_{i}_{l}_{i}_{t}_{y} can be approached. Certain topological properties of these
_{b}_{o}_{u}_{n}_{d}_{s} will be discussed and, finally, we will develop an expression describing
_{t}_{h}_{e} _{r}_{e}_{g}_{i}_{o}_{n} of _{a}_{p}_{p}_{r}_{o}_{a}_{c}_{h}_{a}_{b}_{l}_{e} rates in terms of a limiting process.
_{6}_{1}_{4} _{F}_{O}_{U}_{R}_{T}_{H} _{B}_{E}_{R}_{K}_{E}_{L}_{E}_{Y} SYMPOSIUM: SHANNON
2. Summary of results _{W}_{e} _{w}_{i}_{l}_{l} _{s}_{u}_{m}_{m}_{a}_{r}_{i}_{z}_{e} _{h}_{e}_{r}_{e}_{,} _{b}_{r}_{i}_{e}_{f}_{l}_{y} and somewhat roughly, the main results of _{t}_{h}_{e} _{p}_{a}_{p}_{e}_{r}_{.} _{I}_{t} _{w}_{i}_{l}_{l} _{b}_{e} _{s}_{h}_{o}_{w}_{n} _{t}_{h}_{a}_{t} _{f}_{o}_{r} _{a} _{m}_{e}_{m}_{o}_{r}_{y}_{l}_{e}_{s}_{s} _{d}_{i}_{s}_{c}_{r}_{e}_{t}_{e} channel there exists
_{a} _{c}_{o}_{n}_{v}_{e}_{x} _{r}_{e}_{g}_{i}_{o}_{n} _{G} _{o}_{f} _{a}_{p}_{p}_{r}_{o}_{a}_{c}_{h}_{a}_{b}_{l}_{e} rates. For any point in G, say (R1, R2), there _{e}_{x}_{i}_{s}_{t} _{c}_{o}_{d}_{e}_{s} _{s}_{i}_{g}_{n}_{a}_{l}_{l}_{i}_{n}_{g} _{w}_{i}_{t}_{h} _{r}_{a}_{t}_{e}_{s} _{a}_{r}_{b}_{i}_{t}_{r}_{a}_{r}_{i}_{l}_{y} _{c}_{l}_{o}_{s}_{e} to the point and with arbitrarily
_{s}_{m}_{a}_{l}_{l} _{e}_{r}_{r}_{o}_{r} _{p}_{r}_{o}_{b}_{a}_{b}_{i}_{l}_{i}_{t}_{y}_{.} _{T}_{h}_{i}_{s} _{r}_{e}_{g}_{i}_{o}_{n} _{i}_{s} _{o}_{f} _{t}_{h}_{e} form shown typically in figure 6,
R2
_{G}
RR
G
FIGURE 6
_{b}_{o}_{u}_{n}_{d}_{e}_{d} _{b}_{y} _{t}_{h}_{e} _{m}_{i}_{d}_{d}_{l}_{e} _{c}_{u}_{r}_{v}_{e} _{G} _{a}_{n}_{d} _{t}_{h}_{e} _{t}_{w}_{o} _{a}_{x}_{i}_{s} segments. This curve can be
_{d}_{e}_{s}_{c}_{r}_{i}_{b}_{e}_{d} _{b}_{y} a limiting expression involving mutual informations for long se _{q}_{u}_{e}_{n}_{c}_{e}_{s} of inputs and outputs. In addition, we find an inner and outer bound, GI and Go, which are more _{e}_{a}_{s}_{i}_{l}_{y} evaluated, involving, as they do, only a maximizing process over single
_{l}_{e}_{t}_{t}_{e}_{r}_{s} in the channel. _{G}_{O} is the set of points (R12, R21) that may be obtained by _{a}_{s}_{s}_{i}_{g}_{n}_{i}_{n}_{g} probabilities P{x1, x2} to the input letters of the channel (an arbitrary
_{j}_{o}_{i}_{n}_{t} distribution) and then evaluating
R12
(1)
_{E}
_{(}_{l}_{o}_{g} _{P}_{{}_{X}_{I} X2, _{Y}_{2}_{r}_{)} = _{E} _{P}_{{}_{x}_{l}_{x}_{2}_{y}_{2}_{}} log P{XlIX2, 2}
~~~~{XI2}1
XIZ2Y2
P{X1IX2}
_{R}_{=} E (logP{x21xI,Y1})
_{w}_{h}_{e}_{r}_{e} _{E}_{(}_{,}_{U}_{)} _{m}_{e}_{a}_{n}_{s} expectation of ,u. The inner bound GI is found in a similar way
_{b}_{u}_{t} _{r}_{e}_{s}_{t}_{r}_{i}_{c}_{t}_{i}_{n}_{g} the distribution to an independent one
_{T}_{h}_{e}_{n} _{G}_{,} _{i}_{s} _{t}_{h}_{e} convex hull of (R12, R21) points found under
_{P}_{{}_{x}_{i}_{,} x2} = P{xl}P{x2}.
_{t}_{h}_{i}_{s} restriction.
_{I}_{t} _{i}_{s} _{s}_{h}_{o}_{w}_{n} _{t}_{h}_{a}_{t} _{i}_{n} _{c}_{e}_{r}_{t}_{a}_{i}_{n} _{i}_{m}_{p}_{o}_{r}_{t}_{a}_{n}_{t} _{c}_{a}_{s}_{e}_{s} _{t}_{h}_{e}_{s}_{e} _{b}_{o}_{u}_{n}_{d}_{s} are identical so the
_{c}_{a}_{p}_{a}_{c}_{i}_{t}_{y} _{r}_{e}_{g}_{i}_{o}_{n} _{i}_{s} _{t}_{h}_{e}_{n} _{c}_{o}_{m}_{p}_{l}_{e}_{t}_{e}_{l}_{y} determined from the bounds. An example is
_{a}_{l}_{s}_{o} _{g}_{i}_{v}_{e}_{n} _{(}_{t}_{h}_{e} _{b}_{i}_{n}_{a}_{r}_{y} multiplying channel) where there is a discrepancy between the bounds.
The _{t}_{h}_{r}_{e}_{e} regions G,, G and _{G}_{o} are all convex and have the same intercepts _{o}_{n} _{t}_{h}_{e} _{a}_{x}_{e}_{s}_{.} These _{i}_{n}_{t}_{e}_{r}_{c}_{e}_{p}_{t}_{s} are the capacities in the two directions when the
other _{i}_{n}_{p}_{u}_{t} letter is fixed at its best value _{[}_{f}_{o}_{r} example, _{x}_{i} is held at the value
_{]}_{.} _{F}_{o}_{r} _{a}_{n}_{y} _{p}_{o}_{i}_{n}_{t} inside G the error
_{w}_{h}_{i}_{c}_{h} _{m}_{a}_{x}_{i}_{m}_{i}_{z}_{e}_{s} _{R}_{2}_{1} under variation of P
probabilities approach zero exponentially with the block length n. For any point
{x2}
_{T}_{W}_{O}_{}_{W}_{A}_{Y} COMMUNICATION CHANNELS
615
_{o}_{u}_{t}_{s}_{i}_{d}_{e} _{G} _{a}_{t} _{l}_{e}_{a}_{s}_{t} _{o}_{n}_{e} _{o}_{f} _{t}_{h}_{e} _{e}_{r}_{r}_{o}_{r} probabilities for the two codes will be bounded _{a}_{w}_{a}_{y} _{f}_{r}_{o}_{m} _{z}_{e}_{r}_{o} _{b}_{y} _{a} _{b}_{o}_{u}_{n}_{d} independent of the block length.
_{F}_{i}_{n}_{a}_{l}_{l}_{y}_{,} _{t}_{h}_{e}_{s}_{e} _{r}_{e}_{s}_{u}_{l}_{t}_{s} _{m}_{a}_{y} _{b}_{e} _{p}_{a}_{r}_{t}_{i}_{a}_{l}_{l}_{y} generalized to channels with certain _{t}_{y}_{p}_{e}_{s} _{o}_{f} _{m}_{e}_{m}_{o}_{r}_{y}_{.} _{I}_{f} _{t}_{h}_{e}_{r}_{e} _{e}_{x}_{i}_{s}_{t}_{s} an internal state of the channel such that it is _{p}_{o}_{s}_{s}_{i}_{b}_{l}_{e} to return to this state in a bounded number of steps (regardless of _{p}_{r}_{e}_{v}_{i}_{o}_{u}_{s}_{t}_{r}_{a}_{n}_{s}_{m}_{i}_{s}_{s}_{i}_{o}_{n}_{)} thentherewillexistagainacapacityregionG withsimilar properties. A limiting expression is given determining this region.
_{3}_{.} Basic definitions
memoryless twoway channel consists of a set of transition probabil
_{w}_{h}_{e}_{r}_{e} _{x}_{I}_{,} _{x}_{2}_{,} _{Y}_{l}_{,} _{Y}_{2} _{a}_{l}_{l} _{r}_{a}_{n}_{g}_{e} _{o}_{v}_{e}_{r} _{f}_{i}_{n}_{i}_{t}_{e} alphabets (not neces
_{A} _{b}_{l}_{o}_{c}_{k} _{c}_{o}_{d}_{e} _{p}_{a}_{i}_{r} _{o}_{f} _{l}_{e}_{n}_{g}_{t}_{h} _{n} _{f}_{o}_{r} _{s}_{u}_{c}_{h} _{a} _{c}_{h}_{a}_{n}_{n}_{e}_{l} _{w}_{i}_{t}_{h} _{M}_{1} messages in the for
_{w}_{a}_{r}_{d} _{d}_{i}_{r}_{e}_{c}_{t}_{i}_{o}_{n} _{a}_{n}_{d} _{M}_{2} _{i}_{n} _{t}_{h}_{e} _{r}_{e}_{v}_{e}_{r}_{s}_{e} _{d}_{i}_{r}_{e}_{c}_{t}_{i}_{o}_{n} _{c}_{o}_{n}_{s}_{i}_{s}_{t}_{s} _{o}_{f} _{t}_{w}_{o} _{s}_{e}_{t}_{s} of n functions
sarily the same).
itiesP{y1,Y2!x1,x2}
A discrete
_{(}_{2}_{)}
fO(Ml),fl(MI,Yll),f2(Ml)YllyY12) * go(rM2),gl(m2,Y21),92(m2, Y21, Y2),
fnl(Ml)Ylly *.*.* Yl,n1)
gXn(m2 , Y21 **, Y2,nl)
_{H}_{e}_{r}_{e} _{t}_{h}_{e} _{f} _{f}_{u}_{n}_{c}_{t}_{i}_{o}_{n}_{s} all take values in the _{x}_{i} alphabet and the g functions in
_{t}_{h}_{e} _{x}_{2} _{a}_{l}_{p}_{h}_{a}_{b}_{e}_{t}_{,} while _{m}_{l} takes values from 1 to M1 _{(}_{t}_{h}_{e} forward messages) and
_{m}_{2} _{t}_{a}_{k}_{e}_{s} _{v}_{a}_{l}_{u}_{e}_{s} from 1 to M2 (the backward messages). Finally y1i, for i = _{1}_{,} _{2}_{,}  _{*} _{*}_{,} n  _{1}_{,} takes values from _{t}_{h}_{e} Yi alphabet and similarly for Y2i. The f
_{f}_{u}_{n}_{c}_{t}_{i}_{o}_{n}_{s} _{s}_{p}_{e}_{c}_{i}_{f}_{y} _{h}_{o}_{w} _{t}_{h}_{e} _{n}_{e}_{x}_{t} _{i}_{n}_{p}_{u}_{t} letter at terminal 1 should be chosen as _{d}_{e}_{t}_{e}_{r}_{m}_{i}_{n}_{e}_{d} _{b}_{y} _{t}_{h}_{e} _{m}_{e}_{s}_{s}_{a}_{g}_{e} _{m}_{l} _{t}_{o} _{b}_{e} _{t}_{r}_{a}_{n}_{s}_{m}_{i}_{t}_{t}_{e}_{d} and the observed outputs
_{Y}_{l}_{l}_{,} _{Y}_{1}_{2}_{,} _{*} * * at terminal 1 up to the current time. Similarly
_{d}_{e}_{t}_{e}_{r}_{m}_{i}_{n}_{e} _{h}_{o}_{w} _{m}_{e}_{s}_{s}_{a}_{g}_{e} _{m}_{2} is encoded as a function of the
_{a}_{t} each time in the process.
_{A} _{d}_{e}_{c}_{o}_{d}_{i}_{n}_{g} _{s}_{y}_{s}_{t}_{e}_{m} _{f}_{o}_{r} a _{b}_{l}_{o}_{c}_{k} code _{p}_{a}_{i}_{r} of length n consists of a pair of func
_{Y}_{2}_{n}_{)}_{.} These functions take
the _{g} functions
information available
tions O(ml, _{Y}_{l}_{l}_{y}
_{Y}_{1}_{2}_{,} _{*} _{*}_{,} _{y}_{l}_{n}_{)} _{a}_{n}_{d} _{#}_{(}_{m}_{2}_{,} _{Y}_{2}_{1}_{,} _{Y}_{2}_{2}_{,} _{*} _{*} _{t}_{o} _{M}_{2} and 1 to M1 respectively.
values from 1
The decoding function so represents a way of deciding on the original trans
_{m}_{i}_{t}_{t}_{e}_{d} _{m}_{e}_{s}_{s}_{a}_{g}_{e} _{f}_{r}_{o}_{m} _{t}_{e}_{r}_{m}_{i}_{n}_{a}_{l} 2 _{g}_{i}_{v}_{e}_{n} the information available at terminal 1 at
_{t}_{h}_{e} _{e}_{n}_{d} _{o}_{f} a block of n received letters, namely, yll, y12, * * *, yln together with the
_{t}_{r}_{a}_{n}_{s}_{m}_{i}_{t}_{t}_{e}_{d} _{m}_{e}_{s}_{s}_{a}_{g}_{e} _{m}_{l} _{a}_{t} _{t}_{e}_{r}_{m}_{i}_{n}_{a}_{l} _{1}_{.} Notice that the transmitted _{s}_{e}_{q}_{u}_{e}_{n}_{c}_{e} _{x}_{l}_{l}_{,} _{x}_{1}_{2}_{,} _{*} _{*} _{*}_{,} _{x}_{l}_{n} _{a}_{l}_{t}_{h}_{o}_{u}_{g}_{h} _{k}_{n}_{o}_{w}_{n} at terminal 1 need not enter as an argument
_{i}_{n} _{t}_{h}_{e} _{d}_{e}_{c}_{o}_{d}_{i}_{n}_{g} _{f}_{u}_{n}_{c}_{t}_{i}_{o}_{n} _{s}_{i}_{n}_{c}_{e} _{i}_{t} is determined _{(}_{v}_{i}_{a} the _{e}_{n}_{c}_{o}_{d}_{i}_{n}_{g} functions) by _{m}_{l} _{a}_{n}_{d} the received sequence.
_{W}_{e} will _{a}_{s}_{s}_{u}_{m}_{e}_{,} _{e}_{x}_{c}_{e}_{p}_{t} when the contrary is stated, that all messages ml are
equiprobable (probability 1/Ml), that all messages m2 are equiprobable (prob
_{a}_{b}_{i}_{l}_{i}_{t}_{y} _{1}_{/}_{M}_{2}_{)}_{,} and that these events are _{s}_{t}_{a}_{t}_{i}_{s}_{t}_{i}_{c}_{a}_{l}_{l}_{y} independent. We also as
sume that the successive operations of the channel are independent,
_{(}_{3}_{)}
_{P}_{f}_{Y}_{l}_{l}_{,} _{Y}_{1}_{2}_{,} _{*}_{*}_{*}_{X}_{Y}_{l}_{n}_{,} Y21, _{Y}_{2}_{2}_{,}
YUnlX11, X12 **
Xln, X21} X22, * ,X2nj
n
= ii _{I}_{I}
P{Yli,Y2ilXli, X2I}.
_{6}_{1}_{6} FOURTH BERKELEY SYMPOSIUM: SHANNON
_{T}_{h}_{i}_{s} _{i}_{s} _{t}_{h}_{e} meaning of the memoryless condition. This implies that the probabil _{i}_{t}_{y} _{o}_{f} _{a} _{s}_{e}_{t} _{o}_{f} _{o}_{u}_{t}_{p}_{u}_{t}_{s} _{f}_{r}_{o}_{m} _{t}_{h}_{e} _{c}_{h}_{a}_{n}_{n}_{e}_{l}_{,} _{c}_{o}_{n}_{d}_{i}_{t}_{i}_{o}_{n}_{i}_{a}_{l} on the corresponding in puts, is the same as this probability conditional on these inputs and any previous inputs. _{T}_{h}_{e} _{s}_{i}_{g}_{n}_{a}_{l}_{l}_{i}_{n}_{g} _{r}_{a}_{t}_{e}_{s} _{R}_{,} _{a}_{n}_{d} _{R}_{2} _{f}_{o}_{r} _{a} _{b}_{l}_{o}_{c}_{k} code pair with M1 and M2 messages for the _{t}_{w}_{o} directions are defined by
(4)
R, =
^{I}
n
R2 = 
n
log l1,
log M2.
Given a code pair and a decoding system, together with the conditional prob abilitiesdefininga _{c}_{h}_{a}_{n}_{n}_{e}_{l} and our assumptions concerning message probability, _{i}_{t} _{i}_{s} _{p}_{o}_{s}_{s}_{i}_{b}_{l}_{e}_{,} in _{p}_{r}_{i}_{n}_{c}_{i}_{p}_{l}_{e}_{,} _{t}_{o} _{c}_{o}_{m}_{p}_{u}_{t}_{e} _{e}_{r}_{r}_{o}_{r} probabilities for a code. Thus one
could _{c}_{o}_{m}_{p}_{u}_{t}_{e} for each _{m}_{e}_{s}_{s}_{a}_{g}_{e} pair the probabilities of the various possible
received sequences, if these messages were transmitted by the given coding functions. Applying the decoding functions, the probability of an incorrect de
_{c}_{o}_{d}_{i}_{n}_{g} could be _{c}_{o}_{m}_{p}_{u}_{t}_{e}_{d}_{.} This could be averaged over all messages for each
_{d}_{i}_{r}_{e}_{c}_{t}_{i}_{o}_{n} to arrive at final _{e}_{r}_{r}_{o}_{r} _{p}_{r}_{o}_{b}_{a}_{b}_{i}_{l}_{i}_{t}_{i}_{e}_{s} _{P}_{e}_{l} and _{P}_{e}_{2} for the two directions. We will _{s}_{a}_{y} that a _{p}_{o}_{i}_{n}_{t} _{(}_{R}_{1}_{,} _{R}_{2}_{)} belongs to the capacity region G of a given _{m}_{e}_{m}_{o}_{r}_{y}_{l}_{e}_{s}_{s} _{c}_{h}_{a}_{n}_{n}_{e}_{l} K _{i}_{f}_{,} _{g}_{i}_{v}_{e}_{n} _{a}_{n}_{y} e _{>} _{0}_{,} there exists a block code and decoding
_{s}_{y}_{s}_{t}_{e}_{m} for the channel with signalling rates _{R}_{1} and _{R}_{*}_{2} satisfying _{a}_{n}_{d} _{1}_{R}_{2}  _{R}_{2} _{<} e and such that the error probabilities satisfy
JR,  RTI < e
Pel < e and
Pe2 < E
4. _{A}_{v}_{e}_{r}_{a}_{g}_{e} mutual information rates The _{t}_{w}_{o}_{}_{w}_{a}_{y} discrete memoryless channel
with finite _{a}_{l}_{p}_{h}_{a}_{b}_{e}_{t}_{s} has been _{Y}_{2}_{I}_{x}_{1}_{,} _{x}_{2}_{}}_{.} Here _{x}_{i} and _{x}_{2} are the _{Y}_{2} are the _{o}_{u}_{t}_{p}_{u}_{t} letters. Each of
defined _{b}_{y} a set of transition _{p}_{r}_{o}_{b}_{a}_{b}_{i}_{l}_{i}_{t}_{i}_{e}_{s}
P{yl,
_{i}_{n}_{p}_{u}_{t} _{l}_{e}_{t}_{t}_{e}_{r}_{s} _{a}_{t} terminals 1 and 2 and Yi and
these _{r}_{a}_{n}_{g}_{e}_{s} over its corresponding finite alphabet.
If a set of probabilities
P{xl}
is assigned (arbitrarily) to the different letters
another set of probabilities P
{x2} to the alphabet
for _{x}_{2} _{(}_{t}_{h}_{e}_{s}_{e} two taken statistically independent) then there will be definite cor
of the _{i}_{n}_{p}_{u}_{t} alphabet for xl and
responding probabilities for Yi and _{Y}_{2} and, in fact, for the set of four random
variables X1, _{X}_{2}_{,} _{Y}_{i}_{,} Y2, namely, P{xI, x2, _{Y}_{l}_{,}_{Y}_{2}_{}} = P {xl}P{x2}P{YI, _{Y}_{2}_{l}_{X}_{1}_{,} x2}
and so forth.
PJYlY = L
Z1,X2,1I2
_{P}_{{}_{x}_{l}_{,} x2,Yi, _{Y}_{2}_{'}_{f}
Thinking first intuitively, and in analogue to the oneway channel, we might
_{t}_{h}_{i}_{n}_{k} of the rate of transmission from _{x}_{i} to the _{t}_{e}_{r}_{m}_{i}_{n}_{a}_{l} 2 as _{g}_{i}_{v}_{e}_{n} _{b}_{y} _{H}_{(}_{x}_{1}_{)} 
_{H}_{(}_{x}_{l}_{l}_{x}_{2}_{,} _{Y}_{2}_{)}_{,} _{t}_{h}_{a}_{t} _{i}_{s}_{,} _{t}_{h}_{e} _{u}_{n}_{c}_{e}_{r}_{t}_{a}_{i}_{n}_{t}_{y} _{o}_{r} _{e}_{n}_{t}_{r}_{o}_{p}_{y} _{o}_{f} _{x}_{i} less its _{e}_{n}_{t}_{r}_{o}_{p}_{y} conditional
_{o}_{n} _{w}_{h}_{a}_{t} _{i}_{s} available at terminal _{2}_{,} _{n}_{a}_{m}_{e}_{l}_{y}_{,} _{Y}_{2} _{a}_{n}_{d} _{x}_{2}_{.} _{T}_{h}_{u}_{s}_{,} _{w}_{e} _{m}_{i}_{g}_{h}_{t} _{w}_{r}_{i}_{t}_{e}
TWOWAY COMMUNICATION CHANNELS
617
(6) 
R= H(XI)  H(xIlx2, y2) 

_{=} 
H [log P{Xl,X2, Y2}J 

_{E}_{[}_{l}_{o}_{g} P{xI!X2,Y}2] 

(7) 
R21 = H(x2)  H(x21xI,y1) 
_{=} _{E} _{[}_{l}_{o}_{g} P{XI,X2,Y} ]
_{E}_{H} _{l}_{o}_{g} PIX21r,Yl}].
I
P {X2}
I
_{T}_{h}_{e}_{s}_{e} _{a}_{r}_{e} _{t}_{h}_{e} _{a}_{v}_{e}_{r}_{a}_{g}_{e} _{m}_{u}_{t}_{u}_{a}_{l} _{i}_{n}_{f}_{o}_{r}_{m}_{a}_{t}_{i}_{o}_{n}_{s} _{w}_{i}_{t}_{h} _{t}_{h}_{e} _{a}_{s}_{s}_{i}_{g}_{n}_{e}_{d} input probabilities
between the input at one terminal and the inputoutput pair at the other ter
minal. _{W}_{e} might expect, then, that by suitable coding it should be possible to send in the two directions simultaneously with arbitrarily small error probabilities
_{c}_{l}_{o}_{s}_{e} _{t}_{o} _{R}_{1}_{2} and R21. The codes would be based on these
in generalization of the oneway channel. We will
and at _{r}_{a}_{t}_{e}_{s} arbitrarily
probabilities P{xi} and P{x2}
show that in fact
P{x2} which do this.
_{i}_{t} _{i}_{s} _{p}_{o}_{s}_{s}_{i}_{b}_{l}_{e} _{t}_{o} find _{c}_{o}_{d}_{e}_{s} based on the probabilities _{P}_{{}_{x}_{i}_{}} and
_{H}_{o}_{w}_{e}_{v}_{e}_{r} the capacity region may be larger than the set of rates available by this means. Roughly speaking, the difference comes about because of the prob
_{a}_{b}_{i}_{l}_{i}_{t}_{y} _{o}_{f} _{h}_{a}_{v}_{i}_{n}_{g} _{x}_{i} and _{X}_{2} _{d}_{e}_{p}_{e}_{n}_{d}_{e}_{n}_{t} random variables. In this case the appropri
ate mutual informations are given by _{H}_{(}_{x}_{2}_{l}_{x}_{l}_{)} 
_{H}_{(}_{x}_{i}_{I}_{x}_{2}_{,} _{Y}_{2}_{)}_{.} The above expressions for _{R}_{2}_{1} and R12 of course reduce to these
H(x2lxl,
_{y}_{i}_{)} and H(xIlx2) 
when _{x}_{i} and x2 are independent.
5. The distribution of information
_{T}_{h}_{e} _{m}_{e}_{t}_{h}_{o}_{d} _{w}_{e} _{f}_{o}_{l}_{l}_{o}_{w} _{i}_{s} based on random codes _{u}_{s}_{i}_{n}_{g} techniques similar to
_{t}_{h}_{o}_{s}_{e} _{u}_{s}_{e}_{d} in _{[}_{1}_{]} _{f}_{o}_{r} _{t}_{h}_{e} _{o}_{n}_{e}_{}_{w}_{a}_{y} channel. Consider a _{s}_{e}_{q}_{u}_{e}_{n}_{c}_{e} of n uses of the
channel _{o}_{r}_{,} mathematically, the product probability space. The inputs are
_{X}_{l} = (x,,,x12,
, _{x}_{l}_{n}_{)} and X2 = (x21,x22,  *, x2,) and the outputs Y, =
(Yll, Y12, _{*} _{*} _{*}_{,}
_{Y}_{l}_{n}_{)} and Y2 = _{(}_{Y}_{2}_{1}_{,} _{Y}_{2}_{2}_{,}
from the corresponding alphabets.
_{,} y2n), that is, sequence3 of n choices
_{T}_{h}_{e} _{c}_{o}_{n}_{d}_{i}_{t}_{i}_{o}_{n}_{a}_{l} _{p}_{r}_{o}_{b}_{a}_{b}_{i}_{l}_{i}_{t}_{i}_{e}_{s} for these blocks are _{g}_{i}_{v}_{e}_{n} _{b}_{y}
_{(}_{8}_{)}
_{P}_{{}_{Y}_{1}_{,} _{Y}_{2}_{[}_{X}_{1}_{,} _{X}_{2}_{}} = TI _{P}_{{}_{Y}_{l}_{k}_{,} Y2kIXlk, X2k}.
k
_{T}_{h}_{i}_{s} _{u}_{s}_{e}_{s} _{t}_{h}_{e} _{a}_{s}_{s}_{u}_{m}_{p}_{t}_{i}_{o}_{n} _{t}_{h}_{a}_{t} _{t}_{h}_{e} _{c}_{h}_{a}_{n}_{n}_{e}_{l} is _{m}_{e}_{m}_{o}_{r}_{y}_{l}_{e}_{s}_{s}_{,} or successive _{o}_{p}_{e}_{r}_{a}_{t}_{i}_{o}_{n}_{s}
_{i}_{n}_{d}_{e}_{p}_{e}_{n}_{d}_{e}_{n}_{t}_{.} We also associate a probability measure with input blocks Xi
_{a}_{n}_{d} _{X}_{2} _{g}_{i}_{v}_{e}_{n} _{b}_{y} _{t}_{h}_{e} _{p}_{r}_{o}_{d}_{u}_{c}_{t} measure of that taken for _{x}_{i}_{,} _{x}_{2}_{.} Thus
618 FOURTH BERKELEY SYMPOSIUM: SHANNON
^{(}^{9}^{)}
P{X1} = II P{Xlk}
k
P{X2}
1
=
k
P{X2k4
_{I}_{t} _{t}_{h}_{e}_{n} _{f}_{o}_{l}_{l}_{o}_{w}_{s} _{t}_{h}_{a}_{t} other probabilities are also the products of those for the individual letters. Thus, for example,
^{(}^{1}^{0}^{)}
P{X1,X2, Y1,Y2} = _{H} P{Xlk,X2k, Ylk,Y2k}
k
_{P}_{{}_{X}_{2}_{I}_{X}_{1}_{,} Y1} = H P{X2kIXlk, Ylk}.
k
_{T}_{h}_{e} _{(}_{u}_{n}_{a}_{v}_{e}_{r}_{a}_{g}_{e}_{d}_{)} mutual information between, say, X1 and the pair X2, Y2 may be written as a sum, as follows:
(11)
I(X1; X2, Y2) = log
P. {X1,JX2,7YJ}I P {Xlk, X2k,Y21,
X loP{X2,gY2
_{P}_{{}_{X}_{k}_{l}_{}} _{H}_{J} P{X2k, Y2k}
k
k
= L log
k
P
Xlk, X2k,
P
{Xlk}P
{X2k,
Y2k}
Y2k}
I(X1; X2, Y2) = E I(Xlk;X2k, Y2k)
k
_{T}_{h}_{u}_{s}_{,} _{t}_{h}_{e} _{m}_{u}_{t}_{u}_{a}_{l} information is, as usual in such independent situations, the sum of the individual mutual informations. Also, as usual, we may think of the
mutual _{i}_{n}_{f}_{o}_{r}_{m}_{a}_{t}_{i}_{o}_{n} as a random variable. Here I(XI; X2, Y2) takes on different
values with _{p}_{r}_{o}_{b}_{a}_{b}_{i}_{l}_{i}_{t}_{i}_{e}_{s} given by _{P}_{{}_{X}_{1}_{,} X2, Y2}. The distribution function for _{I}_{(}_{X}_{i}_{;} _{X}_{2}_{,} _{Y}_{2}_{)} will be denoted by _{P}_{1}_{2}_{(}_{Z}_{)} and similarly for I(X2; Xi, Y1)
_{(}_{1}_{2}_{)}
P12(Z) =
P{I(X1;X2, Y2) < Z}
P21(Z) = P{I(X2; XI, Y1) _
Z'.
_{S}_{i}_{n}_{c}_{e} each of the _{r}_{a}_{n}_{d}_{o}_{m} variables _{I}_{(}_{X}_{i}_{;} _{X}_{2}_{,} _{Y}_{2}_{)} and _{I}_{(}_{X}_{2}_{;} _{X}_{1}_{,} _{Y}_{1}_{)} is the sum of n _{i}_{n}_{d}_{e}_{p}_{e}_{n}_{d}_{e}_{n}_{t} random _{v}_{a}_{r}_{i}_{a}_{b}_{l}_{e}_{s}_{,} each with the same distribution, we have
_{t}_{h}_{e} _{f}_{a}_{m}_{i}_{l}_{i}_{a}_{r} statistical _{s}_{i}_{t}_{4}_{a}_{t}_{i}_{o}_{n} to which one _{m}_{a}_{y} _{a}_{p}_{p}_{l}_{y} various central limit _{t}_{h}_{e}_{o}_{r}_{e}_{m}_{s} and laws of _{l}_{a}_{r}_{g}_{e} numbers. The mean of the distributions _{P}_{1}_{2} and _{P}_{2}_{1}
_{w}_{i}_{l}_{l} _{b}_{e} _{n}_{R}_{1}_{2} _{a}_{n}_{d} _{n}_{R}_{2}_{1} _{r}_{e}_{s}_{p}_{e}_{c}_{t}_{i}_{v}_{e}_{l}_{y} and the variances n times the _{c}_{o}_{r}_{r}_{e}_{s}_{p}_{o}_{n}_{d}_{i}_{n}_{g}
variances for one letter. As n * _{o}_{o}_{,} p12[n(RI2  e)] O0 for any fixed e > 0, and
similarly for P21. In fact, this approach is exponential in n; p12[n(R12  E)] _ exp [A(e)n].
_{6}_{.} Random codes for the _{t}_{w}_{o}_{}_{w}_{a}_{y} channel
_{A}_{f}_{t}_{e}_{r} _{t}_{h}_{e}_{s}_{e} _{p}_{r}_{e}_{l}_{i}_{m}_{i}_{n}_{a}_{r}_{i}_{e}_{s} _{w}_{e} _{n}_{o}_{w} wish _{t}_{o} _{p}_{r}_{o}_{v}_{e} _{t}_{h}_{e} existence of codes with
certain error _{p}_{r}_{o}_{b}_{a}_{b}_{i}_{l}_{i}_{t}_{i}_{e}_{s} bounded by expressions involving the distribution
functions _{P}_{1}_{2} and _{p}_{2}_{j}_{.}
_{W}_{e} will _{c}_{o}_{n}_{s}_{t}_{r}_{u}_{c}_{t} _{a}_{n} ensemble of codes _{o}_{r}_{,} more _{p}_{r}_{e}_{c}_{i}_{s}_{e}_{l}_{y}_{,} of code _{p}_{a}_{i}_{r}_{s}_{,} one
TWOWAY COMMUNICATION CHANNELS
619
code for the 1  2 direction _{a}_{n}_{d} _{a}_{n}_{o}_{t}_{h}_{e}_{r} _{f}_{o}_{r} the 2  1 direction. Bounds will
be established on the error probabilities _{P}_{e}_{j} and Pe2 averaged over the ensemble, and from these will be shown the existence of particular codes in the ensemble
with related bounds on their error probabilities. The random ensemble of code pairs for such a twoway channel with M1 words in the 1  2 code and M2 words in the 2  1 code is constructed as follows. The M1 integers 1, 2, * *, M1 (the messages of the first code) are mapped in all possible ways into the set of input words X1 of length n. Similarly the integers 1, 2, * *, AM2 (the messages of the second code) are mapped in all possible ways into the set of input words X2 of length n. If there were _{a}_{,} possible input letters at terminal 1 and a2 input letters at ter minal 2, there will be _{a}_{l} and _{a}_{2} input words of length n and _{a}_{1}_{M}_{'} mappings in the first code and _{a}_{2}_{M}_{a} in the second code. We consider all pairs of these codes, a
total of a4M`aM2' pairs.
Each code pair is given a weighting, or probability, equal to the probability
of occurrence of that pair if the two mappings were done independently and an integer is mapped into a word with the assigned probability of that word. Thus, a code pair is given a weighting equal to the product of the probabilities as sociated with all the input words that the integers are mapped into for both
codes. This set of _{c}_{o}_{d}_{e} _{p}_{a}_{i}_{r}_{s} with these _{a}_{s}_{s}_{o}_{c}_{i}_{a}_{t}_{e}_{d}
pairs based on the assigned probabilities P{X1} and
probabilities _{w}_{e} call the
random ensemble _{o}_{f} _{c}_{o}_{d}_{e}
P{x2}.
_{A}_{n}_{y}
particular code pair of the ensemble could be used to transmit informa
tion, if we agreed upon a method of decoding. The method of decoding will here consist of two _{f}_{u}_{n}_{c}_{t}_{i}_{o}_{n}_{s} _{+}_{(}_{X}_{1}_{,} _{Y}_{1}_{)} _{a}_{n}_{d} _{4}_{,}_{(}_{X}_{2}_{,} _{Y}_{2}_{)}_{,} _{a} _{s}_{p}_{e}_{c}_{i}_{a}_{l} _{c}_{a}_{s}_{e} _{o}_{f} that defined above. Here X1 varies over the _{i}_{n}_{p}_{u}_{t} words of _{l}_{e}_{n}_{g}_{t}_{h} n at terminal _{1}_{,} and _{Y}_{,} over the _{p}_{o}_{s}_{s}_{i}_{b}_{l}_{e} received _{b}_{l}_{o}_{c}_{k}_{s} of _{l}_{e}_{n}_{g}_{t}_{h} n. The _{f}_{u}_{n}_{c}_{t}_{i}_{o}_{n} 4 takes values from
1 to M2 and represents the decoded message for a received Y1 if X1 was trans mitted. (Of course, _{X}_{I} is used in the decoding procedure in general since it may
influence Y1 and is, therefore, pertinent information for best decoding.) Similarly, _{4}_{,}_{(}_{X}_{2}_{,} Y2) takes values from 1 to M1 and is a way of deciding on the transmitted message _{m}_{l} on the basis of information available at terminal 2.
It should be noted here that the decoding functions, _{0} and ,6, need not be the
same for all code _{p}_{a}_{i}_{r}_{s} in the ensemble.
We also point out that the encoding functions for our random ensemble are more specialized than the general case described above. The sequence of input letters X1 for a given message m1 do not depend on the received letters at ter minal 1. In any particular code of the ensemble there is a strict mapping from messages to input sequences. Given an ensemble of code pairs as described above and decoding functions,
one could compute for each particular code pair two error probabilities for the two codes: _{P}_{e}_{j}_{,} the probability of error in decoding the first code, and Pe2 that
for the second. Here we are _{a}_{s}_{s}_{u}_{m}_{i}_{n}_{g} that the different _{m}_{e}_{s}_{s}_{a}_{g}_{e}_{s} in the first
code occur with _{e}_{q}_{u}_{a}_{l} _{p}_{r}_{o}_{b}_{a}_{b}_{i}_{l}_{i}_{t}_{y} _{1}_{/}_{M}_{l}_{,} and _{s}_{i}_{m}_{i}_{l}_{a}_{r}_{l}_{y} for the second.
_{6}_{2}_{0} _{F}_{O}_{U}_{R}_{T}_{H} _{B}_{E}_{R}_{K}_{E}_{L}_{E}_{Y} SYMPOSIUM: SHANNON
_{B}_{y} _{t}_{h}_{e} _{a}_{v}_{e}_{r}_{a}_{g}_{e} error probabilities for the ensemble of code pairs wve mean the _{a}_{v}_{e}_{r}_{a}_{g}_{e}_{s} _{E}_{(}_{P}_{.}_{1}_{)} _{a}_{n}_{d} _{E}_{(}_{P}_{e}_{2}_{)} _{w}_{h}_{e}_{r}_{e} _{e}_{a}_{c}_{h} _{p}_{r}_{o}_{b}_{a}_{b}_{i}_{l}_{i}_{t}_{y} of error for a particular code is weighted according to the weighting factor or probability associated with the _{c}_{o}_{d}_{e} _{p}_{a}_{i}_{r}_{.} _{W}_{e} _{w}_{i}_{s}_{h} _{t}_{o} _{d}_{e}_{s}_{c}_{r}_{i}_{b}_{e} a particular method of decoding, that is, a choice
_{o}_{f} _{0} _{a}_{n}_{d} _{i}_{P}_{,} _{a}_{n}_{d} _{t}_{h}_{e}_{n} _{p}_{l}_{a}_{c}_{e} _{u}_{p}_{p}_{e}_{r} _{b}_{o}_{u}_{n}_{d}_{s} _{o}_{n} _{t}_{h}_{e}_{s}_{e} _{a}_{v}_{e}_{r}_{a}_{g}_{e} error probabilities
for the ensemble.
7. Error probability for the ensemble of codes
_{T}_{H}_{E}_{O}_{R}_{E}_{M} _{1}_{.} _{S}_{u}_{p}_{p}_{o}_{s}_{e} probability assignments P{X1} and P{X2} in a discrete
_{m}_{e}_{m}_{o}_{r}_{y}_{l}_{e}_{s}_{s} _{t}_{w}_{o}_{}_{w}_{a}_{y} _{c}_{h}_{a}_{n}_{n}_{e}_{l} _{p}_{r}_{o}_{d}_{u}_{c}_{e} _{i}_{n}_{f}_{o}_{r}_{m}_{a}_{t}_{i}_{o}_{n} distributionfunctions _{p}_{1}_{2}_{(}_{Z}_{)} and
_{P}_{2}_{1}_{(}_{Z}_{)}_{.} _{L}_{e}_{t} _{M}_{1} = _{e}_{x}_{p} _{(}_{R}_{i}_{n}_{)} _{a}_{n}_{d} _{M}_{2} = _{e}_{x}_{p} _{(}_{R}_{2}_{n}_{)} be arbitrary integers and _{0}_{1} and 02 be _{a}_{r}_{b}_{i}_{t}_{r}_{a}_{r}_{y} positive numbers. Then the random ensemble of code pairs with M1 and _{M}_{2} messages has (uwth appropriate decoding functions) average error probabil
ities bounded as follows:
_{E}_{(}_{P}_{6}_{j}_{)} <
p12[n(Ri + 01)] + enal
E(Pe2) _ P21[n(R2 + 02)] + e
There will exist in the ensemble at least one code pair whose individual error prob _{a}_{b}_{i}_{l}_{i}_{t}_{i}_{e}_{s} _{a}_{r}_{e} _{b}_{o}_{u}_{n}_{d}_{e}_{d} _{b}_{y} _{t}_{w}_{o} _{t}_{i}_{m}_{e}_{s} _{t}_{h}_{e}_{s}_{e} expressions, that is, satisfying
(14)
Pe. ^{5} 2p12[n(Rl + 01)] + _{2}_{e}_{}_{n}_{'}
_{P}_{e}_{2} . 2p21[n(R2 + 02)] +
2en02.
_{T}_{h}_{i}_{s} _{t}_{h}_{e}_{o}_{r}_{e}_{m} is a generalization of theorem 1 in [1] which gives a similar _{b}_{o}_{u}_{n}_{d} _{o}_{n} _{P}_{e} _{f}_{o}_{r} _{a} _{o}_{n}_{e}_{}_{w}_{a}_{y} _{c}_{h}_{a}_{n}_{n}_{e}_{l}_{.} The proof for the twoway channel is a
generalization of that proof.
The statistical situation here is quite complex. There are several statistical
_{e}_{v}_{e}_{n}_{t}_{s} involved: the choice of messages _{m}_{l} and M2, the choice of code pair in
the ensemble of code pairs, and finally the statistics of the channel itself which
Y2IXi, X2}. The en
produces the output words _{Y}_{,} and Y2 according to _{P}_{{}_{Y}_{i}_{,}
_{s}_{e}_{m}_{b}_{l}_{e} _{e}_{r}_{r}_{o}_{r} _{p}_{r}_{o}_{b}_{a}_{b}_{i}_{l}_{i}_{t}_{i}_{e}_{s} we are _{c}_{a}_{l}_{c}_{u}_{l}_{a}_{t}_{i}_{n}_{g} are averages
events.
_{W}_{e} _{f}_{i}_{r}_{s}_{t} _{d}_{e}_{f}_{i}_{n}_{e} _{d}_{e}_{c}_{o}_{d}_{i}_{n}_{g} _{s}_{y}_{s}_{t}_{e}_{m}_{s} for the various codes in the ensemble. For a
_{g}_{i}_{v}_{e}_{n} _{0}_{2}_{,} _{d}_{e}_{f}_{i}_{n}_{e} _{f}_{o}_{r} each _{p}_{a}_{i}_{r} _{X}_{I}_{,} _{Y}_{,} a corresponding set of words in the X2 space
_{d}_{e}_{n}_{o}_{t}_{e}_{d} _{b}_{y} _{S}_{(}_{X}_{,}_{,} _{Y}_{,}_{)} _{a}_{s} follows:
_{o}_{v}_{e}_{r} _{a}_{l}_{l} these statistical
(15)
21lgP{fX, X2, Y2}
S(Xi,Y0)= {X2 _{l}_{o}_{g}_{P}
R
>n(R2 +
2
2)}.
_{T}_{h}_{a}_{t} _{i}_{s}_{,} _{S}_{(}_{X}_{,}_{,} _{Y}_{1}_{)} _{i}_{s} _{t}_{h}_{e} _{s}_{e}_{t} _{o}_{f} _{X}_{2} _{w}_{o}_{r}_{d}_{s} _{w}_{h}_{o}_{s}_{e} _{m}_{u}_{t}_{u}_{a}_{l} information with the
_{p}_{a}_{r}_{t}_{i}_{c}_{u}_{l}_{a}_{r} _{p}_{a}_{i}_{r} _{(}_{X}_{I}_{,} _{Y}_{,}_{)} _{e}_{x}_{c}_{e}_{e}_{d}_{s} _{a} _{c}_{e}_{r}_{t}_{a}_{i}_{n} _{l}_{e}_{v}_{e}_{l}_{,} _{n}_{(}_{R}_{2} _{+} _{0}_{2}_{)}_{.} In a similar way, we
_{d}_{e}_{f}_{i}_{n}_{e} _{a} _{s}_{e}_{t} _{S}_{'}_{(}_{X}_{2}_{,} _{Y}_{2}_{)} _{o}_{f} _{X}_{i} words for each _{X}_{2}_{,} _{Y}_{2} pair as follows:
_{(}_{1}_{6}_{)}
S'(X2, Y2) = 1X11 log P{X},
y } > n(Rl + 01)1
_{T}_{W}_{O}_{}_{W}_{A}_{Y} COMMUNICATION CHANNELS
621
_{W}_{e} _{w}_{i}_{l}_{l} _{u}_{s}_{e} _{t}_{h}_{e}_{s}_{e} _{s}_{e}_{t}_{s} _{S} _{a}_{n}_{d} _{S}_{'} _{t}_{o} _{d}_{e}_{f}_{i}_{n}_{e} _{t}_{h}_{e} _{d}_{e}_{c}_{o}_{d}_{i}_{n}_{g} procedure and to aid
_{i}_{n} _{o}_{v}_{e}_{r}_{b}_{o}_{u}_{n}_{d}_{i}_{n}_{g} the error probabilities. The decoding process will be as follows. In any particular code pair in the random ensemble, suppose message ml is _{s}_{e}_{n}_{t} _{a}_{n}_{d} _{t}_{h}_{i}_{s} _{i}_{s} _{m}_{a}_{p}_{p}_{e}_{d} _{i}_{n}_{t}_{o} _{i}_{n}_{p}_{u}_{t} _{w}_{o}_{r}_{d} _{X}_{1}_{.} _{S}_{u}_{p}_{p}_{o}_{s}_{e} _{t}_{h}_{a}_{t} _{Y}_{1} is received at ter _{m}_{i}_{n}_{a}_{l} _{1} _{i}_{n} _{t}_{h}_{e} corresponding block of n letters. Consider the subset of X2 words, _{S}_{(}_{X}_{1}_{,} _{Y}_{,}_{)}_{.} _{S}_{e}_{v}_{e}_{r}_{a}_{l} situations may occur. (1) There is no message m2 mapped into
_{t}_{h}_{e} _{s}_{u}_{b}_{s}_{e}_{t} _{S}_{(}_{X}_{1}_{,} _{Y}_{1}_{)} _{f}_{o}_{r} _{t}_{h}_{e} _{c}_{o}_{d}_{e} _{p}_{a}_{i}_{r} in question. In this case, _{X}_{i}_{,} _{Y}_{,} is decoded
(conventionally) as message number one. (2) There is exactly one message _{m}_{a}_{p}_{p}_{e}_{d} _{i}_{n}_{t}_{o} _{t}_{h}_{e} _{s}_{u}_{b}_{s}_{e}_{t}_{.} _{I}_{n} _{t}_{h}_{i}_{s} _{c}_{a}_{s}_{e}_{,} _{w}_{e} _{d}_{e}_{c}_{o}_{d}_{e} _{a}_{s} this particular message. _{(}_{3}_{)} _{T}_{h}_{e}_{r}_{e} _{a}_{r}_{e} _{m}_{o}_{r}_{e} _{t}_{h}_{a}_{n} _{o}_{n}_{e} _{s}_{u}_{c}_{h} messages. In this case, we decode as the smallest numbered such message. _{T}_{h}_{e} error probabilities that we are estimating would normally be thought of as calculated in the following manner. For each code pair one would calculate the error probabilities for all messages ml and M2, and from their averages get
the _{e}_{r}_{r}_{o}_{r} probabilities for that code pair. Then these error probabilities are
_{a}_{v}_{e}_{r}_{a}_{g}_{e}_{d} _{o}_{v}_{e}_{r} _{t}_{h}_{e} _{e}_{n}_{s}_{e}_{m}_{b}_{l}_{e} _{o}_{f} _{c}_{o}_{d}_{e} _{p}_{a}_{i}_{r}_{s}_{,} _{u}_{s}_{i}_{n}_{g} the appropriate weights or probabilities. We may, however, interchange this order of averaging. We may _{c}_{o}_{n}_{s}_{i}_{d}_{e}_{r} _{t}_{h}_{e} _{c}_{a}_{s}_{e}_{s} where _{a} _{p}_{a}_{r}_{t}_{i}_{c}_{u}_{l}_{a}_{r} _{m}_{l} and i2 are the messages and these are _{m}_{a}_{p}_{p}_{e}_{d} into particular X1 and X2, and the received words are Ti and F2. _{T}_{h}_{e}_{r}_{e} _{i}_{s} _{s}_{t}_{i}_{l}_{l}_{,} _{i}_{n} _{t}_{h}_{e} statistical picture, the range of possible code pairs, that is, mappings of the other M1  1 messages for one code and M2 1 for the _{o}_{t}_{h}_{e}_{r}_{.} _{W}_{e} _{w}_{i}_{s}_{h} _{t}_{o} _{s}_{h}_{o}_{w} _{t}_{h}_{a}_{t}_{,} _{a}_{v}_{e}_{r}_{a}_{g}_{e}_{d} _{o}_{v}_{e}_{r} this subset of codes, the probabil _{i}_{t}_{i}_{e}_{s} _{o}_{f} _{a}_{n}_{y} _{o}_{f} _{t}_{h}_{e}_{s}_{e} _{m}_{e}_{s}_{s}_{a}_{g}_{e}_{s} _{b}_{e}_{i}_{n}_{g} _{m}_{a}_{p}_{p}_{e}_{d} into subsets S'(X2, Y2) and S(X1, Fl) respectively do not exceed exp _{(}_{}_{n}_{O}_{l}_{)} and exp (nO2).
_{N}_{o}_{t}_{e} first that if X1 belongs to the set S'(X2, Y2) then by the definition of this set
(17)
log ^{I}^{X}
log
2 ^{2}
2}
> n(R1 + 01)
_{P}_{{}_{X}_{1}_{I}_{X}_{2}_{,} _{F}_{2}_{}} > P{Xi}en(Ri+00).
_{N}_{o}_{w} _{s}_{u}_{m} _{e}_{a}_{c}_{h} _{s}_{i}_{d}_{e} _{o}_{v}_{e}_{r} the set _{o}_{f} _{X}_{1} _{b}_{e}_{l}_{o}_{n}_{g}_{i}_{n}_{g} to S'(X2, F2) to obtain
_{(}_{1}_{8}_{)} 1
_{_}
_{E} P{X11X2, V2} > en(RI+0)
XI ES'(X%,Y2)
E
Xl ES'(Xi, '2)
P{X1}.
The left inequality
_{h}_{e}_{r}_{e} _{h}_{o}_{l}_{d}_{s} since a sum of _{d}_{i}_{s}_{j}_{o}_{i}_{n}_{t} probabilities cannot exceed
_{o}_{n}_{e}_{.} _{T}_{h}_{e} _{s}_{u}_{m} _{o}_{n} _{t}_{h}_{e} _{r}_{i}_{g}_{h}_{t} _{w}_{e} _{m}_{a}_{y} denote _{b}_{y} _{P}_{{}_{S}_{'}_{(}_{X}_{2}_{,} _{F}_{2}_{)}_{}}_{.} Combining the
_{f}_{i}_{r}_{s}_{t} _{a}_{n}_{d} _{l}_{a}_{s}_{t} _{m}_{e}_{m}_{b}_{e}_{r}_{s} _{o}_{f} this relation
_{(}_{1}_{9}_{)}
_{P}_{{}_{S}_{'}_{(}_{X}_{2}_{,} _{F}_{2}_{)}_{}} < en(RI+O1).
That _{i}_{s}_{,} the total probability
an _{e}_{x}_{p}_{r}_{e}_{s}_{s}_{i}_{o}_{n} involving n,
associated with _{a}_{n}_{y} set _{S}_{'}_{(}_{X}_{2}_{,} _{F}_{2}_{)} is bounded by
R1 and _{O}_{1} but independent of the particular X2, F2.
independently into the input
The probability of a particular
_{S}_{'}_{(}_{X}_{2}_{,} _{V}_{2}_{)} in the ensemble of code _{p}_{a}_{i}_{r}_{s} is _{j}_{u}_{s}_{t}
Now recall that the messages
_{w}_{o}_{r}_{d}_{s} _{u}_{s}_{i}_{n}_{g} the probabilities
message _{b}_{e}_{i}_{n}_{g} mapped into
P{S'(X2, Y2)}. The
were mapped
_{P}_{{}_{X}_{1}_{}} and P{X2}.
_{p}_{r}_{o}_{b}_{a}_{b}_{i}_{l}_{i}_{t}_{y} of _{b}_{e}_{i}_{n}_{g} in the complementary set is 1 
622 _{F}_{O}_{1}_{U}_{R}_{T}_{I} BERKELEY SYMPOSIUM: SHANNON
I'{S'(X2, Y2)' .
_{i}_{n}_{t}_{o} this complementary set is
'The _{p}_{r}_{o}_{b}_{a}_{b}_{i}_{l}_{i}_{t}_{y} that all messages other thanT _{m}_{l} wvill b)e _{i}_{l}_{a}_{l}_{p}_{l}_{p}_{e}_{d}
(20)
[1  P{S'(X2, Y2)}]AJ11
_ 1 
(M1l)I S'(X2, 12))
> 1 M1PYS'(X2, Y2)}
_ 1  Mle4l(RL+6l)
= 1  en9. _{I}_{l}_{e}_{r}_{e} _{w}_{e} used the inequality (t  )P _ 1  px, the relationi (19) anld finially the fact that M1 = exp _{(}_{n}_{R}_{l}_{)}_{.} We have established, then, that in the subset of cases being considered _{(}_{m}_{l}
_{a}_{n}_{i}_{d} _{M}_{2} _{m}_{a}_{p}_{p}_{e}_{d} _{i}_{n}_{t}_{o} X1 and _{X}_{2} and _{r}_{e}_{c}_{e}_{i}_{v}_{e}_{d} _{a}_{s} _{F}_{,} and _{F}_{2}_{)}_{,} with probability
_{a}_{t} _{l}_{e}_{a}_{s}_{t} 1  _{e}_{x}_{p} _{(}_{}_{n}_{O}_{l}_{)}_{,} there will be _{n}_{o} _{o}_{t}_{h}_{e}_{r} _{m}_{e}_{s}_{s}_{a}_{g}_{e}_{s} mapped into S'(X2, Y2).
A _{s}_{i}_{m}_{i}_{l}_{a}_{r} _{c}_{a}_{l}_{c}_{u}_{l}_{a}_{t}_{i}_{o}_{n} _{s}_{h}_{o}_{w}_{s} that with probability exceeding I  exp (0n2) there will be no other messages mapped into S(X1, Y1). These bounds, as lioted,
are independent of the particular X1, Y1 and X2, Y2.
_{W}_{V}_{e} _{n}_{o}_{w} _{b}_{o}_{u}_{n}_{d} the probability of the actual message _{m}_{1} beinig withini the
_{s}_{u}_{b}_{s}_{e}_{t} _{S}_{'}_{(}_{X}_{2}_{,} _{Y}_{2}_{)}_{.} _{R}_{e}_{c}_{a}_{l}_{l} that from the definition of _{P}_{1}_{2}_{(}_{Z}_{)}
_{(}_{2}_{1}_{)}
P12[n(Ri + Of)] = P {log
P{Xl, X2, _{Y}_{2}_{}} < _{n}_{(}_{R} _{+} _{0}_{,}_{)}
P{x1}P{X2, Y2} =
In the ensemble of code pairs a message m, say, is mapped into words X1 with
probabilities just equal to P{X1}. Consequently, the probability in the full en
_{s}_{e}_{m}_{b}_{l}_{e} _{o}_{f} _{c}_{o}_{d}_{e} _{p}_{a}_{i}_{r}_{s}_{,} _{m}_{e}_{s}_{s}_{a}_{g}_{e} _{c}_{h}_{o}_{i}_{c}_{e}_{s} _{a}_{n}_{d} channel statistics, that the actual
_{m}_{e}_{s}_{s}_{a}_{g}_{e} is _{m}_{a}_{p}_{p}_{e}_{d} into _{S}_{'}_{(}_{X}_{2}_{,} _{Y}_{2}_{)} is precisely 1  _{P}_{1}_{2}_{[}_{n}_{(}_{R}_{i} + _{0}_{1}_{)}_{]}_{.}
The _{p}_{r}_{o}_{b}_{a}_{b}_{i}_{l}_{i}_{t}_{y} that the actual message is mapped outside S'(X2, F2) is there fore given by _{p}_{1}_{2}_{[}
Mult mai mult decât documente.
Descoperiți tot ce are Scribd de oferit, inclusiv cărți și cărți audio de la editori majori.
Anulați oricând.