Notes on Infomation Entropy

Notes on Infomation Entropy

Notes on Infomation Entropy

ii

Contents
1 2 3 4 Introduction Analysis of the entropy function Example Some properties 1 1 3 4

Notes on Infomation Entropy

1/4

Javier Goizueta <jgoizueta@gmail.com> 2010-10-01

Introduction

Well consider two representative applications to illustrate the information entropy function: Information measure Biodiversity measure (or diversity, in general) Information measure will be done as in the classic statistical (Shannon) information theory: information is encoded as strings of symbols. The equivalence between them can be seen if we imagine this method of measuring the biodiversity of a region: a random walk in which species observed are registered as information and then quantity of information needed to record the observations is measured. An effective measure in both cases is the information entropy:
n

H = pi log pi
i=1

The pi are the probabilities or frequencies of symbols (information interpretation) or species individuals (diversity). The value n is the number of symbols or species. For frequencies, we have pi = xi where xi is the number of individuals of class i and N N = n xi i=1 The base used for the logarithms affects the scale of the measure.

Analysis of the entropy function

Each summand in H corresponds to a class i and has two factors: pi the frequency or probability of the class i log pi the unexpectancy of class i is the optimal length of code for the class, or equivalently, the information contants of nding one specimen of the class.

Notes on Infomation Entropy

2/4

Figure 1: Logarithm function We can interpret the logarithm as the length of a positional representation of a number: 1/x for x < 1 and x for x > 1. In our case we have 0 pi 1 and pi = 1, and the unexpectancy factor looks like this:

Notes on Infomation Entropy

3/4

We have, for pi = 0 the maximum surprise (impossible event) and for pi = 1 no surprise at all (sure fact).

Example

Heres an example with one degree of freedom, so it can easily be visualized. We have N = 2; the graph shows H and the terms for p1 and p2 ; note that the maximum H is at p1 = p2 = 1/2.

Notes on Infomation Entropy

4/4

The example can be interpreted as the information measure of a coin toss (maximum when both results are equiprobable) or as the diversity of the distributions of two classes (species). Uneveness reduces informacion (because expectancy is higher), reduces surprise, reduces diversity.

Some properties
1 N

If pi =

(equiprobable classes) then:

pi ln pi = ln N
So, the more classes there are, the higher the diversity value H. Since we use a logarithm measure (ln N$), doubling the number of classes increments by a contant the measure value.
1 For xed N the maximum value of H occurs when pi = N . The uneveness of the distribution reduces the diveristy measure.