427

Appendix A

Summary of
Basic Formulae
Summary
Two sets of tables are provided for reference. The first records the definition, and
the first two moments of the most common probability distributions used in this
volume. The second records the basic elements of standard Bayesian inference
processes for a number of special cases. In particular, it records the appropriate
likelihood function, the sufficient statistics, the conjugate prior and corresponding posterior and predictive distributions, the reference prior and corresponding
reference posterior and predictive distributions.

A.1

PROBABILITY DISTRIBUTIONS

The first section of this Appendix consists of a set of tables which record the
notation, parameter range, variable range, definition, and first two moments of the
probability distributions (discrete and continuous, univariate and multivariate) used
in this volume.

428

A. Summary of Basic Formulae

Univariate Discrete Distributions

Br(x | ) Bernoulli (p. 115)
0<<1

x = 0, 1

p(x) = x (1 )1x
V [x] = (1 )

E[x] =
Bi(x | , n) Binomial (p. 115)
0 < < 1, n = 1, 2, . . .
 
n x
p(x) =
(1 )nx
x

x = 0, 1, . . . , n

E[x] = n

V [x] = n(1 )

Bb(x | , , n) Binomial-Beta (p. 117)

> 0, > 0, n = 1, 2, . . .
 
n
p(x) = c
( + x)( + n x)
x
E[x] = n

x = 0, 1, . . . , n
c=

( + )
()()( + + n)

V [x] =

n ( + + n)
( + )2 ( + + 1)

Hy(x | N, M, n) Hypergeometric (p. 115)

N = 1, 2, . . .
M = 1, 2, . . .
n = 1, . . . , N + M
 

N
M
p(x) = c
x
nx
E[x] = n

N
N +M

x = a, a + 1, . . . , b
a = max(0, n M )
b = min(n, N )

1
N +M
c=
n
V [x] =

nN M N + M n
(N + M )2 N + M 1

429

A.1 Probability Distributions

Univariate Discrete Distributions (continued)

Nb(x | , r) Negative-Binomial (p. 116)
0 < < 1, r = 1, 2, . . .


r+x1
p(x) = c
(1 )x
r1
E[x] = r

x = 0, 1, 2, . . .
c = r
V [x] = r

1
2

Nbb(x | , , r) Negative-Binomial-Beta (p. 118)

> 0, > 0, r = 1, 2 . . .
x = 0, 1, 2, . . .


( + x)
( + )( + r)
r+x1
p(x) = c
c=
r1
( + + r + x)
()()


r
r
++r1
r
E[x] =
V [x] =
+
1
( 1)
( 2)
( 1)( 2)
Pn(x | ) Poisson (p. 116)
>0

x
p(x) = c
x!
E[x] =

x = 0, 1, 2, . . .
c = e
V [x] =

Pg(x | , , n) Poisson-Gamma (p. 119)

> 0, > 0, > 0

x = 0, 1, 2, . . .

( + x)
x
x!
( + )+x

E[x] =

()



V [x] =
1+

p(x) = c

c=

430

A. Summary of Basic Formulae

Univariate Continuous Distributions

Be(x | , ) Beta (p. 116)

> 0, > 0

0<x<1

p(x) = c x1 (1 x)1

c=

E[x] =

( + )
()()

V [x] =

( + )2 ( + + 1)

Un(x | a, b) Uniform (p. 117)

b>a
p(x) = c

a<x<b
c = (b a)1

E[x] = 12 (a + b)

V [x] =

1
12 (b

a)2

Ga(x | , ) Gamma (p. 118)

> 0, > 0
1 x

p(x) = c x

E[x] = 1

x>0

c=
()
V [x] = 2

Ex(x | ) Exponential (p. 118)

>0
p(x) = c ex
E[x] = 1/

x>0
c=
V [x] = 1/2

Gg(x | , , n) Gamma-Gamma (p. 120)

> 0, > 0, n > 0
xn1
p(x) = c
( + x)+n
E[x] = n

x>0
( + n)
c=
() (n)
V [x] =

2 (n2 + n( 1))
( 1)2 ( 2)

431

A.1 Probability Distributions

Univariate Continuous Distributions (continued)

2 (x | ) = 2

Chi-squared (p. 120)

>0

x>0
(1/2)/2
c=
(/2)
V [x] = 2

p(x) = c x(/2)1 ex/2

E[x] =

2 (x | , ) Non-central Chi-squared (p. 121)

> 0, > 0
  



p(x) =
Pn i 
2 (x | + 2i)
2
i=0

x>0

E[x] = +

V [x] = 2( + 2)

Ig(x | , ) Inverted-Gamma (p. 119)

> 0, > 0

x>0

c=
()

p(x) = c x(+1) e/x

E[x] =

V [x] =

2
( 1)2 ( 2)

1 (x | ) Inverted-Chi-squared (p. 119)

>0
2

p(x) = c x(/2+1) e1/2x

E[x] =

1
2

x>0
(1/2)/2
c=
(/2)
V [x] =

2
( 2)2 ( 4)

Ga1/2 (x | , ) Square-root Inverted-Gamma (p. 119)

> 0, > 0
2

p(x) = c x(2+1) e/x

( 1/2)
E[x] =
()

x>0
2
c=
()
V [x] =

E[x]2
1

432

A. Summary of Basic Formulae

Univariate Continuous Distributions (continued)

Pa(x | , ) Pareto (p. 120)
x < +
c =

> 0, > 0
p(x) = c x(+1)
E[x] =

>1

V [x] =

2
(1)2 (2)

>2

Ip(x | , ) Inverted-Pareto (p. 120)

> 0, > 0
p(x) = c x1
E[x] = 1 ( + 1)1

0 < x < 1
c =
V [x] = 2 ( + 1)2 ( + 2)1

N(x | , ) Normal (p. 121)

< < +, > 0


p(x) = c exp 12 (x )2
E[x] =

< x < +
c = 1/2 (2)1/2
V [x] = 1

St(x | , , ) Student t (p. 122)

< < +, > 0, > 0

(+1)/2
p(x) = c 1 + 1 (x )2

< x < +

 1/2
12 ( + 1)

c=
1

( 2 )

E[x] =

V [x] = 1 ( 2)1

F(x | , ) = F,

Snedecor F (p. 123)

> 0, > 0
/21

p(x) = c
E[x] =

x
( + x)(+)/2

,
2

>2

x>0


12 ( + ) /2 /2
c=
( 12 )( 12 )
V [x] =

2 2 ( + 2) ,
>4
( 2)2 ( 4)

433

A.1 Probability Distributions

Univariate Continuous Distributions (continued)
Lo(x | , ) Logistic (p. 122)
< < +, > 0
< x < +



2
p(x) = 1 exp 1 (x ) 1 + exp 1 (x )
V [x] = 2 2 /3

E[x] =

Multivariate Discrete Distributions

Muk (x | , n) Multinomial (p. 133)
= (1 , . . . , k )
k
0 < i < 1,
=1  1
n = 1, 2, . . .
k+1

n!
p(x) = k+1
x ,
=1 x ! =1

k+1

x = (x1 , . . . , xk )
k
=1 x n
xi = 0, 1, 2, . . .
k
k


=1
 , xk+1 = n
x
=1

=1

V [xi ] = ni (1 i )

E[xi ] = ni

C[xi , xj ] = ni j

Mdk (x | , n) Multinomial-Dirichlet (p. 135)

= (1 , . . . , k+1 )
i > 0
n = 1, 2, . . .
k+1
 [x ]

p(x) = c
x !
=1

[s] =

s

=1 (

E[xi ] = npi
i
pi = k+1
=1

+  1)

x = (x1 , . . . , xk )
xi = 0, 1, 2, . . .
n
=1 xl n
c= 


n!

k+1
=1

xk+1 = n

[n]


k

n+

=1

x

k+1

npi (1 pi )


n + k+1
=1 
C[xi , xj ] =
npi pj
k+1
1 + =1 
V [xi ] =

1+

=1
k+1
=1

434

A. Summary of Basic Formulae

Multivariate Continuous Distributions

Dik (x | ) Dirichlet (p. 134)
= (1 , . . . , k+1 )
i > 0
k
k

k+1 1 

1
p(x) = c 1
x
x 
=1

E[xi ] = k+1
=1

=1

x = (x1 , . . . , xk )
k
0 < xi < 1,
=1 x 1
k+1
 )
(
c = k+1=1
=1 ( )

E[xi ](1 E[xi ])

V [xi ] =

1 + k+1
=1 

C[xi , xj ] =

E[xi ]E[xj ]

1 + k+1
=1 

Ng(x, y | , , , ) Normal-Gamma (p. 136)

, > 0, > 0, > 0,
x , y > 0
p(x, y) = N(x | , y) Ga(y | , )
E[x] =
E[y] = 1
V [x] = 1 ( 1)1
p(x) = St(x | , 1 , 2)
Nk (x | , )

V [y] = 2

Multivariate Normal (p. 136)

= (1 , . . . , k ) k
symmetric positive-definite


p(x) = c exp 12 (x )t (x )

c = | | 1/2 (2)k/2

E[x] =

V [x] = 1

Pa2 (x, y | , 0 , 1 )

x = (x1 , . . . , xk ) k

Bilateral Pareto (p. 141)

(0 , 1 ) 2 , 0 < 1 , > 0
p(x, y) = c (y x)(+2)
0 1
1 0
E[x] =
E[y] =
1
1

(x, y) 2 , x < 0 , y > 1

c = ( + 1)(1 0 )
(1 0 )2
V [x] = V [y] =
( 1)2 ( 2)

435

A.1 Probability Distributions

Multivariate Continuous Distributions (continued)
Ngk (x, y | , , , ) Multivariate Normal-Gamma (p. 140)
< i < +, > 0, > 0
symmetric positive-definite

(x, y) = (x1 , . . . , xk , y)
< xi < , y > 0

p(x, y) = Nk (x | , y) Ga(y | , )
E[x, y] = (, 1 ),

V [x] = ( 1)1 1 ,

p(x) = Stk (x | , 1 , 2)

V [y] = 2

p(y) = Ga(y | , )

Nwk (x, y | , , , ) Multivariate Normal-Wishart (p. 140)

< i < +, > 0
2 > k 1
symmetric non-singular

x = (x1 , . . . , xk )
< xi < +
y symmetric positive-definite

p(x, y) = Nk (x | , y) Wik (y | , )
E[x, y] = {, 1 }

V [x] = ( 1)1 1

p(x) = Stk (x | , 1 , 2)

p(y) = Wik (y | , )

Stk (x | , , ) Multivariate Student (p. 139)

< i < +, > 0
x = (x1 , . . . , xk )
symmetric positive-definite
< xi < +



(+k)/2
12 ( + k)
1
t
p(x) = c 1 + (x ) (x )
c=
||1/2

( 12 )()k/2
E[x] = ,

V [x] = 1 ( 2)1

Wik (x | , ) Wishart (p. 138)

2 > k 1
symmetric non-singular

x symmetric positive-definite

p(x) = c |x|(k+1)/2 exp{ tr(x)}

c = k

E[x] = ,

E[x ] = (

k+1 1
2 )

k(k1)/4 ||

=1

( 12 (2 + 1 ))

436
A.2

A. Summary of Basic Formulae

INFERENTIAL PROCESSES

The second section of this Appendix records the basic elements of the Bayesian
learning processes for many commonly used statistical models.
For each of these models, we provide, in separate sections of the table, the
following: the sufficient statistic and its sampling distribution; the conjugate family, the conjugate prior predictives for a single observable and for the sufficient
statistic, the conjugate posterior and the conjugate posterior predictive for a single
observable.
When clearly defined, we also provide, in a final section, the reference prior
and the corresponding reference posterior and posterior predictive for a single
observable. In the case of uniparameter models this can always be done. We recall,
however, from Section 5.2.4 that, in multiparameter problems, the reference prior
is only defined relative to an ordered parametrisation. In the univariate normal
model (Example 5.17), the reference prior for (, ) happens to be the same as that
for (, ), namely (, ) = (, ) 1 , and we provide the corresponding
reference posteriors for and , together with the reference predictive distribution
for a future observation.
In the multinomial, multivariate normal and linear regression models, however, there are very many different reference priors, corresponding to different
inference problems, and specified by different ordered parametrisations. These are
not reproduced in this Appendix.
Bernoulli model
z = {x1 , . . . , xn },
xi {0, 1}
p(xi | ) = Br(xi | ),
0<<1
n
t(z) = r = i=1 xi
p(r | ) = Bi(r | , n)
p() = Be( | , )
p(x) = Bb(r | , , 1)
p(r) = Bb(r | , , n)
p( | z) = Be( | + r, + n r)
p(x | z) = Bb(x | + r, + n r, 1)
() = Be( | 12 , 12 )
( | z) = Be( | 12 + r, 12 + n r)
(x | z) = Bb(x | 12 + r, 12 + n r, 1)

A.2 Inferential processes

Poisson Model
z = {x1 , . . . , xn },
xi = 0, 1, 2, . . .
p(xi | ) = Pn(xi | ),
0

t(z) = r = ni=1 xi
p(r | ) = Pn(r | n)
p() = Ga( | , )
p(x) = Pg(x | , , 1)
p(r) = Pg(x | , , n)
p( | z) = Ga( | + r, + n)
p(x | z) = Pg(x | + r, + n, 1)
() 1/2
( | z) = Ga( | r + 12 , n)
(x | z) = Pg(x | r + 12 , n, 1)

Negative-Binomial model
z = (x1 , . . . , xn ),
xi = 0, 1, 2, . . .
0<<1
p(xi | ) = Nb(xi | , r),
n
t(z) = s = i=1 xi
p(s | ) = Nb(s | , nr)
p() = Be( | , )
p(x) = Nbb(x | , , r)
p(s) = Nbb(s | , , nr)
p( | z) = Be( | + nr, + s)
p(x | z) = Nbb(x | + nr, + s, r)
() 1 (1 )1/2
( | z) = Be( | nr, s + 12 )
(x | z) = Nbb(x | nr, s + 12 , r)

437

438

A. Summary of Basic Formulae

Exponential Model
z = {x1 , . . . , xn },
0 < xi <
p(xi | ) = Ex(xi | ),
>0

t(z) = t = ni=1 xi
p(t | ) = Ga(t | n, )
p() = Ga( | , )
p(x) = Gg(x | , , 1)
p(t) = Gg(t | , , n)
p( | z) = Ga( | + n, + t)
p(x | z) = Gg(x | + n, + t, 1)
() 1
( | z) = Ga( | n, t)
(x | z) = Gg(x | n, t, 1)

Uniform Model
z = {x1 , . . . , xn },
0 < xi <
>0
p(xi | ) = Un(xi | 0, ),
t(z) = t = max{x1 , . . . , xn }
p(t | ) = Ip(t | n, 1 )
p() = Pa( | , )

p(x) = +1
Un(x | 0, ), if x ,

1
+n Ip(t | n, ),

1
+1 Pa(x | , ),
n
+n Pa(t | , ),

if x >

p(t) =
if t ,
if t >
p( | z) = Pa( | + n, n ), n = max{, t}
+n
1
p(x | z) = +n+1
Un(x | 0, n ), if x n , +n+1
Pa(x | , n ), if x > n
() 1
( | z) = Pa( | n, t)
n
(x | z) = n+1
Un(x | 0, t), if x t,

1
n+1 Pa(x | n, t),

if x > t

A.2 Inferential processes

Normal Model (known precision )
z = {x1 , . . . , xn },
< xi <
p(xi | , ) = N(xi | , ),
< <

t(z) = x
= n1 ni=1 xi
p(
x | , ) = N(x | , n)
p() = N(
 | 0 , 0 )

p(x) = N x | 0 , 0 (0 + )1
p(x) = N x | 0 , n 0 1
n = 0 + n,
n ,
,

),

=
1
p( | z) = N(
|

n
n
n
n (0 0 + nx)

1
p(x | z) = N x | n , n (n + )
() = constant
( | z) = N( | x
, n)
(x | z) = N(x | x
, n(n + 1)1 )

Normal Model (known mean )

z = {x1 , . . . , xn },
< xi <
>0
p(xi | , ) = N(xi | , ),

t(z) = t = ni=1 (xi )2
p(t | , ) = Ga(t | 12 n, 12 ),
p(t) = 2 (t | n)
p() = Ga( | , )
p(x) = St(x | , 1 , 2)
p(t) = Gg(x | , 2, 12 n)
p( | z) = Ga( | + 12 n, + 12 t)
p(x | z) = St(x | , ( + 12 n)( + 12 t)1 , 2 + n)
() 1
( | z) = Ga( | 12 n, 12 t)
(x | z) = St(x | , nt1 , n)

439

440

A. Summary of Basic Formulae

Normal Model (both parameters unknown)

z = {x1 , . . . , xn },
< xi <
p(xi | , ) = N(xi | , ),
< < , > 0
n

t(z) = (
x, s),
n
x = i=1 xi ,
ns2 = ni=1 (xi x
)2
p(
x | , ) = N(
x | , n)
2
p(ns | , ) = Ga(ns2 | 12 (n 1), 12 ),
p(ns2 ) = 2 (ns2 | n 1)
p(, ) = Ng(, | 0 , n0 , , ) = N( | 0 , n0 ) Ga( | , )
p() = St( | 0 , n0 1 , 2)
p() = Ga( | , )
p(x) = St(x | 0 , n0 (n0 + 1)1 1 , 2)
p(x) = St(x | 0 , n0 n(n0 + n)1 1 , 2)
p(ns2 ) = Gg(ns2 | , 2, 12 (n 1))
p( | z) = St( | n , (n + n0 )( + 12 n)n1 , 2 + n),
n = (n0 + n)1 (n0 0 + n
x),
1
1
2
n = + 2 ns + 2 (n0 + n)1 n0 n(0 x)2
p( | z) = Ga( | + 12 n, n )


p(x | z) = St x | n , (n + n0 )(n + n0 + 1)1 ( + 12 n)n1 , 2 + n
(, ) = (, ) 1 ,
n>1


2
( | z) = St | x
, (n 1)s , n 1


( | z) = Ga | 12 (n 1), 12 ns2


(x | z) = St x | x
, (n 1)(n + 1)1 s2 , n 1

441

A.2 Inferential processes

Multinomial Model
z = {r1 , . . . , rk , n},
ri = 0, 1, 2, . . . ,
p(z | ) = Muk (z | , n),
0 < i < 1,

k
r n
k =1 i

i=1 i 1

t(z) = (r, n), r = (r1 , . . . , rk )

p(r | ) = Muk (r | , n)
p() = Dik ( | ), = {1 , . . . , k+1 }
p(r) = Mdk (r | , n)



p( | z) = Dik | 1 + r1 , . . . , k + rk , k+1 + n k=1 r



p(x | z) = Mdk x | 1 + r1 , . . . , k + rk , k+1 + n k=1 r , n

Multivariate Normal Model

z = {x1 , . . . , xn },
x i k
p(xi | , ) = Nk (xi | , ),
k , k k positive-definite


)(xi x
)t
t(z) = (
x, S),
x = n1 ni=1 xi ,
S = ni=1 (xi x
p(
x | , ) = Nk (
x | , n)


p(S | ) = Wik S | 12 (n 1), 12
p(, ) = Nwk (, | 0 , n0 , , ) = Nk ( | 0 , n0 ) Wik ( | , )


p(x) = Stk x | 0 , (n0 + 1)1 n0 ( 12 (k 1)) 1 , 2 k + 1


p( | z) = Stk | n , (n + n0 )n 1
n , 2n ,
n = (n0 + n)1 (n0 0 + n
x),
n = + 12 S + 12 (n + n0 )1 nn0 (0 x)(0 x)t
p( | z) = Wik ( | + 12 n, n )
p(x | z) = Stk (x | n , (n0 + n + 1)1 (n0 + n)n 1
n , 2n ),
1
1
n = + 2 n 2 (k 1)

442

A. Summary of Basic Formulae

Linear Regression
z = (y, X), y = (y1 , . . . , yn )t n , xi = (xi1 , . . . , xik ) k , X = (xij )
p(y | X, , ) = Nn (y | X, I n ),
k , > 0
t(z) = (X t X, X t y)
p(, ) = Ng(, | 0 , n0 , , ) = Nk ( | 0 , n0 ) Ga( | , )
p() = Stk ( | 0 , n0 1 , 2)
p() = Ga( | , )


p(y | x) = St y | x 0 , f (x) 1 , 2
f (x) = 1 x(xt x + n0 )1 xt ,


p( | z) = Stk | n , (n0 + X t X)( + 12 n)n1 , 2 + n ,
n = (n0 + X t X)1 (n0 0 + X t y),
n = + 12 (y X n )t y + 12 ( 0 n )t n0 0
p( | z) = Ga( | + 12 n, n )
p(y | x, z) = St(y | x n , fn (x)( + 12 n)n1 , 2 + n),
fn (x) = 1 x(xt x + n0 + X t X)1 xt
(, ) = (, ) (k+1)/2 (for all reorderings of the j )


n , 1 X t X(n k)1 , n k ,
( | z) = Stk |
n
2
n = (X t X)1 X t y,

n )t y
n = 1 (y X
2

( | z) = Ga( | 12 (n k), n )
n , 1 fn (x)(n k)1 , n k),
(y | x, z) = St(y | x
n
2
fn (x) = 1 x(xt x + X t X)1 xt