Introduction to Linear Algebra

Matrix
representation
of size n X d
Symmetric Matrix

Diagonal Matrix
Matrix multiplication with a vector
gives a transformed vector
Inner Product

Norm of a
vector

Cosine of the angle

between 2 vectors

Cauchy Schwartz
Inequality
Linear Independence of vectors / Basis
vectors
Outer Product of 2 vectors
 Rank of a matrix
Number of independent rows / columns of a
matrix

Outer product between 2 vectors gives a

matrix of rank 1.
Gradient of a scalar function
Gradient of a vector-valued
function (Jacobian)
Matrix / Vector Derivatives

Gradient of
Vector function

Gradient of scalar
function
 Eigen values and vectors

Diagonalization property

Matrix, whose columns correspond to

V Eigenvectors of M
1
M VDV Diagonal matrix, corresponding to
D eigenvalues of M
 Rank deficient square matrices
A matrix A of rank r will have r independent
rows / columns, and is said to be rank-
deficient.

There will be r independent eigen

vectorscorresponding to r non- zero eigen
values

r eigen values are zero.

Matrix Inverse

For square matrix M

 Orthogonal Matrix

{q1 , q2 ,.....qM } M vectors

1 i j
qi q j
T

0 i j
Q QI
T
columns of Q denote
q1 q2 ..... qM
Q is of size M XM and is said to be
an orthogonal matrix
 Positive Definite Matrix

Positive definite square xT Ax 0

matrix A

x denotes a vector of size d x 1 .

A denotes a vector of size d x d .

Eigen values of positive definite matrix A are

non-negative.
 Real symmetric matrix
Elements of a matrix are real numbers
Matrix is symmetric
Eigen values of a real symmetric matrix are
real.
Probability Theory
 Random variables
A random variable x denotes a quantity that is
uncertain
May be result of experiment (flipping a coin) or a real
world measurements (measuring temperature)
If observe several instances of x we get different
values
Some values occur more than others and this
information is captured by a probability distribution

19
Discrete Random Variables

20
Continuous Random Variable

23
Expectation/ Variance of continuous
random variable
 Joint Probability
Consider two random variables x and y
If we observe multiple paired instances, then some
combinations of outcomes are more likely than
others
This is captured in the joint probability distribution
Written as P(x,y)
Can read P(x,y) as probability of x and y

25
For 2 random variables x and y,

Marginalization property

26
 Marginalization
We can recover probability distribution of any variable in a joint distribution
by integrating (or summing) over the other variables

27
 Marginalization
We can recover probability distribution of any variable in a joint distribution
by integrating (or summing) over the other variables

28
 Marginalization
We can recover probability distribution of any variable in a joint distribution
by integrating (or summing) over the other variables

Works in higher dimensions as well leaves joint distribution between

whatever variables are left

29
 Conditional Probability
Conditional probability of x given that y=y1 is relative
propensity of variable x to take different outcomes given that
y is fixed to be equal to y1.
Written as Pr(x|y=y1)

30
 Conditional Probability
Conditional probability can be extracted from joint probability
Extract appropriate slice and normalize

31
 Conditional Probability

More usually written in compact form

Can be re-arranged to give

32
 Conditional Probability

This idea can be extended to more than two

variables

33
 Bayes Rule
From before:

Combining:

Re-arranging:

34
 Bayes Rule Terminology
Likelihood propensity for Prior what we know
observing a certain value of about y before seeing x
x given a certain value of y

Posterior what we Evidence a constant to

know about y after ensure that the left hand
seeing x side is a valid distribution
35
 Independence
If two variables x and y are independent then variable x tells
us nothing about variable y (and vice-versa)

36
 Independence
When variables are independent, the joint factorizes into a
product of the marginals:

37
Discrete Random vectors
Continuous Random vectors
Continuous random vectors
Discrete random vectors
Properties of covariance matrix
1 dimensional Gaussian
2 dimensional Gaussian
Multi dimensional Gaussian