Development of Bayes Theorem Terminology: P(A): Probability of occurrence of event A (marginal) P(B): Probability of occurrence of event B (marginal) P(A,B):

Probability of simultaneous occurrence of events A and B (joint) P(A|B): Probability of occurrence of A given that B has occurred (conditional) P(B|A): Probability of occurrence of B given that A has occurred (conditional) Relationship of joint probability to conditional and marginal probabilities: P(A,B) = P(A B)P(B) or P(A, B) = P(B A)P(A) So . . . P(A|B)P(B) = P(B|A)P(A)

Rearranging gives simplest statement of Bayes theorem:

( | )

PA PABPB P B A Often, B represents an underlying model or hypothesis and A represents observable consequences or data, so Bayes theorem can be written schematically as Pmodel dataPdatamodelPmodel This lets us turn a statement about the forward problem: P(data|model): probability of obtaining observed data given certain model into statements about the corresponding inverse problem: P(model|data): probability that certain model gave rise to observed data as long as we are willing to make some guesses about the

( | ) ( ) ( )

probability of occurrence of that model, P(model), prior to taking the data into account.

Or graphically, Bayes theorem lets us turn information about the probability of different effects from each possible cause: into information about the probable cause given the observed effects:
(Illustration styled after Sivia, 1996, Figure 1.1)

Assume that Bi represents one of n possible mutually exclusive events and that the conditional probability for the occurrence of A given that Bi has occurred is P(A|Bi). In this case, the total probability for the occurrence of A is
n i ii 1

PAPABPB

() and the conditional probability that event Bi has occurred given that event A has been observed to occur is given by

PA P AB P B PABPB PABPB

P B A ii
n j jj ii

i 1

. That is, if we assume that event A arises with probability P(A|Bi), from each of the underlying states Bi, i=1,,n, we can use our observation of the occurrence of A to update our a priori assessment of the probability of occurrence of each state, P(Bi), to an improved a posteriori estimate, P(Bi|A).

-wikipediaINTRO In probability theory and statistics, Bayes' theorem which is alternatively Bayes' law is a theorem with two distinct interpretations. In the Bayesian interpretation, it expresses how a subjective degree of belief should rationally change to account for evidence and in the frequentist interpretation, it relates inverse representations of the probabilities concerning two events.

http://www.answers.com/topic/bayes-theorem

Bayes' theorem deals with the role of new information in revising probability estimates. The theorem assumes that the probability of a hypothesis known as the posterior probability is a function of new evidence, the likelihood and previous knowledge is a prior probability. The theorem is named after Thomas Bayes (17021761), a nonconformist minister who Bayes' theorem is a logical consequence of the product rule of probability, which is the probability (P) of two events (A and B) happening P(A,B) is equal to the conditional probability of one event occurring given that the other has already occurred P(A|B) multiplied by the probability of the other event happening P(B). The derivation of the theorem is as follows: P(A,B) = P(A|B)P(B) = P(B|A)P(A) Thus: P(A|B) = P(B|A)P(A)/P(B). Bayes' theorem has been frequently used in the areas of diagnostic testing and in the determination of genetic predisposition. Bayes' theorem can be used as a normative model to assess how well people use empirical information to update the probability that a hypothesis is true.

http://www.investopedia.com/terms/b/bayes-theorem.asp#axzz2GYJYVEHl

Definition of 'Bayes' Theorem'

A formula for determining conditional probability named after 18th-century British mathematician Thomas Bayes. The theorem provides a way to revise existing predictions or theories given new or additional evidence. In finance, Bayes Theorem can be used to rate the risk of lending money to potential borrowers. The formula is as follows:

Also called "Bayes Rule."

Bayes's theorem gives us the relationship between what we know and what we want to know

History
Bayes' theorem was named after the Reverend Thomas Bayes (170161), who studied how to compute a distribution for the probability parameter of a binomial distribution (in modern terminology). His friend Richard Price edited and presented this work in 1763, after Bayes' death, as An Essay towards solving a Problem in the Doctrine of Chances. The French mathematician Pierre-Simon Laplace reproduced and extended Bayes' results in 1774, apparently quite unaware of Bayes' work.[3] Stephen Stigler suggested in 1983 that Bayes' theorem was discovered by Nicholas Saunderson sometime before Bayes. However, this interpretation has been disputed.

Stephen Fienberg describes the evolution from "inverse probability" at the time of Bayes and Laplace (a term still used by Harold Jeffreys in 1939), to "Bayesian" in the 1950s and notes that Ronald A. Fisher was the first to use the term "Bayesian" (albeit in a derogatory sense).

Some advantages to using Bayesian analysis include the following:

It provides a natural and principled way of combining prior information with data, within a solid decision theoretical framework. You can incorporate past information about a parameter and form a prior distribution for future analysis. When new observations become available, the previous posterior distribution can be used as a prior. All inferences logically follow from Bayes theorem. It provides inferences that are conditional on the data and are exact, without reliance on asymptotic approximation. Small sample inference proceeds in the same manner as if one had a large sample. Bayesian analysis also can estimate any functions of parameters directly, without using the "plug-in" method (a way to estimate functionals by plugging the estimated parameters in the functionals). It obeys the likelihood principle. If two distinct sampling designs yield proportional likelihood functions for , then all inferences about should be identical from these two designs. Classical inference does not in general obey the likelihood principle. It provides interpretable answers, such as the true parameter has a probability of 0.95 of falling in a 95% credible interval. It provides a convenient setting for a wide range of models, such as hierarchical models and missing data problems. MCMC, along with other numerical methods, makes computations tractable for virtually all parametric models.

There are also disadvantages to using Bayesian analysis:

It does not tell you how to select a prior. There is no correct way to choose a prior. Bayesian inferences require skills to translate subjective prior beliefs into a mathematically formulated prior. If you do not proceed with caution, you can generate misleading results. It can produce posterior distributions that are heavily influenced by the priors. From a practical point of view, it might sometimes be difficult to convince subject matter experts who do not agree with the validity of the chosen prior. It often comes with a high computational cost, especially in models with a large number of parameters. In addition, simulations provide slightly different answers unless the same random seed is used. Note that slight variations in simulation results do not contradict the early claim that Bayesian inferences are exact. The posterior distribution of a parameter is exact, given the likelihood function and the priors, while simulation-based estimates of posterior quantities can vary due to the random number generator used in the procedures.

-Example-

A rare genetic disease is discovered. Although only one in a million people carry it, you consider getting screened. You are told that the genetic test is extremely good; it is 100% sensitive (it is always correct if you have the disease) and 99.99% specific (it gives a false positive result only 0.01% of the time). Having recently learned Bayes' theorem, you decide not to take the test. Why? (From Durbin et.al. "Biological Sequence Analysis", Cambridge University Press, 1998)

solution:

Bayes' Theorem states that for events X and Y: P(X|Y)=P(Y|X)*P(X)/P(Y). We want to know the probability of being healthy(X) given the positive test(PT) results(Y). According to the Bayes' Theorem, P(healthy|PT)=P(PT|healthy)*P(healthy)/P(PT).

From the problem we know that P(healthy)=1-0.000001=0.999999 and getting a false positive P(PT|healthy)=0.0001. The only unknown in the formula above is the probability of having a positive test P(PT). It can be calculated using the definition of marginal probability P(Y)=P(Y|Z1)*P(Z1)+...+P(Y|Zn)*P(Zn), where Zi, i=1...n are all possible events. In our case there are only two possible events: "being healthy" and "being sick". Therefore P(PT)=P(PT|healthy)*P(healthy)+P(PT|sick)*P(sick).

From the problem we know that P(PT|sick)=1.0

(test is always correct in presence of the disease) and P(sick)=0.000001.

Substituting the numbers into the formula we get P(PT)=0.0001*0.999999+1.0*0.000001=0.000101. Finally, P(healthy|PT)=0.0001*0.999999/0.000101=0.990098,

that is very close to 1. So, the probability of still being healthy given that the results of the test turned positive is above 99%. That is a good reason for not taking the test.