A basketball player makes 3 of her first 4 free throws. What is her real hit rate p? Classical statistics answers with one number, 3/4. Bayesian statistics answers with a whole distribution for p, built from three ingredients: the likelihood (how well each possible p explains the data), the prior (what you believed about p before the data) and Bayes' rule, which multiplies the two into the posterior (what you believe after). This chapter builds each ingredient on its own, with the binomial for the data and the beta distribution for the belief, because that pair is where almost every course and exercise starts.
By the end you can
- Write the likelihood of a set of observations and say why it is not a probability distribution for p.
- Find the value of p that makes the data most likely, y/n, from the log-likelihood.
- Read a Beta(a, b) prior as a belief: its centre, its strength a + b, and choose a and b from a sentence.
- Turn a beta prior and binomial data into the posterior Beta(a + y, b + n − y), and explain why.
- Answer questions with the posterior: its mean, P(p > ½), an interval, the chance of the next success.
Each free throw is a trial with two outcomes, hit or miss, and we assume the same hit chance p every time and no influence between throws (independent trials). Then the number of hits Y in n throws has the binomial distribution, met in the Common Distributions chapter.
Before the throws, p is fixed and Y is unknown: the formula tells you how likely each count is. After the throws it is the other way round. The data are known, y = 3 of n = 4, and p is what we do not know. Keep the data fixed and let p vary: the same expression now tells you, for each candidate p, how probable the data you actually saw would have been. That function of p is the likelihood, L(p).
| probability: p = ½ fixed, y varies | y = 0 | y = 1 | y = 2 | y = 3 | y = 4 | total |
|---|---|---|---|---|---|---|
| P(Y = y | ½) | 1/16 | 4/16 | 6/16 | 4/16 | 1/16 | 1 |
| likelihood: y = 3 fixed, p varies | p = 0 | p = ¼ | p = ½ | p = ¾ | p = 1 | area |
|---|---|---|---|---|---|---|
| L(p) = 4p³(1 − p) | 0 | 0.047 | 0.250 | 0.422 | 0 | 1/5 |
The first row is a probability distribution: its values add up to 1. The second is not. Its values are probabilities of the same event (3 hits) under different hypotheses, and nothing makes them add up to anything: the area under L(p) from 0 to 1 is ∫ 4p³(1 − p) dp = 4(1/4 − 1/5) = 1/5. So the likelihood compares values of p with one another (p = ¾ explains 3 hits in 4 about 1.7 times better than p = ½, since 0.422/0.25 ≈ 1.7), but it is not yet a belief about p. Turning it into one is the job of the prior and Bayes' rule.
Building a likelihood, step by step
The binomial is one case of a general recipe that works for any model:
| step | what you do | for the throws hit, hit, miss, hit |
|---|---|---|
| 1 | Write the probability of one observation, as a function of the unknown. | hit: p; miss: 1 − p |
| 2 | Independent observations: multiply. | p · p · (1 − p) · p |
| 3 | Simplify, and read the result as a function of the unknown. | L(p) = p³(1 − p) |
| 4 | Drop every factor that does not contain the unknown. | nothing to drop here |
This likelihood, for the throws in that order, is p³(1 − p). The binomial one, for "3 hits in some order", is 4p³(1 − p). They differ only by the constant 4, the number of orders, so they rank every value of p the same way and lead to the same conclusions. Whether you record the order or only the count, the information about p is the same.
The best-supported value: y/n
The value of p where L is highest is called the maximum-likelihood estimate, p̂. To find it, take the logarithm first: ln turns the product into a sum, which is easier to differentiate, and since ln only ever increases, L and ln L peak at the same place.
So the value the data alone support best is the observed proportion, which is what common sense says. What the likelihood adds is the spread: with 3 of 4 the curve is wide and many values are almost as good; with 30 of 40 it has the same peak but is far narrower. The figure below shows both.
| model of one observation | likelihood of n observations | maximum at |
|---|---|---|
| success or failure, chance p | pʸ(1 − p)ⁿ⁻ʸ | y / n |
| a count with Poisson rate λ | λ^(x₁ + … + xₙ) e^(−nλ) | the mean count x̄ |
| a measurement, normal with mean μ and known σ | exp(−Σ(xᵢ − μ)² / 2σ²) | the mean x̄ |
Before seeing the player, you are not completely ignorant: most players make between 50 % and 90 % of free throws. A prior is a probability distribution for p that says how plausible each value is before the data. Since p lives in [0, 1], the natural family is the beta distribution of the Common Distributions chapter.
| prior | mean | standard deviation | what it says |
|---|---|---|---|
| Beta(1, 1) | 0.5 | 0.289 | flat: every value of p equally plausible, "no idea" |
| Beta(2, 2) | 0.5 | 0.224 | a mild hunch that p is near ½, easily overruled |
| Beta(10, 10) | 0.5 | 0.109 | fairly sure p is close to ½, worth 20 trials of evidence |
| Beta(8, 2) | 0.8 | 0.121 | p is probably high, around 0.8 |
| Beta(½, ½) | 0.5 | 0.354 | U-shaped: p is probably near 0 or near 1, rarely in the middle |
Choosing a and b from a sentence
A belief usually comes as words: "I think she makes about 70 %, and I am as sure as if I had watched 10 throws." The centre gives the mean, the "as sure as" gives the strength. If instead you can say how uncertain you are, as a standard deviation s, use the variance of the beta, v = s², to find the strength.
Bayes' rule for events said: P(A | B) is P(A) times P(B | A), divided by P(B). With a whole range of values of p it says the same thing value by value: the new plausibility of each p is its old plausibility, times how well it explains the data, divided by one total that makes everything add up to 1.
When the prior and the posterior belong to the same family, the prior is called conjugate to the likelihood; the beta is conjugate to the binomial. For the player, a mild prior Beta(2, 2) and 3 hits in 4 give the posterior Beta(5, 3). Its mean is 5/8 = 0.625, its highest point (the mode, (a′ − 1)/(a′ + b′ − 2)) is 4/6 ≈ 0.667, and its standard deviation is √(5 · 3 / (8² · 9)) ≈ 0.161.
Amber: what you believed before the data (the prior). Dashed blue: how well each p explains y successes in n trials (the likelihood, rescaled to area 1). Green: the posterior, prior times likelihood, rescaled. The triangles under the axis mark the three centres: the posterior mean always sits between the prior mean and y/n, at the share w of the way from y/n to the prior mean. Try a firm prior with little data, then push n up with the same proportion of successes: the prior stops mattering and the green curve narrows onto y/n.
The posterior mean is a compromise
As the data grow, w = (a + b)/(a + b + n) shrinks toward 0, and the posterior mean moves toward y/n. Two people who start from different priors end up agreeing once there is enough data. With the same 75 % hit rate:
| data | posterior (prior Beta(2, 2)) | mean | standard deviation | w |
|---|---|---|---|---|
| 3 / 4 | Beta(5, 3) | 0.625 | 0.161 | 0.50 |
| 30 / 40 | Beta(32, 12) | 0.727 | 0.066 | 0.09 |
| 300 / 400 | Beta(302, 102) | 0.748 | 0.022 | 0.01 |
Updating in batches
Today's posterior is tomorrow's prior. After 3 hits in 4 the belief is Beta(5, 3). If she then makes 2 of 3 more, add again: Beta(5 + 2, 3 + 1) = Beta(7, 4). Using all the data at once, 5 hits in 7 from the original Beta(2, 2), gives Beta(2 + 5, 2 + 2) = Beta(7, 4), the same. The order in which evidence arrives does not matter, only the totals.
The posterior is the full answer; questions about p are read off it. The most common ones:
| question | how | Beta(5, 3) |
|---|---|---|
| a best single guess | the posterior mean a′/(a′ + b′) | 0.625 |
| the most plausible value | the mode (a′ − 1)/(a′ + b′ − 2) | 0.667 |
| is she better than 50 %? | P(p > ½): the area of the posterior to the right of ½ | 99/128 ≈ 0.773 |
| a range of plausible values | an interval holding 95 % of the area (a credible interval) | see below |
| will the next throw go in? | the average of p over the posterior, which is its mean | 5/8 |
For whole-number a and b there is a shortcut with no integral. A Beta(a, b) variable has the same distribution as the a-th smallest of a + b − 1 random points dropped uniformly on [0, 1]. That point lies above x exactly when at most a − 1 of the points fall below x, and the number of points below x is binomial. So P(p > x) = P(Binomial(a + b − 1, x) ≤ a − 1). For Beta(5, 3) and x = ½: P(Bin(7, ½) ≤ 4) = (1 + 7 + 21 + 35 + 35)/128 = 99/128, the same answer.
An exact 95 % interval needs the beta's quantiles (a table or software). With more data the posterior is close to a normal curve, and mean ± 1.96 standard deviations is a good approximation: for 30 of 40, Beta(32, 12) gives 0.727 ± 1.96 · 0.066, roughly [0.60, 0.86]. Unlike a classical confidence interval, this one means what people usually want it to mean: given the data and the prior, p is in it with probability 95 %.
A round of 10 questions
Answer as many as you can. Mistakes come back at the end for a review round.
Every exercise is made on the spot, so a new round is never the same. The computer picks the answer first and builds the question around it, working with exact fractions, so the answer key is always right. Answers are read exactly: 0.75, 3/4 and 6/8 count as the same number (where lowest terms are asked for, 6/8 just gets a reminder). A wrong answer shows the worked solution line by line, in the order the lesson does it; the new number in each line is coloured.
| Wrong | Right | Why |
|---|---|---|
| the likelihood is the probability of p | it is the probability of the data, for each p | its area is not 1 (1/5 for 3 of 4); only prior × likelihood, rescaled, is a distribution for p |
| Beta(a, b) → Beta(a + y, b + n) | Beta(a + y, b + n − y) | b collects the failures, n − y, not all the trials |
| the prior mean is (a − 1)/(a + b − 2) | a / (a + b) | that is the mode, the highest point; they differ unless a = b |
| keeping C(n, y) and B(a, b) through the update | drop them, then name the beta | the constant of the posterior is fixed by the area being 1 |
| a strong prior can never be overruled | its weight is (a + b)/(a + b + n) | with enough data the weight goes to 0 |
| P(next success) = y/n | the posterior mean (a + y)/(a + b + n) | y/n ignores the prior; with 4 of 4 it would claim certainty |
Key ideas
- The likelihood is the data's probability read as a function of the unknown: data fixed, p varies. It is not a distribution for p.
- Build it by multiplying the probabilities of independent observations and dropping factors without p: pʸ(1 − p)ⁿ⁻ʸ.
- Its maximum is at the observed proportion y/n; its width shows how much the data pin p down.
- A Beta(a, b) prior is centred at a/(a + b) and is as firm as a + b trials; choose a and b from a mean and a strength.
- Posterior ∝ prior × likelihood. Beta(a, b) with y of n gives Beta(a + y, b + n − y).
- The posterior mean is a weighted average of the prior mean and y/n, with prior weight (a + b)/(a + b + n).
- Batches can be added one at a time or all at once; the result is the same.
- P(p > x) for whole a, b is P(Binomial(a + b − 1, x) ≤ a − 1); the next trial succeeds with the posterior mean.