Mathwords logoMathwords

Bayesian Inference — Definition, Formula & Example

Bayesian inference is a way of doing statistics where you start with a belief about an unknown quantity, collect data, and then update that belief. The result is a probability for each possible value of the unknown, adjusted by the evidence.

Bayesian inference treats an unknown parameter θ\theta as a random variable with a prior distribution P(θ)P(\theta). After observing data DD, it computes the posterior distribution P(θ∣D)P(\theta \mid D) by Bayes' theorem: the posterior is proportional to the likelihood P(D∣θ)P(D \mid \theta) times the prior. All estimates, such as posterior means and credible intervals, come from this posterior.

Key Formula

P(θ∣D)=P(D∣θ) P(θ)P(D),P(D)=∑θP(D∣θ) P(θ)P(\theta \mid D) = \frac{P(D \mid \theta)\,P(\theta)}{P(D)}, \qquad P(D) = \sum_{\theta} P(D \mid \theta)\,P(\theta)
Where:
  • θ\theta = The unknown parameter or hypothesis
  • DD = The observed data
  • P(θ)P(\theta) = Prior: your probability for θ\theta before seeing the data
  • P(D∣θ)P(D \mid \theta) = Likelihood: how probable the data are if θ\theta is true
  • P(θ∣D)P(\theta \mid D) = Posterior: your updated probability for θ\theta after seeing the data
  • P(D)P(D) = Evidence: total probability of the data, which makes the posteriors sum to 1. For a continuous parameter, the sum becomes an integral.

Worked Example

Problem: A factory machine is either in good condition, with a 10% defect rate, or worn, with a 40% defect rate. Maintenance records give a prior of P(good)=0.8P(\text{good}) = 0.8 and P(worn)=0.2P(\text{worn}) = 0.2. You inspect 2 parts, chosen independently, and both are defective. What is the posterior probability that the machine is worn?
Step 1: Find the likelihood of two defects under each hypothesis. Square each defect rate because the parts are independent.
P(D∣good)=0.12=0.01,P(D∣worn)=0.42=0.16P(D \mid \text{good}) = 0.1^2 = 0.01, \qquad P(D \mid \text{worn}) = 0.4^2 = 0.16
Step 2: Multiply each likelihood by its prior. Then add the products to get the evidence P(D)P(D).
0.01×0.8=0.008,0.16×0.2=0.032,P(D)=0.008+0.032=0.040.01 \times 0.8 = 0.008, \qquad 0.16 \times 0.2 = 0.032, \qquad P(D) = 0.008 + 0.032 = 0.04
Step 3: Divide each product by the evidence to get the posterior probabilities.
P(worn∣D)=0.0320.04=0.8,P(good∣D)=0.0080.04=0.2P(\text{worn} \mid D) = \frac{0.032}{0.04} = 0.8, \qquad P(\text{good} \mid D) = \frac{0.008}{0.04} = 0.2
Answer: The posterior probability that the machine is worn is 0.8. Two defects moved your belief from 20% to 80%. The prior and the posterior have swapped.

Why It Matters

Bayesian inference gives a direct probability that a hypothesis is true, which a p-value cannot do. It powers spam filters, A/B testing at tech companies, and clinical trials that update as patients enroll. It is also the core of courses in Bayesian data analysis and machine learning.

Common Mistakes

Mistake: Treating the likelihood P(D∣θ)P(D \mid \theta) as if it were the posterior P(θ∣D)P(\theta \mid D).
Correction: In the example, two defects have probability 0.16 if the machine is worn. The probability the machine is worn given two defects is 0.8. These are different quantities. To reverse the conditioning, you must weight by the prior and divide by P(D)P(D).

Related Terms