Statistics and Probability: The Complete Guide
With Formulas, Rules, and Worked Examples

Brief History of Probability and Statistics

The story of probability and statistics spans five centuries and crosses mathematics, economics, medicine, and physics. Understanding how the field developed helps place every formula in context.

The systematic study of probability began with Gerolamo Cardano in the 1500s, who analyzed games of dice and first described what we now call the sample space. In 1654, a correspondence between Blaise Pascal and Pierre de Fermat over a gambling dispute laid the rigorous groundwork for combinatorics and expected value. Shortly after, Christiaan Huygens published the first probability textbook in 1657.

The 18th century brought two landmarks. Jakob Bernoulli proved the Law of Large Numbers, establishing that empirical frequencies converge to theoretical probabilities with enough trials. Thomas Bayes, in a paper published posthumously in 1763, introduced the theorem now bearing his name — the mathematical rule for updating belief on new evidence. Pierre-Simon Laplace then formalized much of probability theory in his 1812 treatise.

The 19th century gave birth to modern statistics. Francis Galton developed the concepts of regression toward the mean and correlation. Karl Pearson introduced the chi-square test and the Pearson correlation coefficient. Ronald Fisher in the 1920s and 1930s unified experimental design, analysis of variance (ANOVA), and maximum likelihood estimation. In 1933, Andrei Kolmogorov placed all of probability on an axiomatic foundation — the mathematical bedrock used to this day.

Branches of Statistics: Descriptive vs. Inferential

Before diving into probability, it helps to understand the two major branches of statistics that probability supports.

Descriptive statistics summarize and describe the data you actually have using measures of central tendency (mean, median, mode) and spread (variance, standard deviation, IQR). Inferential statistics use sample data and probability models to make predictions and test claims about a larger population.

A third branch — Bayesian statistics — treats probability as a degree of belief and updates it as new data arrives. It sits between the two and is foundational to modern machine learning.

Levels of Measurement in Statistics

Before you can choose the right statistical method, you need to know what kind of data you have. Stanley Stevens (1946) identified four levels of measurement, each with different properties and permitted operations.

The level of measurement determines which statistical tests are valid. Applying a mean to nominal data (e.g., averaging blood types coded 1–4) produces a meaningless number — a common mistake in practice.

Key Measures in Descriptive Statistics

These are the core measures every analyst uses daily.

Mean, Median, and Mode

These three measures describe the center of a dataset.

Example: Exam scores: 65, 72, 75, 78, 80, 82, 85, 88, 90, 95 → Sum = 810 → Mean = 810/10 = 81

The median is the middle value when sorted. For 10 values: Median = (80 + 82) / 2 = 81. Add an outlier score of 200 and the mean jumps to 90.5 — but the median stays at 82. That robustness makes the median the right choice for skewed data.

The mode is the most frequent value. Shoe sizes: 7, 8, 8, 8, 9, 9, 10 → Mode = 8. Mode is the only measure that works with categorical (nominal) data.

Variance and Standard Deviation

These measure how spread out data points are from the mean.

Z-Score

Z-scores standardize data to a common scale. A z-score of 1.45 means the score of 95 is 1.45 standard deviations above average. Use the Z-table to convert any z-score to a probability.

Sampling Methods

Because studying an entire population is usually impractical, statistical inference depends on collecting a representative sample. The method of sampling determines how reliably results generalize.

Sample size matters as much as sampling method. Larger samples reduce sampling error and narrow confidence intervals. Use a sample size calculator to determine the minimum n needed for your study's desired margin of error.

Introduction to Probability — Definition and Types

Probability is the numerical measure of the likelihood that an event will occur. It is always a value between 0 (impossible) and 1 (certain), calculated as the ratio of favorable outcomes to total possible outcomes.

Venn Diagram Symbols and Notation

Probability problems frequently use set notation from Venn diagrams. This table covers every symbol you will encounter — see our Venn Diagrams guide for full visual examples and our Venn Diagram Maker to build your own.

Type 1: Theoretical Probability

Based on mathematical reasoning assuming all outcomes are equally likely. Example: The probability of drawing an Ace from a standard deck = 4/52 = 1/13 ≈ 0.077. No experiment needed — it follows from the structure of the deck.

Type 2: Experimental (Empirical) Probability

Based on actual observed frequencies from repeated trials.

Type 3: Subjective Probability

Based on personal judgment, expert opinion, or experience. Examples: A weather forecaster says "70% chance of rain tomorrow." A surgeon estimates a "90% chance of successful recovery." These are not calculated — they are expert estimates informed by experience.

Type 4: Axiomatic Probability (Kolmogorov's Axioms)

The mathematical foundation of all probability theory. Andrei Kolmogorov (1933) defined three axioms:

• Axiom 1: P(A) ≥ 0 for any event A (probability is non-negative)
• Axiom 2: P(S) = 1 (the probability of the entire sample space is 1)
• Axiom 3: For mutually exclusive events A and B: P(A∪B) = P(A) + P(B)

All other probability rules — the addition rule, multiplication rule, Bayes' theorem — are derived from these three axioms.

Types of Events in Probability

Different event types require different calculation approaches. Misidentifying whether events are independent or dependent is one of the most common errors in applied probability.

Simple Events

A simple event has exactly one outcome. Rolling a 4 on a die is a simple event. P(rolling 4) = 1/6. There is only one way to get a 4, and it has a single, well-defined probability.

Compound Events

A compound event combines two or more simple events. Rolling an even number (2, 4, or 6) is compound. P(even) = 3/6 = 0.5. The probability of a compound event is calculated using the addition or multiplication rules depending on the relationship between the events.

Independent Events

Two events are independent when the outcome of one has no effect on the probability of the other. Each flip of a fair coin is independent: getting heads on flip 1 does not change the probability of heads on flip 2.
Test: P(A∩B) = P(A) × P(B) — if this equation holds, the events are independent.
Example: P(heads on flip 1 AND heads on flip 2) = 0.5 × 0.5 = 0.25

Dependent Events

Two events are dependent when the outcome of one changes the probability of the other. Drawing cards without replacement creates dependency.
Example: A bag has 3 blue and 2 red marbles. P(blue on draw 1) = 3/5 = 0.60. If a blue marble is removed and not replaced, P(blue on draw 2) = 2/4 = 0.50 — the probability changed because of the first outcome.
Use the formula: P(A∩B) = P(A) × P(B|A)

Mutually Exclusive Events

Two events are mutually exclusive (also called disjoint) when they cannot both occur at the same time. Rolling a 3 and rolling a 5 on a single die roll cannot both happen.
Property: P(A∩B) = 0, so P(A∪B) = P(A) + P(B)
Important distinction: Mutually exclusive events are always dependent — knowing A occurred immediately tells you B did not. Independent events are never mutually exclusive (unless one has probability 0).

Complementary Events

The complement of event A, written A' or Aᶜ, contains all outcomes NOT in A. Together A and A' cover the entire sample space.
Property: P(A) + P(A') = 1, so P(A') = 1 − P(A)
Example: P(rolling a 6) = 1/6, so P(not rolling a 6) = 1 − 1/6 = 5/6.
The complement rule is especially useful when it is easier to calculate P(A') than P(A) directly — for instance, P(at least one 6 in two rolls) = 1 − P(no 6 in two rolls) = 1 − (5/6)² ≈ 0.306.

Probability Rules and Formulas

These rules are the core toolkit for solving almost every probability problem. The table below is a printable cheat sheet — bookmark it as your probability formulas quick reference.

See also: Probability Rules deep-dive | Conditional Probability | Mutually Exclusive Events

Conditional Probability

The probability of A occurring given that B has already occurred:

Worked Example: In a class of 30 students: 18 study Math (M), 12 study Science (S), 6 study both. What is P(M|S)?
P(M∩S) = 6/30 = 0.2  |  P(S) = 12/30 = 0.4  →  P(M|S) = 0.2 / 0.4 = 0.5
Half of Science students also study Math.

Bayes' Theorem

Bayes' Theorem lets you reverse a conditional probability — updating the probability of a hypothesis given new evidence. It underlies Bayesian statistics, spam filters, medical diagnosis, and recommendation engines.

Example 1 — Medical Diagnosis: A disease affects 1% of the population. A test has 95% sensitivity (true positive rate) and 5% false positive rate. If you test positive, what is the probability you have the disease?

• P(Disease) = 0.01 (prior)
• P(Positive | Disease) = 0.95
• P(Positive | No Disease) = 0.05
• P(Positive) = (0.95 × 0.01) + (0.05 × 0.99) = 0.0095 + 0.0495 = 0.059
• P(Disease | Positive) = 0.0095 / 0.059 ≈ 16.1%

Example 2 — Spam Filter: 30% of emails are spam. A spam filter correctly flags spam 98% of the time and incorrectly flags legitimate email 2% of the time. If an email is flagged, what is the probability it is actually spam?

• P(Spam) = 0.30 (prior)
• P(Flagged | Spam) = 0.98
• P(Flagged | Not Spam) = 0.02
• P(Flagged) = (0.98 × 0.30) + (0.02 × 0.70) = 0.294 + 0.014 = 0.308
• P(Spam | Flagged) = 0.294 / 0.308 ≈ 95.5%

With a common prior of 30% spam, the filter's high accuracy translates to 95.5% confidence when it flags — very different from the medical test case because the base rate is much higher. See: More Bayes' Theorem worked examples | Prior Probability | Posterior Probability

Probability Tree Diagrams

A probability tree diagram maps out all possible outcomes of a multi-step random experiment as branches, with each branch labeled with its probability. They make sequential probability calculations visual and systematic. See: Probability Trees guide | Probability Tree Diagram tool

Each branch probability is multiplied along the path to get the final outcome probability. All four outcomes sum to 1.00 ✓. To find P(at least one Head) = 1 − P(TT) = 1 − 0.25 = 0.75.

Random Variables and Probability Distributions

A random variable is a numerical value assigned to the outcome of a random experiment. It connects probability theory to measurable data. See: Probability Density Function | Cumulative Distribution Function

Expected Value — Formula and Examples

The expected value E(X) is the long-run average outcome of a random variable — the value you would expect to get if you repeated the experiment many times. It is the most searched formula on this site. See the full guide: Expected Value | Mean vs Expected Value

Example 1 — Fair Die:

Example 2 — Insurance (EV = p · b − (1−p) · c): A 40-year-old buys a $500,000 life insurance policy for $300/year. The insurer estimates P(death this year) = 0.001. Expected payout to the insurer: E(X) = 0.001 × (−499,700) + 0.999 × 300 = −499.70 + 299.70 = −$200. The insurer expects to profit $200 per policy per year on average. This is expected value at the core of actuarial science. See: Expected Value Calculator

Binomial Distribution

Models the number of successes in n independent trials, each with probability p of success. Conditions: fixed n, binary outcome (success/failure), constant p, independent trials. See: Binomial Distribution guide | Binomial Calculator

Example: P(exactly 3 heads in 5 fair coin flips): P(X=3) = C(5,3) × (0.5)³ × (0.5)² = 10 × 0.125 × 0.25 = 0.3125

Poisson Distribution

Models events occurring in a fixed time or space interval at constant average rate λ. See: Poisson Distribution | Poisson Calculator

Example: λ = 4 calls/minute. P(exactly 2 calls in one minute) = (4² × e⁻⁴) / 2! = (16 × 0.0183) / 2 = 0.1465

Normal (Gaussian) Distribution

The most important distribution in statistics. Symmetric, bell-shaped, defined by mean (μ) and standard deviation (σ). See: Normal Distribution guide | Normal Distribution Curve visualizer | Z-Table

The Empirical Rule (68-95-99.7 Rule):

See: Empirical Rule | Normal Distribution Calculator | Empirical Rule Calculator

Other Key Distributions (Quick Reference)

Inferential Statistics — Hypothesis Testing and Confidence Intervals

Inferential statistics uses probability distributions to make decisions about populations from sample data. This is where probability and statistics merge most powerfully. See: Hypothesis Testing | Inferential Statistics

What is Hypothesis Testing?

Hypothesis testing is a formal 5-step process to determine whether sample data provides enough evidence to reject a claim about a population.

1. State hypotheses: H₀ (null) and H₁ (alternative)
2. Choose significance level: Usually α = 0.05
3. Select and compute test statistic: t, z, F, χ², etc.
4. Calculate the p-value
5. Decision: If p < α → reject H₀; If p ≥ α → fail to reject H₀

P-Value Explained

The p-value is the probability of observing results as extreme as your data, assuming the null hypothesis is true. A small p-value indicates the observed result is unlikely under H₀.

Confidence Intervals

A confidence interval is a range of plausible values for a population parameter, calculated from sample data at a specified confidence level.

Common Statistical Tests — Quick Reference

Not sure which test to use? Try the Statistical Test Selector.

The Central Limit Theorem

The Central Limit Theorem (CLT) is arguably the most important theorem in statistics: as sample size increases, the sampling distribution of the sample mean approaches a normal distribution — regardless of the original population's shape.

In practice, n ≥ 30 is typically sufficient. This is why t-tests, z-tests, and confidence intervals work even when your data isn't perfectly normal — the sample mean will be approximately normally distributed anyway. Verify normality assumptions with the Shapiro-Wilk test.

Probability Laws and Theorems

Law of Large Numbers

The Law of Large Numbers (LLN) is one of the most fundamental theorems in probability. It states that as the number of independent trials of a random experiment increases, the sample average converges to the expected value (population mean). See the full guide: Law of Large Numbers

There are two versions: the Weak LLN (convergence in probability, proven above) and the Strong LLN (almost sure convergence, proven by Kolmogorov). In practice, the distinction rarely matters — both guarantee that larger samples produce better estimates.

Real-world examples:

• Casinos: A single roulette spin is unpredictable. Over 100,000 spins, the house edge of 5.26% on American roulette will manifest almost exactly. The LLN guarantees profitability at scale — the casino doesn't gamble; it runs a statistical operation.
• Insurance: No insurer can predict whether a specific 45-year-old will die next year. But across 500,000 policyholders, mortality rates converge to actuarial tables. The LLN transforms individual unpredictability into business predictability.
• Quality control: Sampling 5 units from a production run leaves high uncertainty. Sampling 500 units yields a defect rate estimate that closely mirrors the true population defect rate — the LLN is why sampling works in manufacturing.

Law of Total Probability

If events B₁, B₂, ..., Bₙ form a partition of the sample space (mutually exclusive and exhaustive), then:

Permutations and Combinations

Counting methods determine the total number of possible outcomes — the denominator in any classical probability calculation. See: Permutations and Combinations | Counting Methods

Permutation worked example — PIN code: How many unique 4-digit PINs can be created from the digits 1–9 with no repetition?
9P4 = 9! / (9−4)! = 9 × 8 × 7 × 6 = 3,024 possible PINs.
Order matters here: PIN 1234 ≠ PIN 4321.

Combination worked example — Lottery: A lottery requires choosing 6 numbers from 1–49. How many possible tickets exist?
C(49,6) = 49! / [6! × 43!] = 13,983,816 possible combinations.
Order does not matter: {1,7,14,22,33,49} is the same ticket regardless of the order picked.

Use the Combination Calculator for fast computation.

Common Probability Misconceptions

Gambler's Fallacy: After 10 consecutive heads, many think tails is "due." Each flip remains 50/50 — past independent outcomes have zero influence on future ones.

Base Rate Neglect: Ignoring the prior probability of an event (see the Bayes' Theorem medical test example above). A highly accurate test can still produce mostly false positives when the condition is rare.

Hot Hand Fallacy: Believing a player on a streak is more likely to succeed next time. Research shows most streaks in sports are consistent with random chance rather than genuine momentum.

7 Real-World Applications of Statistics and Probability

Statistics and probability are not abstract — they drive decisions in virtually every field.

1. 1

   Medicine and Clinical Trials — Hypothesis testing determines whether a new drug outperforms a placebo. Bayes' Theorem improves diagnostic accuracy. Confidence intervals define the range of effective doses. The p-value threshold α = 0.05 governs drug approval in most regulatory systems.
2. 2

   Finance and Insurance — Actuaries use probability distributions to price insurance policies. Portfolio managers measure investment risk using standard deviation and Value at Risk (VaR). Options pricing (Black-Scholes) relies on normal distribution assumptions.
3. 3

   Weather Forecasting — Meteorologists apply Bayesian methods to update precipitation probabilities as new atmospheric data arrives. "70% chance of rain" is subjective probability made operational.
4. 4

   Machine Learning and AI — Naive Bayes classifiers, Gaussian mixture models, and model validation metrics (precision, recall, AUC) all depend on probability theory. Regression models are grounded in statistical inference.
5. 5

   Sports Analytics — Win probability models update in real time using conditional probability. Correlation analysis identifies which player statistics predict team wins. Regression to the mean explains why breakout seasons are often followed by normal ones.
6. 6

   Quality Control in Manufacturing — Statistical Process Control (SPC) uses control charts to detect drift. Six Sigma programs reduce defects to fewer than 3.4 per million using normal distribution principles.
7. 7

   Social Sciences and Polling — Political polls use random sampling and report margins of error (confidence intervals). Chi-square tests analyze relationships in survey data. Sampling theory ensures results from 1,000 respondents can reliably represent millions.

Statistics vs. Probability — Key Differences

Statistics and Probability Formulas — Quick Reference Sheet

Bookmark this section as your printable probability and statistics formulas cheat sheet.

Descriptive Statistics Formulas

Probability Rules Summary

Key Distributions Reference

Conclusion

Statistics and probability are inseparable tools for understanding uncertainty and making data-driven decisions. Probability gives you the mathematical language to describe randomness; statistics gives you the methods to learn from it.

Start with the basics — sample space, the four probability types, event types, and the core rules. Then build toward distributions, expected value, and inferential statistics. Every advanced topic in data science, machine learning, and research rests on the foundation covered in this guide. Explore deeper with Bayesian vs. Frequentist statistics or dive into Correlation vs. Causation.

Frequently Asked Questions