Bootstrap Sampling: Definition, How It Works & Examples

What Is Bootstrap Sampling? (Definition)

The method was introduced by Bradley Efron in his 1979 paper "Bootstrap Methods: Another Look at the Jackknife," published in The Annals of Statistics. The name comes from the phrase "pulling oneself up by one's bootstraps" — the idea of generating information about a population from the sample itself, with no external data required.

The core assumption is the plug-in principle: treat your observed sample as the best available proxy for the population. If you had access to the actual population you would sample from it repeatedly; since you do not, you sample from your empirical data repeatedly instead. As the original sample size grows and becomes more representative of the population, bootstrap estimates become more accurate.

What Does "Sampling With Replacement" Mean?

Sampling with replacement means each observation is placed back into the pool after being drawn, so the same data point can be selected more than once in a single bootstrap resample. For a dataset of n = 5 values, each bootstrap resample still contains n = 5 draws, but some values may appear two or three times while others may not appear at all.

On average, any single bootstrap resample includes approximately 63.2% of the original dataset's unique observations (since the probability of any given observation being excluded from one resample is (1 − 1/n)ⁿ → 1/e ≈ 0.368). The remaining ~36.8% not selected in a given resample are called the out-of-bag (OOB) observations — a concept used directly in Random Forest to estimate generalization error without a separate test set.

How Bootstrap Sampling Works — 6 Steps

Worked Example — Bootstrap Sampling Step by Step

The following example uses a small dataset to keep the arithmetic transparent. In practice you would use hundreds or thousands of resamples; here, three resamples show the mechanics clearly. This mirrors the approach described in Efron and Tibshirani's foundational textbook An Introduction to the Bootstrap (Chapman & Hall, 1993).

Bootstrap Confidence Intervals

The most common reason practitioners use bootstrap sampling is to construct a confidence interval for a statistic without making distributional assumptions. There are several methods; the two most widely used are the percentile method and the BCa method.

Percentile Method

After running B = 1,000 bootstrap resamples and computing the mean for each, sort the 1,000 means from smallest to largest. A 95% confidence interval takes the value at position 25 (the 2.5th percentile) as the lower bound and the value at position 975 (the 97.5th percentile) as the upper bound. No formula for the standard normal or t-distribution is needed.

BCa Method (Bias-Corrected and Accelerated)

The BCa method adjusts the percentile boundaries for two sources of distortion: bias (the bootstrap distribution is not centered on the original estimate) and acceleration (the standard error of the statistic changes with the parameter value). BCa intervals are more accurate than the basic percentile method, especially for skewed statistics or small samples. Most statistical software packages (R's boot library, Python's scipy.stats.bootstrap) default to BCa or offer it as an option.

Bootstrap Sampling in Python

Python's NumPy library makes bootstrap sampling straightforward with np.random.choice. The scipy.stats.bootstrap function (added in SciPy 1.7) provides BCa confidence intervals directly. The scikit-learn library does not expose a standalone bootstrap function, but the underlying mechanism is used inside BaggingClassifier and RandomForestClassifier.

import numpy as np

# Original dataset
data = np.array([62, 70, 68, 75, 65])
n = len(data)
B = 10000  # number of bootstrap resamples

def bootstrap_replicate_1d(data, func):
    """Draw one bootstrap resample and compute the statistic."""
    bs_sample = np.random.choice(data, size=len(data), replace=True)
    return func(bs_sample)

def draw_bs_reps(data, func, size=10000):
    """Draw B bootstrap replicates of a statistic."""
    return np.array([bootstrap_replicate_1d(data, func)
                     for _ in range(size)])

# Generate bootstrap distribution of the mean
bs_means = draw_bs_reps(data, np.mean, size=B)

# Bootstrap standard error
se_boot = np.std(bs_means, ddof=1)

# 95% percentile confidence interval
ci_lower = np.percentile(bs_means, 2.5)
ci_upper = np.percentile(bs_means, 97.5)

print(f"Original mean: {np.mean(data):.2f}")
print(f"Bootstrap SE: {se_boot:.4f}")
print(f"95% CI: [{ci_lower:.2f}, {ci_upper:.2f}]")

from scipy import stats
import numpy as np

data = np.array([62, 70, 68, 75, 65])

result = stats.bootstrap(
    (data,),
    np.mean,
    n_resamples=9999,
    confidence_level=0.95,
    method='BCa'   # also try 'percentile' or 'basic'
)

print(f"BCa 95% CI: {result.confidence_interval}")
print(f"Bootstrap SE: {result.standard_error:.4f}")

Bootstrap Sampling Formulas

Bootstrap Sampling in Machine Learning

Bootstrap sampling is not just a tool for statisticians — it is the foundation of some of the most widely used ensemble methods in machine learning. Understanding bootstrap sampling unlocks an intuitive grasp of bagging, Random Forest, and gradient boosting frameworks.

Bagging (Bootstrap Aggregating)

Bagging, short for Bootstrap Aggregating, was introduced by Leo Breiman in 1996. The procedure: generate B bootstrap samples from the training data, fit a separate model (decision tree, or any other learner) to each sample, then aggregate predictions by averaging (regression) or majority voting (classification). Because each model is trained on a slightly different bootstrap sample, individual models make different errors, and averaging reduces overall variance.

Bootstrap Sampling in Random Forest

Random Forest extends bagging by adding a second source of randomization: at each node split, only a random subset of features is considered (typically √p for classification, p/3 for regression, where p is the total number of features). This decorrelates the individual trees further, reducing variance beyond what bagging alone achieves.

The role of bootstrap sampling in Random Forest is to create B different training sets, ensuring that no single influential data point dominates every tree. The out-of-bag observations (the ~36.8% not selected in each resample) are used to estimate generalization error — a free cross-validation estimate that requires no separate test split. This is described in Breiman's original 2001 paper in Machine Learning (vol. 45, pp. 5–32), available through the Springer archive.

Bootstrap vs Cross-Validation vs Jackknife

Bootstrap sampling, hypothesis testing variants, cross-validation, and the jackknife all estimate statistical uncertainty through resampling, but they serve different purposes and have different tradeoffs. The table below compares the three most common methods.

Bootstrap Sampling Variations

Bootstrap Sampling Simulator

Limitations of Bootstrap Sampling

Bootstrap sampling is powerful, but it is not a cure-all. Several situations call for caution or a different approach.

Frequently Asked Questions

Related Topics on Statistics Fundamentals

Bootstrap sampling connects to several other statistical concepts covered in depth on Statistics Fundamentals. The pages below cover the prerequisite and follow-on topics most relevant to understanding and applying bootstrap methods.