Why Statistical Mistakes Matter
Statistics is the language that data speaks in. When that language is garbled, every downstream decision made from the data is built on a shaky foundation. Statistical errors are not just academic problems. They have real consequences measured in lives, money, and time.
The consequences span every field. In healthcare, a statistical error in a clinical trial can delay an effective treatment or approve a harmful one. In business, misreading statistical significance in an A/B test can cost more in engineering time than the revenue gained. In machine learning, overfitting a model makes it useless on real-world data. In social science, sampling bias produces research that describes the researcher's bubble rather than the actual population.
Understanding these mistakes also matters for consuming information. News headlines, government reports, and company press releases all use statistics, and the errors in this guide appear in all of them. Spotting a truncated axis or a confused correlation is a practical skill, not just a test topic.
Each mistake section follows the same structure: what goes wrong, a real example, and the fix. The cheat sheet near the end compresses all 15 into a single reference table. The checklist provides a practical pre-analysis workflow to prevent these errors before they happen.
The 15 Most Common Statistics Mistakes
Confusing Correlation with Causation
Two variables that move together are correlated. That shared movement does not mean one causes the other. This is the most widely cited statistical error precisely because it appears everywhere data is discussed and because the human brain is wired to see patterns and invent explanations for them.
Symptom
Saying "X causes Y" when the analysis only shows that X and Y tend to move togetherCause
Causal inference requires experimental control, temporal ordering, and ruling out confounders — correlation provides none of theseFix
Use causal language only after a controlled experiment; for observational data, say "associated with" or "correlated with"Real example. Ice cream sales and drowning rates both peak in summer. The correlation is real. The cause is a third variable, hot weather, which drives both independently. A policy that restricted ice cream sales would not reduce drowning. In a famous example from economics, per-capita cheese consumption and deaths from becoming tangled in bedsheets correlate almost perfectly across years — a textbook spurious correlation.
Establishing causation requires a controlled experiment or a natural experiment, temporal evidence that the cause precedes the effect, a plausible mechanism, and ruling out confounders. Observational data alone, even with a strong correlation, cannot establish causation.
"People who drink coffee score higher on memory tests" — this tells us about correlation. It tells us nothing about whether coffee improves memory, whether smarter people drink more coffee, or whether a third variable (education, income) drives both.
Misinterpreting p-values
The p-value is probably the most misunderstood number in science. It has a specific, narrow meaning that is frequently replaced with broader, more comfortable meanings that happen to be wrong.
Symptom
Saying a p-value of 0.03 means "there is a 3% chance the result is due to chance" or "97% chance the hypothesis is true"Cause
The formal definition is counterintuitive; the wrong interpretations feel more naturalFix
Report p-values alongside effect sizes and confidence intervals; never interpret p alone as evidence of truthA p-value of 0.04 means: if the null hypothesis were true, you would see a result at least this extreme about 4% of the time. That is it. Whether 4% is surprising enough to reject the null is a judgment call, and the threshold of 0.05 is a convention, not a law of nature.
The American Statistical Association published a landmark statement in 2016 warning against the misuse of p-values, calling p < 0.05 an inadequate standard for scientific evidence when used in isolation. For the full framework behind p-values, see the p-values guide.
Ignoring Sample Size
Sample size determines whether a study can detect a real effect if one exists — a property called statistical power. Studies that are too small fail to find real effects. Studies that are very large find effects too small to matter. Both failures are common, and both are avoidable.
Symptom
"No significant difference was found" — when the study never had enough participants to find a moderate effectCause
Sample size is not calculated before the study, or is set by convenience rather than by the effect size the study was designed to detectFix
Calculate required sample size before data collection using a power analysis; use the sample size calculatorSmall samples. A trial with 20 participants has roughly 20% power to detect a medium effect at α = 0.05. That means the study will miss the real effect 80% of the time. A null result from such a trial is nearly uninformative.
Large samples. With 1 million participants, a difference of 0.001% between two groups can reach statistical significance while being completely irrelevant in practice. Statistical significance does not equal importance. See effect size for the measure that fills this gap.
The standard target for statistical power is 80%, meaning the study is designed to detect the effect of interest 4 times out of 5. For high-stakes research, 90% is more appropriate. The statistical power guide covers the full calculation.
Using Biased Samples
A sample is only useful if it represents the population it is meant to describe. When it does not, every conclusion drawn from it is about the wrong people, the wrong cases, or the wrong conditions.
Symptom
Results do not replicate when the study is repeated with a different recruitment methodCause
Convenience, self-selection, or flawed inclusion criteria make the sample systematically different from the populationFix
Use random sampling methods; compare sample demographics to population data; report all exclusion criteriaThree types of sampling bias appear most often. Selection bias occurs when the sampling process favors certain members of the population over others. Volunteer bias occurs when participants self-select, meaning the people who respond to a survey are systematically different from those who do not. Survivorship bias occurs when only successful or surviving cases are analyzed — the classic example being World War II aircraft damage: studying only returned planes showed where planes could take a hit and survive, not where hits were fatal.
A famous example of sampling bias: the Literary Digest predicted a landslide victory for Alf Landon over Franklin Roosevelt in 1936, based on 10 million poll responses. The poll drew its sample from telephone directories and car registrations — which heavily overrepresented wealthy, Republican-leaning voters during the Depression. Roosevelt won in a landslide.
Mishandling Outliers
An outlier is a data point that sits far from the rest of the distribution. What to do with it is one of the most context-dependent decisions in data analysis, and the default answer — remove it — is often wrong.
Symptom
Analysis results change dramatically depending on whether a small number of extreme values are included or excludedCause
Outliers are removed to tidy results rather than investigated for their own information valueFix
Investigate before excluding: is the outlier an error, a special case, or a real extreme value? Report how many were excluded and whyWhen to remove an outlier: when there is documented evidence of a recording error, instrumentation failure, or data entry mistake. When not to remove an outlier: because it makes the result less significant, because it looks inconvenient, or because it is extreme without any known error. In fraud detection, the outlier is the point of the analysis. In quality control, outliers are the signal. Removing them destroys the value of the data.
The effect of a single outlier on the mean can be severe, which is why the mean versus median distinction matters in practice. For a dataset of incomes, one billionaire shifts the mean dramatically while leaving the median unchanged. The median is the more informative summary for skewed data with extreme values. Learn more about detecting outliers at the outliers guide.
Confusing Statistical Significance with Practical Significance
A result can be statistically significant and completely unimportant. With a large enough sample, a difference of 0.001% will clear any significance threshold. Whether that difference matters in the real world is a separate question, and one that p-values cannot answer.
Symptom
Reporting p < 0.001 on a difference too small to change any real decisionCause
Large datasets make tiny effects significant; researchers report significance without effect sizeFix
Always report effect size (Cohen's d, r², odds ratio) alongside p-values; ask whether the effect is large enough to act on| Property | Statistical Significance | Practical Significance |
|---|---|---|
| Question answered | Is this result unlikely under the null hypothesis? | Is this result large enough to matter? |
| Driven by | Sample size and effect size together | Effect size alone |
| Expressed as | p-value compared to α | Effect size, relative difference, or business impact |
| Risk of ignoring it | Treating noise as a real finding | Acting on a real but negligible difference |
Effect size measures the magnitude of a finding independent of sample size. Cohen's d for comparing two means, r² for regression, and odds ratios for categorical outcomes all provide this information. The Cohen's d guide covers the most commonly used measure in detail.
Choosing the Wrong Statistical Test
Every statistical test is built on assumptions about the data: its distribution, its measurement scale, its independence structure. Using the wrong test produces invalid results, even when the math is done correctly.
Symptom
Using a t-test on non-normal, small-sample data, or a chi-square test on continuous variablesCause
Test selection is made by habit or by copying a previous study, rather than by checking what the data and research question actually requireFix
Use a test selection guide; check assumptions formally before running any test| Situation | Correct Test | Common Wrong Choice |
|---|---|---|
| Compare means, 2 independent groups, normal data | Independent samples t-test | One-sample t-test |
| Compare means, 3+ groups | ANOVA | Multiple t-tests (inflates error rate) |
| Relationship between 2 categorical variables | Chi-square test | Correlation coefficient |
| Non-normal data or small sample (n < 30) | Mann-Whitney U or Wilcoxon signed-rank | t-test (violates normality assumption) |
| Paired / repeated measures data | Paired samples t-test | Independent samples t-test (ignores pairing) |
The statistical test selector at Statistics Fundamentals provides a step-by-step guide for matching research questions and data types to the correct test. The statistical assumptions guide covers how to formally check each assumption before proceeding.
Misreading Graphs and Charts
A graph can make a trivial difference look dramatic or make a dramatic difference disappear, depending on how it is constructed. These effects are sometimes accidental and sometimes deliberate, but they mislead either way.
Symptom
A bar chart suggests a massive difference between groups, but the y-axis starts at 97 rather than 0Cause
Presenters optimize for visual impact; readers do not check axis originsFix
Always check axis origins; verify that visual proportions match numerical proportions; treat 3D charts with deep suspicionThe most common data visualization mistakes are: a truncated y-axis that makes a small difference look large; inconsistent bin widths in histograms that distort the shape of a distribution; dual axes with different scales on the same chart, allowing the presenter to suggest any relationship they want; cherry-picked time ranges that show a trend which would reverse or flatten with more data; and area charts used to represent a quantity where only length should be used.
A practical rule: before accepting a conclusion from a chart, look at the axes, check where zero is, verify the scale is consistent, and ask what the same data would look like plotted differently. The data visualization guide covers chart selection and common misleading techniques in detail.
Ignoring Confidence Intervals
A p-value answers a yes/no question about whether a result is significant. A confidence interval answers a more useful question: how large is the effect, and how precisely do we know it?
Symptom
Reporting only p = 0.04 with no interval, leaving readers unable to judge whether the effect is meaningfully large or trivially smallCause
Journals historically prioritized p-values; effect size and interval reporting has been a more recent requirementFix
Report 95% confidence intervals for all main estimates; interpret the practical range the interval spans, not just whether it excludes zeroA 95% confidence interval that spans [0.01, 0.03] tells a very different story from one spanning [0.01, 4.7] — even if both produce p = 0.04 and both technically exclude zero. The first estimate is precise and small. The second is consistent with anything from a negligible to a very large effect. The interval is the finding; the p-value is only a filter.
A common misreading of confidence intervals: a 95% CI does not mean there is a 95% probability the true value falls within this particular interval. It means that the procedure used to construct the interval would contain the true value 95% of the time across many repetitions. The distinction matters. See the confidence intervals guide for the full framework, and the confidence interval for a mean page for step-by-step calculation.
Violating the Assumptions Behind Statistical Tests
Every parametric test carries a set of assumptions about the data. When those assumptions fail, the test's output — p-values, confidence intervals, everything — cannot be trusted.
Symptom
Running a t-test on five data points that clearly do not follow a normal distributionCause
Assumptions are listed in textbooks but rarely checked in practice; statistical software runs any test on any data without warningFix
Check normality with a Shapiro-Wilk test or Q-Q plot; check equal variance with Levene's test; switch to nonparametric tests if assumptions failThe three most commonly violated assumptions are normality (many tests assume data follows a normal distribution), independence (observations must be independent of each other), and equal variance (many two-sample tests assume both groups have similar spread). For normality checking, the Q-Q plot guide shows the visual method, and the normality tests guide covers formal testing. For variance, the equal vs unequal variance guide explains when to use Welch's correction.
Multiple Testing Without Correction
Run enough tests at α = 0.05 and you will eventually find something "significant" by pure chance. If you test 20 independent hypotheses and all are actually null, you expect to find one false positive on average — not because the finding is real, but because probability says you will.
Symptom
A study tests 40 genetic variants and reports the 2 that reached p < 0.05 without mentioning the other 38Cause
Researchers test multiple outcomes or subgroups and report selectively, or genuinely do not realize this inflates the false positive rateFix
Pre-register the primary outcome; apply Bonferroni correction or the false discovery rate (FDR) procedure when running multiple testsThe Bonferroni correction is the simplest fix: divide α by the number of tests. If you run 10 tests, use α = 0.005 rather than 0.05 for each. This is conservative. The Benjamini-Hochberg false discovery rate procedure is less conservative and more appropriate when many tests are run simultaneously, such as in genomics or large-scale A/B testing programs. The Bonferroni adjusted critical value table provides the corrected thresholds for common scenarios.
The related problem of p-hacking — running many tests and reporting only significant ones — is a key driver of the replication crisis in psychology and biomedicine, documented extensively by researchers including John Ioannidis in his influential 2005 paper "Why Most Published Research Findings Are False."
Overfitting Models
Overfitting means a model has learned the specific noise in the training data rather than the underlying pattern. It produces spectacular performance on the data it was trained on and poor performance on new data — which is the only performance that matters.
Symptom
Training accuracy of 98%, test set accuracy of 61% — the model memorized the training dataCause
Too many parameters relative to data; no regularization; evaluating model performance only on training dataFix
Use cross-validation; apply regularization (ridge, lasso, dropout); reserve a held-out test set that is never used in developmentOverfitting is not only a machine learning problem. In regression analysis, adding more predictors always increases R² even when those predictors are random noise. Adjusted R² and information criteria like AIC penalize complexity for this reason. In survey research, fitting a model to every quirk of one dataset produces results that do not replicate in any other dataset. For the bias-variance tradeoff that underlies this problem, see the bias-variance tradeoff guide.
Cross-validation splits the data into training and validation folds, evaluating each model on data it has never seen. K-fold cross-validation is the standard method. The separate test set must remain untouched throughout all model development decisions — it is evaluated exactly once, at the very end, to estimate real-world performance.
Ignoring Missing Data
Most real datasets have missing values. How those values are missing matters as much as how many are missing. Dropping incomplete rows — the default in most software — is valid only under one specific and usually unmet assumption: that data is missing completely at random.
Symptom
A dataset with 40% missing values in a key variable, analyzed by dropping all incomplete rowsCause
Default software behavior deletes missing rows; researchers do not investigate why values are missingFix
Investigate missingness patterns; use multiple imputation when data is not MCAR; report the extent and pattern of missing dataThere are three missingness mechanisms. Missing Completely at Random (MCAR): the probability a value is missing has no relationship to any variable in the dataset. Listwise deletion is valid here. Missing at Random (MAR): the probability of missingness depends on other observed variables, not the missing variable itself. Multiple imputation works here. Missing Not at Random (MNAR): the probability of missingness depends on the missing value itself — for example, sicker patients are more likely to miss appointments and also have worse health outcomes. This is the hardest case and requires domain knowledge to address.
A clinical trial where patients with the worst outcomes are the ones most likely to drop out will, if only completers are analyzed, systematically overestimate the treatment's effectiveness.
Reporting Only Significant Results (Publication Bias)
Publication bias is the tendency for significant, positive results to be published while null results sit in file drawers. When only published studies enter a field's body of knowledge, the evidence base is systematically distorted toward showing effects that are larger and more consistent than reality.
Symptom
A meta-analysis finds 50 studies showing an effect and only 3 showing no effect — statistically implausible without selective reportingCause
Journals prefer novel positive findings; researchers expect rejection for null results and do not submit themFix
Pre-register studies; register null results in open registries; use funnel plots to detect publication bias in meta-analysesThe effect of publication bias is measurable. Funnel plots in meta-analyses frequently show asymmetric distributions of effect sizes — a signature of missing small null studies. When corrected for, many celebrated effects in psychology and social science shrink substantially or disappear entirely. This is a central mechanism behind the replication crisis: the published literature presents a curated, optimistic picture of reality.
Pre-registration — publicly committing to a hypothesis, sample size, and analysis plan before data collection — is the most reliable structural fix. The Open Science Framework and ClinicalTrials.gov provide public pre-registration for academic and clinical research.
Overcomplicating Simple Problems
More complex is not more accurate. A common failure mode, especially among analysts who have just learned a powerful technique, is to apply it where a simpler method would answer the question more clearly, more interpretably, and with fewer assumptions to violate.
Symptom
A neural network trained to predict house prices when a linear regression with three variables would perform equally well and explain its predictionsCause
Complex models are associated with sophistication; simple models are associated with naivety (wrongly)Fix
Start with the simplest model that could plausibly work; add complexity only when it produces measurable improvement on out-of-sample dataOccam's razor applies in statistics: among models with equal predictive performance, prefer the simpler one. Simple models are easier to communicate, easier to audit, less prone to overfitting, and more likely to hold up when deployed on new data. A decision tree that fits on a page and achieves 84% accuracy is often more valuable than a gradient-boosted ensemble achieving 86% that no one can explain or maintain.
This principle extends beyond model selection. When a two-by-two contingency table answers the research question, a logistic regression does not improve the answer — it adds complexity without adding insight. Interpretability is a legitimate scientific value, not a consolation prize for failing to use a harder method.
Real-World Case Studies in Statistical Mistakes
These cases span medicine, marketing, social science, and machine learning. Each one illustrates a specific mistake from the list above in a setting where the consequences were real.
The DICE Trial and Subgroup Dredging
Subgroup analysis — testing whether a treatment works better in men vs women, or older vs younger patients — is a legitimate tool when pre-specified. It becomes a problem when conducted post-hoc across dozens of subgroups to find one that looks significant. In the DICE trial studying anti-inflammatory drugs, researchers who analyzed enough subgroups found that the drug appeared beneficial for patients born under certain astrological signs — a finding reported as a cautionary example by epidemiologists at Oxford. The correct procedure is to declare primary and secondary outcomes before seeing the data.
When a Tiny Conversion Lift Is Not Worth Shipping
A major e-commerce platform ran an A/B test on button color across 500,000 sessions and found a 0.12 percentage point improvement in checkout completion (p < 0.001). The result was statistically significant. The team calculated the actual revenue impact: roughly $4,200 per year across all users. The engineering cost to implement, test, and deploy the change exceeded $40,000. Statistical significance told them the effect was real. Practical significance told them not to ship it. Both judgments were correct and both were necessary.
The Radiologist-Beating AI That Failed in Deployment
Several AI systems trained to detect pneumonia from chest X-rays achieved better accuracy on benchmark test sets than radiologists. When deployed in hospitals outside the training institution, performance dropped sharply. Subsequent analysis found the models had learned to identify the X-ray machines and scanner settings used at the training hospital — artifacts correlated with disease prevalence in that dataset — rather than the disease itself. The models were overfitting to institutional fingerprints invisible to humans. Cross-validation within a single hospital's data did not catch this; only external validation across institutions did.
Storks and Birth Rates
A now-famous satirical paper demonstrated a strong positive correlation between stork populations and human birth rates across European countries. Both variables are independently correlated with a third: rural, agricultural land area, which supports both stork habitat and higher birth rates. The paper was written as a pedagogical example of spurious correlation, but the same logical error appears in serious research — including early studies linking dietary fat to heart disease that failed to account for confounders including total caloric intake, smoking, and activity level.
University Admissions and Gender Bias
In the mid-1970s, UC Berkeley's graduate admissions appeared to show significant gender bias: men were admitted at a higher overall rate than women. When researchers examined admissions rates department by department, women were actually admitted at equal or higher rates in most departments. The aggregate bias appeared because women disproportionately applied to more competitive departments with lower admission rates overall. The aggregate statistic described a real pattern but pointed to a misleading cause. This case is now the canonical example of Simpson's paradox in statistics education.
Statistics Mistakes Cheat Sheet
This table compresses all 15 mistakes into a single reference. Use it as a quick pre-analysis review or as a teaching reference.
| # | Mistake | Symptom | Fix |
|---|---|---|---|
| 1 | Correlation ≠ Causation | Saying X causes Y from observational data | Use causal language only from controlled experiments |
| 2 | Misinterpreting p-values | p = 0.04 means 96% chance the hypothesis is true | Report p with effect size and confidence interval |
| 3 | Ignoring sample size | Null result from 15-person study used as evidence | Run power analysis before data collection |
| 4 | Biased samples | Survey responses from only willing participants | Random sampling; check representativeness |
| 5 | Mishandling outliers | Removing outliers that make results look cleaner | Investigate cause; report exclusion criteria |
| 6 | Statistical vs practical significance | p < 0.001 on a 0.001% difference | Report effect size; ask if the finding is actionable |
| 7 | Wrong statistical test | t-test on non-normal count data | Match test to data type and assumptions |
| 8 | Misreading graphs | Y-axis starts at 97, making a 1% difference look 10× | Check axis origins; verify proportions numerically |
| 9 | Ignoring confidence intervals | Reporting only p = 0.04 with no range | Report 95% CI; interpret the interval's width |
| 10 | Violating test assumptions | Running ANOVA without checking normality | Test assumptions formally; use alternatives when needed |
| 11 | Multiple testing without correction | Testing 30 outcomes, reporting 2 significant ones | Apply Bonferroni correction or FDR; pre-register |
| 12 | Overfitting | 99% training accuracy, 55% test accuracy | Cross-validate; hold out a test set; regularize |
| 13 | Ignoring missing data | Dropping all rows with any missing value | Diagnose missingness mechanism; use imputation if MAR |
| 14 | Publication bias | Meta-analysis draws on only positive published studies | Pre-register studies; report null results |
| 15 | Overcomplicating | Neural network where linear regression would work | Start simple; add complexity only when it pays off |
Statistical Test Selection Guide
Choosing the right test requires answering three questions about your data: what is the measurement scale, how many groups are being compared, and whether observations are independent or paired. This flowchart traces those questions to the appropriate test.
Statistical Test Selector — From Data Type to the Right Test
For an interactive version of this guide with detailed explanations of each branch, see the statistical test selector. For the full assumptions that govern each test, see the statistical test assumptions guide.
Statistics Best Practices
The 15 mistakes above share a common thread: they are all prevented by good statistical practice applied at the right stage of an analysis. The checklist below translates that practice into concrete steps.
Plan First
- Define the primary research question before touching data
- Calculate required sample size using a power analysis
- Pre-specify the primary outcome and key covariates
- Choose the statistical test before seeing results
- Register the analysis plan if publishing
Check Assumptions
- Examine data distribution with Q-Q plots and histograms
- Test for normality with Shapiro-Wilk when n < 50
- Check for equal variances with Levene's test
- Verify observation independence
- Investigate missing data patterns before imputation
Report Transparently
- Report effect sizes alongside p-values
- Include confidence intervals for all main estimates
- Disclose all tests run, including non-significant ones
- Describe sample representativeness and exclusion criteria
- Report how missing data was handled
Exploratory data analysis before any inferential test is worth treating as a rule rather than a suggestion. Visualizing distributions, checking for outliers, and examining bivariate relationships before running formal tests prevents the most common assumption violations before they happen. The exploratory data analysis guide covers a full EDA workflow.
Key Comparison Tables
Type I vs Type II Errors
| Property | Type I Error (False Positive) | Type II Error (False Negative) |
|---|---|---|
| What happened | Rejected H₀ when H₀ was actually true | Failed to reject H₀ when H₀ was actually false |
| Controlled by | Significance level α (set to 0.05) | Statistical power 1 − β (target 0.80) |
| Real-world cost | Acting on a false finding; shipping a change that does nothing | Missing a real finding; failing to treat a real disease |
| Reduced by | Lowering α; using Bonferroni correction; pre-registration | Increasing sample size; increasing effect size; increasing α |
See the full treatment at the Type I and Type II errors guide.
Parametric vs Nonparametric Tests
| Property | Parametric Tests | Nonparametric Tests |
|---|---|---|
| Assumes normality | Yes — violating this inflates error rates | No — distribution-free |
| Power (when assumptions met) | Higher — uses more information from the data | Lower — converts data to ranks |
| Best used when | n > 30 and data approximately normal | n < 20, non-normal, ordinal, or skewed data |
| Examples | t-test, ANOVA, Pearson correlation | Mann-Whitney U, Wilcoxon, Spearman rank |
Mean vs Median — When to Use Each
| Situation | Mean | Median |
|---|---|---|
| Symmetric, roughly normal distribution | Preferred — uses all information | Similar result; mean is more informative |
| Skewed distribution (e.g., income) | Misleading — pulled by extreme values | Preferred — resistant to outliers |
| Presence of outliers | Sensitive — a single outlier can shift dramatically | Resistant — not affected by extreme values |
| Ordinal data | Inappropriate — arithmetic on ranks is meaningless | Appropriate |
See the mean vs median vs mode guide for worked examples across different data types.
Statistics Glossary
| Term | Plain-English Definition | See Also |
|---|---|---|
| Correlation | A measure of how two variables tend to change together, ranging from −1 to +1. Does not imply causation. | Pearson Correlation |
| Causation | A relationship where one variable directly produces a change in another, requiring experimental evidence. | — |
| p-value | The probability of observing a result at least as extreme as the one measured, assuming the null hypothesis is true. | p-values guide |
| Statistical Significance | A result is statistically significant when its p-value falls below the chosen significance level α. | Significance Level |
| Confidence Interval | A range of plausible values for a population parameter, constructed so that the interval contains the true value in a stated percentage of repetitions. | Confidence Intervals |
| Sampling Bias | A systematic error in which some members of a population are more likely to be selected than others, making the sample non-representative. | — |
| Effect Size | A measure of the magnitude of a statistical result, independent of sample size. Cohen's d, r², and odds ratios are common measures. | Effect Size |
| Outlier | A data point that falls far from the rest of the distribution. May indicate an error, a rare event, or a genuinely extreme observation. | Outliers |
| Regression | A family of statistical methods that model the relationship between a dependent variable and one or more independent variables. | Simple Linear Regression |
| Type I Error | A false positive: rejecting the null hypothesis when it is actually true. Rate controlled by α. | Type I & II Errors |
| Type II Error | A false negative: failing to reject the null hypothesis when it is actually false. Rate controlled by statistical power (1 − β). | Type I & II Errors |
| Statistical Power | The probability that a test correctly detects a real effect of a given size. Standard target: 80%. | Statistical Power |
| Simpson's Paradox | A phenomenon where a trend present in subgroups reverses or disappears when the groups are combined. | — |
| Multiple Testing | The practice of running several statistical tests on the same dataset, inflating the false positive rate above the nominal level. | Bonferroni correction; FDR |
| Overfitting | When a model learns the noise in training data rather than the underlying pattern, producing poor performance on new data. | Bias-Variance Tradeoff |
| Normal Distribution | A symmetric, bell-shaped probability distribution characterized by its mean and standard deviation. Many parametric tests assume data follows this distribution. | Normal Distribution |
| Standard Deviation | A measure of how spread out the values in a dataset are around the mean. | Standard Deviation |
| Hypothesis Testing | A formal framework for deciding whether observed data provides sufficient evidence against a null hypothesis. | Hypothesis Testing |
Frequently Asked Questions
Key references for this guide: American Statistical Association Statement on p-Values (2016) · Ioannidis, J.P.A. (2005). "Why Most Published Research Findings Are False." PLOS Medicine · Open Science Collaboration. "Estimating the Reproducibility of Psychological Science." Science (2015) · Greenland et al. "Statistical Tests, P Values, Confidence Intervals, and Power." European Journal of Epidemiology (2016) · Statistics Fundamentals — The complete statistics learning resource