What Is Exploratory Data Analysis (EDA)?
Exploratory data analysis (EDA) is the process of examining a dataset using summary statistics and visualizations to understand its structure, detect errors, and uncover patterns before applying formal models. Introduced by statistician John Tukey in 1977, EDA treats data as something to be interrogated rather than assumed, forming the essential first step in any data science or machine learning workflow.
John Tukey coined the term and outlined the approach in his 1977 book Exploratory Data Analysis, arguing that analysis should begin with an open mind rather than a predetermined model.
The most direct way to understand EDA is by contrast with what comes before and after it. Before EDA, a dataset is a black box: a collection of numbers and labels whose reliability, distribution, and relationships are unknown. During EDA, that box is opened. After EDA, the analyst knows which variables matter, which contain errors, how the data is distributed, and what questions the data can and cannot answer.
EDA is often described as both an attitude and a set of techniques. The attitude is a willingness to be surprised by data and to let observations drive the analysis rather than confirming assumptions. The techniques are the specific tools that make that attitude operational: histograms, box plots, correlation matrices, summary statistics tables, and more. This guide covers both.
Why Exploratory Data Analysis Matters
Skipping EDA is one of the most common mistakes in data science, and it consistently leads to the same kinds of problems: models trained on miscoded variables, conclusions drawn from distributions that violate their assumptions, and business decisions based on datasets that contain thousands of duplicate rows. The time spent on EDA almost always returns more value than the time it cost.
Concrete Benefits of EDA
EDA delivers several specific, measurable benefits before any formal analysis begins. It surfaces missing values that would otherwise silently distort model outputs. It reveals the shape of each variable's distribution, which determines which statistical tests are appropriate. It identifies variables that are so highly correlated with each other that including both in a regression model would inflate standard errors and obscure real effects. It catches obvious data entry errors, such as ages recorded as 999 or negative prices, that would corrupt any analysis built on them.
| Benefit | What EDA Reveals | Why It Matters |
|---|---|---|
| Data quality check | Missing values, duplicates, impossible entries | Prevents models from training on corrupted data |
| Distribution understanding | Skewness, kurtosis, modality, outliers | Determines which statistical tests and models are appropriate |
| Relationship discovery | Correlations, non-linear associations, interactions | Identifies which features are worth including in a model |
| Pattern detection | Trends, seasonality, clusters | Generates hypotheses that formal analysis can then test |
| Business insight | Segment differences, anomalies, concentration | Informs decisions independently of any formal model |
A classic illustration: a retail chain runs a demand forecast model that consistently over-predicts sales for one product category. An EDA run after the fact reveals that one store's sales data was recorded in units while every other store recorded in cases. The model was averaging apples and crates. Thirty minutes of EDA at the start would have caught what months of model tuning could not fix.
The EDA Workflow: Ten Steps from Raw Data to Insights
EDA is not a single action but a sequence of increasingly detailed observations. The steps below form a repeatable workflow that can be applied to any structured dataset, from a ten-row spreadsheet to a million-row database table. Later steps build on earlier ones, so following the order matters.
Understand the Dataset and Its Context
Before opening the file, answer: what was this data collected for, who collected it, over what time period, and what does each column represent? This context shapes every interpretation that follows.
Import and Inspect the Structure
Load the data and examine its shape: how many rows and columns, what data types each column contains, and whether the first few rows look as expected. In Pandas, df.info() and df.head() cover this step.
Identify Variable Types
Classify each column as numerical (continuous or discrete) or categorical (nominal or ordinal). Variable type determines which statistics and charts apply to each column.
Handle Missing Values
Count missing values per column, determine whether the missingness is random or systematic, and decide on a strategy: removal, imputation with the mean or median, or forward-fill for time-series data.
Remove or Investigate Duplicate Records
Duplicate rows inflate sample sizes and can systematically bias summary statistics. Check for exact duplicates first, then near-duplicates where a key field (customer ID, transaction ID) appears more than once with slightly different values in other columns.
Compute Descriptive Statistics
For numerical columns, compute mean, median, standard deviation, minimum, maximum, and quartiles. For categorical columns, compute value counts and frequency proportions. In Pandas, df.describe() covers the numerical side.
Visualize Distributions
Plot a histogram or density plot for each numerical variable to see the shape of its distribution. Use bar charts for categorical variables to see how values are distributed across categories.
Detect Outliers
Use box plots to spot values that fall far beyond the interquartile range, and verify whether each outlier is a data error, a legitimate extreme observation, or a signal worth keeping for analysis.
Analyze Relationships Between Variables
Compute the Pearson correlation coefficient for pairs of numerical variables, visualize relationships with scatter plots and pair plots, and use a heatmap to see the full correlation matrix at once.
Summarize Findings and Prepare for Modeling
Document what was found: which variables have strong predictive relationships, which need transformation, which should be excluded, and what questions the EDA raised that the modeling phase should address.
Descriptive Statistics Used in EDA
Descriptive statistics reduce a column of potentially thousands of numbers to a small set of values that capture its essential character. Each statistic answers a specific question about the data. Together, they form a complete summary of what a single variable looks like. The descriptive statistics guide covers the full theory behind each measure; this section explains each one in the context of EDA.
Mean
The mean is the arithmetic average of all values in a variable: the sum of all observations divided by the number of observations. In EDA, the mean gives a quick center-of-gravity estimate for a numerical variable. Its weakness is sensitivity to outliers: a salary dataset with one executive earning $5 million will have a mean that misrepresents the experience of most employees. The mean guide covers population and sample versions, and the mean calculator computes it directly.
x̄ = sample mean
xᵢ = each observed value
n = number of observations
Median
The median is the middle value when observations are sorted from smallest to largest. Half the data falls below it, half above. Because it is based on rank rather than magnitude, the median is resistant to outliers and is often a better summary of "typical" when a distribution is skewed. For the salary dataset above, the median salary reflects what the middle employee earns, not the average pulled upward by the executive. The median guide covers the exact calculation for even- and odd-numbered datasets, and the median calculator computes it from raw data.
Mode
The mode is the most frequently occurring value in a variable. For categorical variables, it is simply the most common category. For numerical variables, it is most useful when examining discrete data (whole numbers), or when looking at a binned histogram to identify the peak of a distribution. A dataset can have one mode (unimodal), two modes (bimodal), or more (multimodal), and a bimodal distribution often indicates that two distinct subpopulations are mixed together. The mode guide covers all three cases.
Variance and Standard Deviation
Variance measures the average squared distance of each observation from the mean, and standard deviation is its square root, expressed in the same units as the original variable. In EDA, standard deviation is the standard way to describe how spread out a numerical variable is. A variable with a small standard deviation has values clustered tightly around the mean; a large standard deviation signals that values are spread widely and that the mean alone may be a poor summary. The standard deviation guide and variance guide cover both, and the standard deviation calculator computes them from any dataset.
xᵢ = each observation
x̄ = sample mean
n - 1 = Bessel's correction for sample data
Quartiles, Percentiles, and the Interquartile Range
Quartiles divide a sorted dataset into four equal parts. The first quartile (Q1) marks the 25th percentile, the second quartile (Q2) is the median at the 50th percentile, and the third quartile (Q3) marks the 75th percentile. The interquartile range (IQR = Q3 - Q1) measures the spread of the middle 50% of the data, making it the outlier-resistant counterpart to standard deviation. Values more than 1.5 × IQR below Q1 or above Q3 are conventionally flagged as outliers in box plots. The IQR guide, percentiles guide, and five-number summary guide cover all of these in depth.
Skewness and Kurtosis
Skewness measures how asymmetric a distribution is. A right-skewed distribution (positive skewness) has a long tail extending to higher values, with the mean pulled above the median. A left-skewed distribution (negative skewness) has a long tail extending to lower values. Kurtosis measures whether a distribution has heavier or lighter tails than a normal distribution. In EDA, both statistics help decide whether a transformation (such as taking the logarithm) would make the variable more suitable for methods that assume normality. The QQ plot guide covers the visual approach to testing normality.
| Statistic | Measures | Sensitive to Outliers? | Best For |
|---|---|---|---|
| Mean | Center of data (average) | Yes | Symmetric distributions without extreme values |
| Median | Middle value (50th percentile) | No | Skewed distributions, income, housing prices |
| Mode | Most frequent value | No | Categorical variables, discrete counts |
| Standard Deviation | Typical spread around the mean | Yes | Comparing variability across numerical variables |
| IQR | Spread of the middle 50% | No | Outlier detection, skewed distributions |
| Skewness | Tail direction and asymmetry | Yes | Deciding whether to log-transform a variable |
| Kurtosis | Tail heaviness relative to normal | Yes | Understanding extreme-value risk |
Data Visualization Techniques in EDA
Numbers summarize but charts reveal. A dataset where the mean, variance, and correlation between two variables are identical can still look completely different depending on the actual shape of the data, a famous demonstration made by Francis Anscombe's quartet and later extended by the Datasaurus Dozen. The following charts form the standard toolkit for EDA visualization. The data visualization guide covers the broader context of when and why each type works.
Histogram
A histogram divides a numerical variable into equal-width bins and shows how many observations fall into each bin. It reveals the shape of the distribution: whether it is symmetric, skewed, bimodal, or uniform. In EDA, histograms are typically the first chart drawn for every numerical variable. They answer the question: where do most values concentrate, and are there any unusual gaps or peaks? The histogram maker generates one from any dataset.
Box Plot
A box plot displays the five-number summary (minimum, Q1, median, Q3, maximum) as a compact diagram, with individual points beyond the whiskers flagged as potential outliers. Box plots are particularly useful for comparing the distribution of a numerical variable across multiple categories side by side: how does delivery time vary by shipping method, or how do exam scores compare across grade levels? The box plot generator and box plot comparison tool both support this.
Scatter Plot
A scatter plot places each observation as a point on a two-dimensional grid, with one variable on each axis. It reveals the direction and rough strength of the relationship between two numerical variables, and whether that relationship is linear, curved, or absent. Clusters, bands, and fan shapes in a scatter plot often point to relationships that a simple correlation coefficient does not fully capture. The scatter plot maker and the scatter plots and correlation guide cover interpretation in detail.
Bar Chart
A bar chart shows the count or proportion of observations in each category of a categorical variable. In EDA, bar charts answer questions like: which product category sells most, which age group is most represented, and which region accounts for the most revenue? The bar chart maker handles both count and percentage formats.
Correlation Heatmap
A correlation heatmap displays the full pairwise correlation matrix as a color-coded grid, where dark colors signal strong correlations and light colors signal weak ones. In EDA, a heatmap is the fastest way to see at a glance which pairs of variables are strongly related and which pairs are independent. It also immediately flags potential multicollinearity issues for regression modeling, where two predictor variables are so correlated with each other that including both causes problems.
Pair Plot
A pair plot (also called a scatterplot matrix) shows scatter plots for every combination of two numerical variables in a dataset, with histograms or density plots on the diagonal. It provides a comprehensive overview of all pairwise relationships in one visualization. For datasets with up to around ten numerical variables, a pair plot is one of the most efficient single charts in EDA.
Histogram
Distribution shape of a single numerical variable
Use for: every numerical column, firstBox Plot
Five-number summary and outlier detection
Use for: comparisons across groupsScatter Plot
Relationship between two numerical variables
Use for: correlation and pattern detectionHeatmap
Full correlation matrix in color-coded grid
Use for: feature selection overviewBar Chart
Category counts or proportions
Use for: categorical variablesViolin Plot
Distribution shape plus box plot summary
Use for: comparing distributions across groupsLine Chart
Change in a variable over time
Use for: time-series dataPair Plot
All pairwise scatter plots in one view
Use for: multi-variable relationship overviewDetecting and Handling Data Quality Issues
Raw data rarely arrives clean. Missing values, duplicates, formatting inconsistencies, and outliers are the rule rather than the exception in real datasets. EDA is the process by which these problems are found; data preprocessing is where they are fixed. Understanding what each type of problem looks like determines the right solution for it.
Missing Values
Missing values appear in a dataset when a measurement was not recorded, a survey question was skipped, or a system failed to log an entry. They require different treatments depending on their pattern. Values that are missing completely at random (MCAR) can often be dropped without introducing bias. Values that are missing in a way related to another variable (MAR or MNAR) require more careful imputation: filling with the column mean or median for numerical variables, or the mode for categorical ones, or using more sophisticated methods like k-nearest-neighbor imputation for datasets where relationships between variables are strong.
Duplicate Records
Duplicate rows inflate sample sizes and can systematically distort any aggregate statistic. A dataset of retail transactions where each sale was logged twice will show sales figures exactly double their real values. Before computing any summary statistics, checking for exact duplicates (all columns match) and near-duplicates (the same transaction ID appears twice with slightly different timestamps) is a necessary early step.
Outliers
An outlier is an observation that falls far from the bulk of the data. In EDA, outliers deserve investigation rather than automatic removal. Some outliers are genuine: a customer who purchases 500 units when everyone else buys between 1 and 10, for example, may represent a wholesale buyer who belongs in a separate analysis segment. Others are errors: a temperature reading of 500°C in a dataset of room temperatures is almost certainly a sensor malfunction. The outliers guide covers both the IQR method and the Z-score method for detection, and the outlier detector tool applies both automatically.
A value is flagged as an outlier if it falls below Q1 - 1.5 × IQR or above Q3 + 1.5 × IQR. These bounds are drawn as the whiskers on a standard box plot. Values beyond the whiskers are plotted as individual points. For skewed distributions, a more conservative threshold of 3.0 × IQR is sometimes used to reduce false positives.
Inconsistent Formatting and Invalid Values
Formatting inconsistencies are subtle but damaging. A categorical variable for country might contain "US", "USA", "United States", and "u.s." as separate entries that all refer to the same country but will be treated as four distinct categories by any model or aggregation. Date columns stored as text strings require parsing before they can be used in time-series analysis. Numerical columns stored as text (because a single entry contains a currency symbol or comma) need conversion before any arithmetic is possible. A systematic check of data types at the start of EDA catches all of these.
Class Imbalance
In datasets intended for classification modeling, class imbalance occurs when one category of the target variable appears far more often than others. A fraud detection dataset where 99% of transactions are legitimate and 1% are fraudulent is class-imbalanced. EDA surfaces this by computing the value counts and proportions of the target variable. A model trained on such data without adjustment will learn to predict "not fraud" for every transaction and still achieve 99% accuracy, while being completely useless for its actual purpose.
Correlation and Relationship Analysis
Relationships between variables are the most analytically rich information EDA can produce. A variable that is strongly correlated with the outcome of interest is a candidate for inclusion in a predictive model. Variables that are highly correlated with each other raise multicollinearity concerns if both are included in the same regression.
The Pearson Correlation Coefficient
The Pearson correlation coefficient measures the strength and direction of the linear relationship between two numerical variables, on a scale from -1 to 1. A value near 1 indicates that the two variables tend to rise and fall together. A value near -1 indicates that one tends to rise when the other falls. A value near 0 indicates that the two variables do not have a consistent linear relationship. The Pearson correlation guide covers the formula and interpretation in full, the correlation calculator computes it from raw data, and the Pearson correlation table provides critical values for testing significance.
-1 ≤ r ≤ 1
r = 1: perfect positive linear relationship
r = -1: perfect negative linear relationship
r ≈ 0: no linear relationship
Covariance vs. Correlation
Covariance measures the direction of the relationship between two variables, but its magnitude depends on the units of each variable, making it impossible to compare covariances across different pairs of variables. Correlation solves this by dividing the covariance by the product of the two standard deviations, standardizing the result to the -1 to 1 scale. In EDA, correlation is almost always the preferred measure because it allows direct comparison across any pair of variables regardless of their scales.
Two variables can be highly correlated without one causing the other. Ice cream sales and drowning rates are positively correlated because both are driven by a third variable (hot weather). In EDA, correlation identifies relationships worth investigating; it does not explain them. Confirming a causal mechanism requires experimental design or more advanced causal inference methods.
Real Example 1: Retail Sales Dataset
A retail company wants to understand its sales data before building a demand forecast. The dataset contains 10,000 rows with columns for date, product category, store region, units sold, unit price, and revenue. Here is how EDA would proceed through the full workflow.
| Step | Finding | Action Taken |
|---|---|---|
| Structure inspection | Revenue column stored as text (contains "$" symbols) | Strip currency symbol, convert to float |
| Missing values | 3.2% of unit price values are missing | Impute with median unit price per product category |
| Duplicates | 47 exact duplicate rows found | Remove all duplicates |
| Revenue distribution | Strong right skew; median ($285) much lower than mean ($412) | Flag for log transformation before regression modeling |
| Outliers | 12 transactions with revenue above $5,000 (IQR bound: $890) | Confirm as bulk wholesale orders; keep but segment separately |
| Category analysis | Electronics represents 41% of revenue but only 18% of units sold | Note for pricing strategy analysis |
| Correlation | Units sold and revenue: r = 0.87; unit price and units sold: r = -0.52 | Both confirm expected price-demand and volume-revenue relationships |
Computing and interpreting the five-number summary for revenue
Sort and compute quartiles: After cleaning, the revenue column for 9,953 transactions (10,000 minus 47 duplicates) has the following five-number summary: Min = $12, Q1 = $145, Median = $285, Q3 = $520, Max = $8,750.
Compute IQR and outlier bounds: IQR = $520 - $145 = $375. Lower bound = $145 - 1.5 × $375 = -$417.50 (not applicable, revenue cannot be negative). Upper bound = $520 + 1.5 × $375 = $1,082.50. Any transaction above $1,082.50 is flagged as a potential outlier.
Identify and investigate outliers: 38 transactions exceed $1,082.50. Joining them to a store lookup table reveals that 36 of the 38 come from two stores designated as wholesale outlets. These are legitimate high-volume orders, not data errors.
Assess skewness: The mean ($412) is considerably higher than the median ($285), and the maximum ($8,750) is far beyond the Q3 bound, both confirming strong positive skew. A log transformation would be appropriate before using revenue as a dependent variable in a regression model.
✓ Result: The retail EDA identified a data type issue, confirmed wholesale outliers as legitimate, detected meaningful right skew requiring transformation, and confirmed expected relationships between price, units sold, and revenue. These findings directly inform the preprocessing steps for any subsequent demand model.
Real Example 2: Student Performance Dataset
A school district wants to understand which factors are associated with exam performance. The dataset has 395 student records with columns for math score, reading score, attendance rate, hours studied per week, parental education level, and free/reduced lunch status.
Correlation analysis and key findings
Distributions: Math scores have a mean of 66.1 and a standard deviation of 15.2. The distribution is approximately normal with a slight left skew (skewness = -0.31). Reading scores follow a similar pattern (mean 69.2, SD 14.6).
Key correlations: Hours studied and math score: r = 0.71 (strong positive). Attendance rate and math score: r = 0.63 (moderate positive). Math score and reading score: r = 0.82 (strong positive, suggesting a common underlying academic ability factor).
Categorical variable analysis: Students receiving free/reduced lunch score on average 11 points lower in math than students who do not. Students whose parents have a college degree score on average 8 points higher than those whose parents have a high school diploma. Both differences are visible in side-by-side box plots.
Outlier review: Three students with math scores below 25 are identified. Cross-referencing attendance reveals all three have attendance rates below 40%, suggesting chronic absenteeism rather than a data error.
✓ Result: EDA on the student dataset reveals that study hours and attendance are the strongest numerical predictors of math scores, that socioeconomic indicators measurably affect outcomes, and that low-scoring outliers are explained by attendance rather than data errors. A regression model built after this EDA would include hours studied, attendance rate, and socioeconomic indicators as features.
Real Example 3: Customer Analytics Dataset
An e-commerce company wants to understand customer purchase behavior for a segmentation project. The dataset has 8,000 customer records with columns for total spend, number of orders, average order value, days since last purchase, age group, and geographic region.
Segmentation insights from EDA
Total spend distribution: Strongly right-skewed. The top 10% of customers account for 63% of total revenue. Median spend is $240; mean spend is $580, pulled upward by a small number of very high-value customers. This immediately suggests that average spend is a poor metric for understanding typical behavior.
Recency analysis: Days since last purchase ranges from 1 to 730 (two years). A histogram reveals two clusters: one group with recency under 30 days and another with recency over 300 days, suggesting an active group and a lapsed group worth treating differently in marketing campaigns.
Spend vs. order count: Scatter plot shows a moderate positive correlation (r = 0.58). Many high-spend customers achieve it through a small number of large orders rather than many small ones, which is visible as a cluster of points in the upper-left of the scatter plot.
Geographic concentration: Bar chart reveals that three regions account for 74% of orders despite representing 40% of the customer base, indicating geographic concentration that any demand planning model must account for.
✓ Result: EDA on the customer dataset reveals a highly skewed revenue distribution, a bimodal recency pattern indicating two natural customer groups, and geographic concentration of demand. These findings directly shape a segmentation strategy before any clustering algorithm is applied.
EDA Before Machine Learning
The relationship between EDA and machine learning is direct: EDA findings determine what preprocessing steps are required, which features are worth engineering, and what kind of model is appropriate. A machine learning pipeline built without EDA is built on assumptions that may be completely wrong.
Feature Selection
A correlation heatmap from EDA immediately identifies which input variables have a meaningful relationship with the target variable. Variables with near-zero correlation to the target are generally poor predictors. Variables with very high mutual correlation are candidates for removal to avoid multicollinearity in linear models. EDA narrows the feature space before model training begins, which speeds up training and often improves model performance by removing noise.
Feature Engineering
EDA sometimes reveals that raw variables are less useful than derived combinations of them. If EDA shows that revenue and units sold are both strong predictors of customer lifetime value, then average order value (revenue divided by orders) might be a more informative engineered feature than either alone. If a date column exists, EDA on its distribution might suggest that month of year, day of week, or whether a date falls on a holiday are more predictive than the raw date.
Data Transformation and Normalization
EDA distributions directly inform which transformations are needed. A strongly right-skewed variable benefits from a log transformation before being used in a linear regression, because most regression assumptions require approximately normally distributed residuals. A variable with a very large range compared to other variables may need min-max scaling or standardization before being fed to a distance-based algorithm like k-nearest neighbors or k-means clustering, where scale differences would otherwise dominate the distance calculation.
Encoding Categorical Variables
EDA on categorical variables reveals how many distinct categories exist and how frequent each is. A variable with two categories needs one binary encoding column. A variable with five ordered categories (like education level) might be best encoded ordinally. A variable with 200 possible values might need target encoding or embedding rather than one-hot encoding, which would add 200 columns to the dataset. EDA makes these decisions explicit before they become hidden assumptions inside a preprocessing pipeline.
Common Mistakes in Exploratory Data Analysis
| Mistake | Why It Causes Problems | What to Do Instead |
|---|---|---|
| Skipping visualization | Summary statistics alone can be misleading. Anscombe's quartet demonstrates four datasets with nearly identical means, variances, and correlations that look completely different when plotted. Numbers without charts can hide bimodal distributions, non-linear relationships, and structured outliers. | Always plot distributions and relationships, not just compute statistics. A histogram and scatter plot take seconds to generate and reveal what no summary table can. |
| Ignoring missing data | Many models drop rows with missing values by default, silently reducing the effective sample size. If missingness is not random (for example, high earners tend to skip income questions), the remaining data is a biased sample of the original. | Quantify missingness per column, visualize which rows contain missing values, and decide deliberately whether to drop, impute, or model the missingness as a feature. |
| Misinterpreting correlation | A high Pearson correlation coefficient only measures the strength of the linear relationship. Two variables can have a perfect curved relationship (r near zero on a Pearson test) that a scatter plot would reveal immediately. | Pair correlation coefficients with scatter plots. For non-linear relationships, consider Spearman rank correlation, which captures monotonic associations that Pearson misses. |
| Drawing conclusions too early | EDA generates hypotheses; it does not confirm them. A correlation found during EDA is a signal worth testing, not a finding worth reporting as established fact. | Clearly separate EDA findings from confirmatory analysis. Use hypothesis tests to formally evaluate relationships identified during exploration. |
| Ignoring domain knowledge | Purely data-driven EDA can generate implausible findings that a domain expert would immediately recognize as data errors. A temperature sensor recording -200°C is an error, not a discovery. | Combine statistical findings with subject-matter knowledge. Consult with domain experts when outliers or unexpected patterns appear that statistics alone cannot classify as error or insight. |
| Treating training data the same as test data | Running EDA on the full dataset before splitting it into training and test sets can leak information from the test set into preprocessing decisions, causing overly optimistic model evaluation. | Perform EDA on the training set only. Decisions about imputation values, outlier bounds, and scaling parameters should be derived from training data and applied to the test set. |
Best Tools for Exploratory Data Analysis
| Tool | EDA Capability | Best For | Consideration |
|---|---|---|---|
| Python (Pandas) | Data loading, cleaning, descriptive statistics, groupby aggregations; df.describe() and df.info() as standard starting points | Programmers working with datasets of any size | Requires coding knowledge; most flexible and scalable option |
| Matplotlib / Seaborn | Static charts: histograms, box plots, scatter plots, heatmaps, pair plots; Seaborn adds statistical annotations | Publication-quality static visualizations in Python | Less interactive than Plotly; requires familiarity with axes and figure objects |
| Plotly | Interactive charts that support zooming, hovering, and filtering; Plotly Express provides one-line chart generation | Exploratory visualization where interactivity helps pattern discovery | Larger file sizes for web output; interactive charts need a browser to render |
| R (ggplot2 / tidyverse) | Grammar-of-graphics charting with ggplot2; dplyr for data manipulation; summary() for quick statistics | Statisticians and researchers; the academic standard for many fields | Steeper learning curve than Python for non-statisticians |
| Excel | Pivot tables, built-in chart wizard, conditional formatting for manual pattern detection | Business analysts without programming experience; quick ad-hoc exploration | Breaks down on large datasets; manual workflow is error-prone at scale |
| Tableau / Power BI | Drag-and-drop visual EDA; dashboard creation; automatic aggregation and filtering | Business intelligence professionals; stakeholder-facing exploration | Limited statistical depth; better for communication than analysis |
| ydata-profiling (Python) | Generates a full HTML EDA report from a single line of code: distributions, missing values, correlations, and outlier alerts | Rapid initial assessment of any new dataset | Reports can be slow on very large datasets; detailed analysis still requires manual work |
EDA Checklist
- Understand the data source: Know what was measured, how, by whom, and over what period before touching the data.
- Check structure and data types: Confirm the number of rows and columns, and verify that each column's data type matches its content.
- Audit missing values: Count missing values per column, visualize their pattern, and choose a deliberate handling strategy.
- Remove or investigate duplicates: Search for exact and near-duplicate rows before computing any aggregate statistics.
- Compute descriptive statistics: Mean, median, standard deviation, quartiles, and skewness for numerical variables; value counts for categorical ones.
- Visualize every distribution: Histogram or density plot for each numerical column; bar chart for each categorical column.
- Detect outliers: Apply the IQR method and box plots; investigate each flagged value before deciding to remove or keep it.
- Analyze pairwise relationships: Correlation matrix and heatmap for all numerical pairs; scatter plots for relationships of interest.
- Check for class imbalance: If the dataset has a target variable, compute its class proportions and flag any severe imbalance.
- Document findings: Record every data quality issue found, every transformation decided upon, and every hypothesis generated for formal testing.
Descriptive Statistics Cheat Sheet
| Statistic | Formula / Notation | Interpretation in EDA |
|---|---|---|
| Mean | x̄ = Σxᵢ / n | Center of data; sensitive to outliers; compare with median to assess skew |
| Median | Middle value when sorted | Robust center; use when distribution is skewed |
| Mode | Most frequent value | Most common category; bimodal mode suggests mixed populations |
| Range | Max - Min | Total span of data; highly sensitive to outliers |
| Variance | s² = Σ(xᵢ - x̄)² / (n-1) | Average squared spread; building block for standard deviation |
| Standard Deviation | s = √s² | Typical distance of observations from the mean; same units as data |
| Q1 / Q3 | 25th / 75th percentile | Boundaries of the middle 50% of the data |
| IQR | Q3 - Q1 | Spread of the middle 50%; used for robust outlier detection |
| Skewness | >0: right tail; <0: left tail; =0: symmetric | Indicates whether log transformation may be needed |
| Pearson r | Cov(X,Y) / (σX × σY) | Direction and strength of linear relationship; -1 to 1 |
EDA vs Related Concepts
EDA vs Data Preprocessing
| Aspect | EDA | Data Preprocessing |
|---|---|---|
| Primary purpose | Understand and describe the data as it is | Transform the data into a form suitable for modeling |
| Output | Observations, findings, hypotheses, visualizations | Cleaned, encoded, scaled, model-ready dataset |
| Sequence | Comes first; informs preprocessing decisions | Comes after EDA; implements decisions EDA revealed |
| Reversibility | Does not modify the original dataset | Transforms the dataset, potentially irreversibly |
EDA vs Confirmatory Data Analysis
| Aspect | Exploratory Data Analysis | Confirmatory Data Analysis |
|---|---|---|
| Starting point | Open question: what is in this data? | Specific hypothesis formed in advance |
| Methods | Visualization, summary statistics, pattern detection | Formal hypothesis tests, confidence intervals, p-values |
| Output | New hypotheses and observations | Decisions about whether to reject or retain a hypothesis |
| Risk | Can find spurious patterns through multiple comparisons | Can miss real effects if hypothesis was poorly specified |
| Relationship | EDA generates the hypotheses that CDA then tests | CDA provides rigorous evaluation of EDA findings |
Histogram vs Box Plot
| Feature | Histogram | Box Plot |
|---|---|---|
| Shows distribution shape | Yes — reveals modality, skew, gaps, peaks | Limited — shows quartile positions but not shape detail |
| Shows outliers | Indirectly — extreme bins are visible but not individually labeled | Yes — individual outlier points plotted beyond whiskers |
| Comparing groups | Requires multiple overlaid histograms or subplots | Ideal — side-by-side box plots are compact and readable |
| Sample size sensitivity | Sensitive to bin width choice; misleading for small samples | Reliable from around 20+ observations |
| Best use in EDA | First look at each variable's distribution individually | Comparing a variable across categories or detecting outliers |
EDA Glossary
| Term | Definition | Role in EDA |
|---|---|---|
| Exploratory Data Analysis (EDA) | The practice of examining a dataset through summary statistics and visualizations to understand its structure before formal modeling | The subject of this entire guide; the foundation of the data science workflow |
| Descriptive Statistics | Numerical summaries that describe the basic features of a dataset: mean, median, standard deviation, quartiles, and more | The primary quantitative tools of EDA |
| Data Visualization | The use of charts and graphs to reveal patterns, trends, and relationships in data | The complement to descriptive statistics in EDA; reveals what numbers alone cannot |
| Mean | The arithmetic average of all values; sum divided by count | Center-of-gravity summary; compare with median to assess skew |
| Median | The middle value when observations are sorted from smallest to largest | Robust alternative to the mean for skewed distributions |
| Standard Deviation | The square root of the average squared deviation from the mean; measures typical spread | Primary measure of variability for numerical variables |
| Quartile | One of three values (Q1, Q2, Q3) that divide a sorted dataset into four equal parts | Foundation of the five-number summary and box plot |
| Interquartile Range (IQR) | Q3 minus Q1; the spread of the middle 50% of the data | Standard basis for outlier detection in box plots |
| Outlier | An observation that falls far beyond the typical range of the data | Must be investigated during EDA to determine if it is an error or a real extreme value |
| Skewness | A measure of the asymmetry of a distribution; positive = right tail, negative = left tail | Indicates whether transformation is needed before linear modeling |
| Kurtosis | A measure of the heaviness of a distribution's tails relative to a normal distribution | Signals whether extreme values are more or less common than a normal distribution would predict |
| Correlation Coefficient | A standardized measure (-1 to 1) of the linear relationship between two numerical variables | Central tool for relationship analysis and feature selection in EDA |
| Covariance | The unstandardized measure of how two variables move together; depends on the units of each variable | Raw input for correlation; rarely interpreted directly in EDA |
| Missing Values | Data points that were not recorded; appear as blanks, NaN, or null in datasets | Must be counted, visualized, and handled deliberately before modeling |
| Histogram | A chart that divides a numerical variable into bins and shows the count of observations in each bin | The standard first chart for every numerical variable in EDA |
| Box Plot | A chart displaying Q1, median, Q3, and outlier points as a compact diagram | Standard tool for outlier detection and group comparisons |
| Scatter Plot | A chart that places each observation as a point on a two-axis grid to reveal bivariate relationships | Standard visualization for pairwise correlation analysis in EDA |
| Feature Engineering | The creation of new variables from existing ones to improve model performance | Directly informed by EDA findings about which variables and combinations are predictive |
| Class Imbalance | A condition where one category of a target variable is far more frequent than others | Detected during EDA by inspecting target variable value counts |
| Multicollinearity | A condition where two or more predictor variables are highly correlated with each other | Identified during EDA through the correlation heatmap; affects regression model reliability |
Frequently Asked Questions
Key sources and further reading: Tukey, J.W. (1977). Exploratory Data Analysis. Addison-Wesley — the foundational text that defined the field · Kaggle — Free data visualization and EDA courses with practice datasets · UCI Machine Learning Repository — Open datasets for practicing EDA techniques · OpenML — Community platform for sharing datasets and experiments · Pandas User Guide — Official documentation covering all EDA operations in Python · Seaborn Tutorial — Official guide to statistical data visualization in Python