warningmean-mistaken-for-typical
What it is
A measure of central tendency — usually the mean — is reported as the "typical" value with no regard for the shape of the distribution. For skewed, multimodal, or heavy-tailed (power-law) data the mean can be unrepresentative of most records: it is pulled toward a long tail, lands in a gap between subgroups, or sits above the majority. Audiences then assume a randomly chosen record is close to the mean when it usually isn't.
How to spot it
In code and prose, flag a mean (or "average") reported as typical/representative with no check of distribution shape — no median/mode beside it, no histogram/quantiles/skewness, no spread. Be suspicious when the mean is quoted for income/salary/wealth/followers/sales (commonly power-law), when the maximum is many times the mean, or when the standard deviation approaches or exceeds the mean. In data, compare mean against median (a large gap implies skew) and look for more than one mode.
Pitfall
"The average NFL player makes $1.5M per year" is presented as the typical salary, but salary is power-law: 76.6% of players earn below the mean and one quarterback's cap hit (≈$24M) is 16x the average, so "$1.5M" describes almost no one.
Done well
The figure is reported with its distribution: "median ≈$0.9M; 77% earn below the $1.5M mean, which is inflated by a few very high earners," ideally with a histogram — so readers see the skew instead of assuming a typical ≈$1.5M player.
How to fix it
Before calling any average "typical," inspect the distribution's shape. Report the median (and the mode where relevant) alongside the mean, show the spread or a histogram, and for skewed/multimodal/heavy-tailed data lead with the median or describe the distribution rather than a single central value.
Avoiding Data Pitfalls (Wiley, 2020), Pitfall 4A: Descriptive Debacles
warningmultiple-comparisons
What it is
Many statistical comparisons are run across many variables or subgroups, and the few that come back "significant" are reported while the rest are ignored — with no correction for the number of tests and no replication. With dozens of tests some will cross p < 0.05 by chance alone, so cherry-picking the low p-values manufactures false discoveries (p-hacking).
How to spot it
In code, flag loops or batches that compute p-values across many columns, subgroups, or model specifications, then filter to the significant ones, with no multiple-comparison correction (Bonferroni, Benjamini-Hochberg/FDR) and no holdout/replication. In prose, watch for a single surprising significant result drawn from a large exploratory search ("we tested everything and found X correlates with Y"). Be suspicious when significance is reported without stating how many comparisons were made.
Pitfall
An analyst computes correlations between an outcome and 40 candidate variables, finds two with p < 0.05, and publishes those as discoveries — never mentioning the 38 other tests, or that about two false positives are expected by chance at that threshold.
Done well
The analyst pre-registers the key hypotheses or applies a multiple-comparison correction (e.g. Benjamini-Hochberg) across all 40 tests, reports how many comparisons were made, and treats surprising findings as hypotheses to confirm on fresh data before claiming them.
How to fix it
Account for the number of comparisons: correct p-values (Bonferroni or FDR), pre-register primary hypotheses, and report every test run, not just the significant ones. Treat findings from large exploratory searches as provisional until replicated on independent data.
Avoiding Data Pitfalls (Wiley, 2020), Pitfall 4B: Inferential Infernos
warningnonresponse-bias
What it is
Conclusions are drawn from the subset of a sample that actually responded (or could be reached), as if it represented the whole target population. When non-responders differ systematically from responders, the results are biased — often badly — no matter how large the responding set. This extends the domain's sampling theme; the chapter gestures at survey nonresponse but doesn't formally cover this bias.
How to spot it
In code and prose, flag survey/poll/feedback conclusions generalized to a population without reporting the response rate or comparing responders to non-responders — no mention of who didn't answer, no weighting or adjustment. Be suspicious of customer-satisfaction, NPS, or opinion results with low or unstated response rates, and of treating "everyone who replied" as "everyone."
Pitfall
A company emails a satisfaction survey, 8% reply, 92% of those are happy, and it announces "92% of customers are satisfied" — even though dissatisfied customers may have already churned or ignored the email, so the responders skew positive.
Done well
The company reports the response rate, compares responders to the full customer base on known attributes, weights or adjusts for the difference (or follows up with a sample of non-responders), and frames the result as "92% of the 8% who responded," caveating the generalization.
How to fix it
Always report the response rate and consider how non-responders might differ from responders. Compare responders to the known population, weight or adjust for nonresponse, follow up with a subsample of non-responders, and scope claims to what the responding sample can actually support.
Avoiding Data Pitfalls (Wiley, 2020), Pitfall 4: Statistical Slip-Ups (domain extension — not a specific sub-pitfall example in the book) · Literary Digest 1936 U.S. presidential poll (nonresponse and frame bias); Groves et al., Survey Methodology (2nd ed., Wiley, 2009)
warningnormality-assumption-unchecked
What it is
Tools that assume a normal (Gaussian) distribution — the standard deviation as the headline measure of spread, the 68–95–99.7 rule, Z-scores, control/spec limits, Six Sigma capability — are applied to data that isn't normal. For skewed or power-law data the SD can exceed the mean and "σ from the mean" becomes meaningless, badly mis-estimating how often extreme values occur.
How to spot it
In code, flag use of `std()`, Z-scores, ±kσ thresholds, normal-based control/spec limits, or `scipy.stats.norm` probabilities with no normality check (histogram, Q-Q plot, skewness/kurtosis, Shapiro-Wilk). Be suspicious when SD ≥ mean, when the measure is a known power-law quantity (income, sales, followers, file sizes), or when outliers far exceed what a normal model predicts. In prose, watch for "within N standard deviations" claims about non-normal data.
Pitfall
Analysts summarize NFL salary as mean ± SD and reason in σ, but the SD ($2.25M) is larger than the mean ($1.49M) and the top earner sits >10σ out — the height equivalent of 8'4". A normal model calls that impossible, yet there he is: the data is power-law, not Gaussian.
Done well
The distribution is checked first (histogram/Q-Q plot); recognizing it is power-law, the analysts describe it with quantiles, a log scale, or tail statistics (Pareto share) instead of mean ± SD, and avoid Z-score/σ claims that assume normality.
How to fix it
Verify the distribution before applying normal-theory tools. Use a histogram or Q-Q plot and a normality test; for non-normal data prefer robust or quantile summaries, log transforms, or distribution-appropriate models, and don't quote σ-based probabilities (68-95-99.7, Z-scores, Six Sigma) for data that isn't normal.
Avoiding Data Pitfalls (Wiley, 2020), Pitfall 4A: Descriptive Debacles
warningomitted-uncertainty
What it is
Point estimates — rates, proportions, averages, rankings — are presented with no expression of uncertainty (confidence interval, error bar, margin of error), so readers treat noisy estimates as exact. Without uncertainty, small-sample estimates look as solid as large-sample ones, and differences that fall well within the margin of error are read as real.
How to spot it
In code, flag charts and tables of estimated rates or means built without confidence intervals or error bars — bar charts of proportions with no CI, rankings with no margin, `mean()` reported with no interval — especially when per-cell sample sizes are small or vary widely. In prose, watch for precise-sounding rates or "best/worst" rankings with no margin of error and no mention of sample size.
Pitfall
Mislabeling rates per city are plotted as plain bars and used to crown the "best" and "worst" cities, even though several rest on a dozen or fewer samples per category — so differences that sit within the binomial confidence interval are presented as a definitive ranking.
Done well
Each rate is shown with its binomial confidence interval (error bars), cells with too few samples to support an estimate are dropped or flagged, and only differences whose intervals separate are described as real — so the chart distinguishes signal from noise.
How to fix it
Attach uncertainty to every estimate: confidence intervals or error bars on rates and means, margins of error on rankings. Suppress or caveat estimates from samples too small to support them, and call a difference real only when its intervals don't overlap (or a proper test confirms it).
Avoiding Data Pitfalls (Wiley, 2020), Pitfall 4C: Slippery Sampling
warningp-value-misinterpretation
What it is
The p-value is misread as something it isn't — most commonly as the probability that the null hypothesis is true (or false), or a low p taken as proof of an effect and a high p as proof of no effect. A p-value is only the probability of observing data at least as extreme as seen, assuming the null is true; it proves nothing, and treating it as proof invites Type 1 (false-positive) and Type 2 (false-negative) errors.
How to spot it
In code and prose, flag statements that a low p "proves" the effect, that the null is "X% likely to be true," that a high p means "no effect, proven," or any binary significant/not-significant verdict with no effect size, confidence interval, or power consideration. Watch for `p < 0.05` used as a yes/no proof gate, and "no difference" concluded from an underpowered test.
Pitfall
An experiment returns p = 0.03 and the analyst declares "proven: the null is false and there's only a 3% chance we're wrong," then runs around waving the printout — conflating the p-value with the probability that the null is true.
Done well
The result is stated carefully: "assuming no real effect, data this extreme would occur about 3% of the time; together with the effect size and CI this is suggestive but not proof," and replication or further evidence is sought rather than treating a single p-value as definitive.
How to fix it
Interpret a p-value as evidence about the data under the null, never as the probability the hypothesis is true or as proof. Always pair it with the effect size and a confidence interval, consider statistical power before reading a high p as "no effect," and seek replication rather than a single significant/not-significant verdict.
Avoiding Data Pitfalls (Wiley, 2020), Pitfall 4B: Inferential Infernos
warningregression-to-the-mean
What it is
Extreme measurements tend to be followed by less extreme ones simply because of chance variation, not because of any intervention. When the highest- or lowest-scoring units are selected and then re-measured (or "treated"), their movement toward the average is mistaken for a real effect of the selection or treatment. This extends the domain's sampling-variance theme but is a distinct named phenomenon not covered in the book.
How to spot it
In code and prose, flag before/after comparisons where units were chosen because they were extreme (worst-performing schools, top salespeople, sickest patients) and their later move toward average is credited to an intervention, with no control group. Be suspicious of "the bottom group improved after we acted" or "the top group declined" claims, and of selecting on the same noisy metric used to judge the outcome.
Pitfall
The worst-performing 10% of stores are given a new training program; next quarter their sales rise and the program is declared a success — but the worst performers were partly unlucky and would have drifted back toward average anyway, with or without training.
Done well
The evaluation uses a randomized control group (some low performers get the program, some don't), or models the expected regression, so the program's effect is measured against what would have happened by chance alone — separating real impact from regression to the mean.
How to fix it
When units are selected for being extreme, expect regression toward the mean on re-measurement and don't credit it to an intervention. Use a control group or a proper baseline model, and avoid selecting and evaluating on the same noisy measure.
Avoiding Data Pitfalls (Wiley, 2020), Pitfall 4: Statistical Slip-Ups (domain extension — not a specific sub-pitfall example in the book) · Galton, F. (1886), 'Regression Towards Mediocrity in Hereditary Stature,' J. Anthropological Institute; Kahneman, Thinking, Fast and Slow (2011)
warningsimpsons-paradox
What it is
A trend or comparison that holds in aggregate reverses (or vanishes) when the data is split by a lurking third variable. Because group sizes and composition differ, the pooled numbers can favor one side while every subgroup favors the other. Reporting only the aggregate — or only the subgroups — without checking the other view can flip the conclusion entirely. This extends the domain's composition-confound theme but is a distinct named phenomenon not covered in the book.
How to spot it
In code and prose, flag aggregate comparisons of rates/proportions across two groups where a confounding category is present but not examined — a pooled `GROUP BY` with no breakdown by the lurking variable, especially with unequal composition within it. Be suspicious when one group is weighted differently across subgroups, or when an aggregate result contradicts intuition. Check both the pooled and the subgroup-level comparison; a reversal between them is the signature.
Pitfall
A hospital reports a higher overall death rate than a rival and is judged worse — but it treats far more severe cases. Within every severity tier its death rate is lower; the aggregate reverses only because its case mix is sicker, so the pooled comparison condemns the better hospital.
Done well
The analyst compares the two hospitals within each severity tier (stratifying on the confounder), finds the larger hospital does better in every tier, and reports the stratified result — or a case-mix-adjusted rate — rather than the misleading pooled number.
How to fix it
When comparing aggregate rates across groups, check whether a confounding variable differs in composition between them, and compare within strata as well as in aggregate. If the two views disagree, prefer the stratified or covariate-adjusted comparison and report the confounder explicitly.
Avoiding Data Pitfalls (Wiley, 2020), Pitfall 4: Statistical Slip-Ups (domain extension — not a specific sub-pitfall example in the book) · Simpson, E.H. (1951), 'The Interpretation of Interaction in Contingency Tables,' J. Royal Statistical Society B; Bickel et al. (1975), UC Berkeley admissions, Science
warningsmall-sample-extremes
What it is
Statistics computed from small samples vary far more than those from large samples (the variance of the mean grows as the sample shrinks), so small groups disproportionately occupy the extremes of any ranking, map, or top/bottom-N list. Mistaking this sampling noise for a real signal leads to spurious explanations of why small units are "best" or "worst."
How to spot it
In code and prose, flag rankings, choropleth maps, or top/bottom lists of a rate/average where the units have widely varying sample sizes and no adjustment is made — no minimum-sample filter, no shrinkage/empirical-Bayes smoothing, no funnel plot. Be suspicious when the extreme units are the smallest ones, when both the highest and the lowest are small groups, or when a rate rests on a handful of events. A funnel-shaped scatter of rate versus sample size confirms it.
Pitfall
U.S. counties are ranked by kidney-cancer rate, and the highest- and lowest-rate counties are both small rural ones — prompting contradictory theories (clean rural living vs rural poverty) when in fact tiny populations (sometimes ≤4 cases/year) simply produce volatile rates that swing to both extremes.
Done well
The analyst plots rate against population (revealing the telltale funnel), applies a minimum-sample threshold or shrinks small-county rates toward the overall mean (empirical Bayes), and treats extreme values from tiny samples as likely noise rather than signal before theorizing.
How to fix it
Account for sample size whenever comparing rates across units: visualize rate versus sample size (a funnel plot), set a minimum-sample threshold, or use shrinkage/empirical-Bayes estimates that pull small-sample rates toward the overall average. Before explaining why a small unit is extreme, first rule out chance.
Avoiding Data Pitfalls (Wiley, 2020), Pitfall 4D: Insensitivity to Sample Size
warningstatistical-vs-practical-significance
What it is
A result is statistically significant (low p-value) but the effect is too small to matter, and the significance is reported as if it were importance. With a large enough sample even a trivial difference becomes statistically significant, so a tiny effect can be dressed up as a meaningful finding whenever the magnitude is ignored.
How to spot it
In code and prose, flag significance claims that omit the effect size and its real-world units — a p-value reported with no mean difference, no confidence interval on the magnitude, no comparison to a meaningful threshold. Be especially suspicious with very large samples (where almost anything is significant) and when the headline is "statistically significant" rather than "large enough to act on."
Pitfall
A clinical trial with thousands of patients reports p < 0.0001 for the drug's effect on lifespan and trumpets a breakthrough — but the actual difference is about two days of additional life, far too small to matter clinically.
Done well
The report leads with the effect size and its confidence interval ("≈2 days, 95% CI 1–3 days") and judges it against a clinically meaningful threshold, concluding the effect is statistically detectable but not practically important — keeping significance and importance distinct.
How to fix it
Always report and lead with the effect size (in meaningful units) and its confidence interval, not just the p-value. Decide in advance what magnitude would be practically important, and interpret statistical significance — especially in large samples — separately from real-world significance.
Avoiding Data Pitfalls (Wiley, 2020), Pitfall 4B: Inferential Infernos
warningsurvivorship-bias
What it is
Conclusions are drawn only from the people, items, or records that "survived" some selection process, while those that dropped out are invisible and ignored. Because the failures are missing from the data, the survivors paint a systematically over-optimistic or distorted picture. This extends the domain's sampling/selection theme; it is a classic bias not formally named as a sub-pitfall in the book.
How to spot it
In code and prose, flag analyses restricted to entities still present — "current" customers, "active" accounts, funds "still operating," products "still on the market," planes that "returned" — then generalized to the original population with no accounting for those that exited. Be suspicious of filters that keep only surviving/active/completed records before computing population-level statistics, and of success studies that sample only successes.
Pitfall
Engineers propose reinforcing returning WWII aircraft where they show the most bullet holes, concluding those areas need armor — but the planes hit elsewhere didn't return at all. The holes in survivors mark where a plane can be hit and still make it back, so the unhit areas are the ones that actually need protecting.
Done well
The analysis explicitly accounts for the non-survivors — modeling where downed planes were likely hit, or including churned/closed/failed cases alongside the survivors — so conclusions reflect the full population (armor the areas the survivors were not hit) rather than the survivors alone.
How to fix it
Identify the selection process and ask what was filtered out before the data reached you. Include the non-survivors where possible, or explicitly scope claims to the surviving subpopulation, and never generalize from survivors to the whole without accounting for what's missing.
Avoiding Data Pitfalls (Wiley, 2020), Pitfall 4: Statistical Slip-Ups (domain extension — not a specific sub-pitfall example in the book) · Abraham Wald, Statistical Research Group (1943), analysis of armor placement on returning WWII aircraft
warningunrepresentative-sample-comparison
What it is
Aggregate rates or averages are compared across groups whose samples were collected without a consistent (stratified) plan, so the groups differ in composition as well as in the factor of interest. The comparison is then confounded by the mix of subcategories rather than reflecting a true difference — an apples-to-oranges comparison disguised as apples-to-apples.
How to spot it
In code and prose, flag city/region/segment comparisons of an overall rate where the underlying samples have different compositions (different mix of outlet types, demographics, product lines) and no stratification or standardization is applied. Look for pooled rates compared directly without breaking down by the confounding subcategory, and for sampling that wasn't designed to be representative. Compare each group's subcategory mix; large differences signal the trap.
Pitfall
Overall fish-mislabeling rates are compared across cities to rank "best and worst," but the sample mix differs sharply — sushi (mislabeled >73%) was 16% of Seattle's samples versus 35% of Southern California's, and Boston sampled no sushi at all — so the ranking mostly reflects how much sushi each city sampled, not how honest its sellers are.
Done well
The analyst compares like with like — grocery-to-grocery, sushi-to-sushi — or standardizes each city to a common subcategory mix, and reports only the comparisons the sampling supports, acknowledging that the unstratified overall rates aren't a fair city-to-city comparison.
How to fix it
Compare groups within consistent strata, or directly standardize/weight them to a common composition before comparing aggregate rates. Inspect each group's subcategory mix for imbalance, prefer a stratified sampling plan up front, and draw only the comparisons the sampling design actually supports.
Avoiding Data Pitfalls (Wiley, 2020), Pitfall 4C: Slippery Sampling
warninguntested-significance-claim
What it is
A difference between groups (in means, rates, conversion, and the like) is declared real or meaningful purely because the observed numbers differ, with no test of whether the gap could arise from chance given the sample sizes and variation. Random samples differ even when the underlying groups don't, so an untested difference may be nothing but noise.
How to spot it
In code and prose, flag conclusions that "group A beats B" or "the treatment worked" drawn straight from a difference in sample means or rates with no hypothesis test, confidence interval, or uncertainty estimate — no t-test/ANOVA/chi-square/CI anywhere. Be suspicious of A/B results, before/after comparisons, or leaderboard rankings reported as decisive from raw point estimates alone, especially with small or unstated sample sizes.
Pitfall
A qualification run shows the new component's parts averaging 2% higher on a test than the old, and the team concludes the change "improved performance" — without testing whether a 2% gap is within the normal run-to-run variation for that sample size.
Done well
The team runs an appropriate hypothesis test (or computes confidence intervals) accounting for sample size and variability, and claims a real effect only if the difference is unlikely under chance — otherwise reporting "no significant difference detected," with the observed effect and its uncertainty.
How to fix it
Don't infer a real difference from raw numbers alone. Quantify uncertainty — run a suitable significance test or compute confidence intervals that account for sample size and variation — and report the effect with its uncertainty rather than treating any gap as decisive.
Avoiding Data Pitfalls (Wiley, 2020), Pitfall 4B: Inferential Infernos
Data Pitfalls