warningconfounding-variable
What it is
A comparison or model attributes an effect to one variable while ignoring a lurking third variable that influences both the supposed cause and the outcome. The uncontrolled confounder produces a real but non-causal association, so the conclusion — and any action based on it — is wrong. Closely related to Simpson's paradox, where a confounder can even reverse the apparent direction of an effect. This extends the domain's analysis-distortion theme but is not a specific sub-pitfall in the book.
How to spot it
In code and prose, flag group comparisons or regressions that estimate an effect with no adjustment for obvious confounders — observational comparisons of outcomes between groups that differ in other ways, a regression of Y on X omitting variables known to affect both, no randomization and no covariate control. Be suspicious when the groups being compared aren't otherwise comparable, or when a plausible common cause goes unmentioned.
Pitfall
A study reports that coffee drinkers have higher rates of a disease and implies coffee is harmful — but coffee drinkers also smoke more, and smoking (the confounder) drives the disease; with smoking uncontrolled, coffee takes the blame.
Done well
The analysis controls for the confounder — stratifying or adjusting for smoking, or randomizing — and finds the coffee association largely disappears, so it reports the adjusted effect and names the confounder rather than the naive association.
How to fix it
Identify plausible common causes before drawing conclusions, and control for them by randomization, stratification, matching, or regression adjustment. Compare only groups that are otherwise comparable, report which confounders were accounted for, and treat unadjusted observational effects as provisional.
Avoiding Data Pitfalls (Wiley, 2020), Pitfall 5: Analytical Aberrations (domain extension — not a specific sub-pitfall example in the book) · Judea Pearl, Causality (2nd ed., Cambridge University Press, 2009); R.A. Fisher, The Design of Experiments (1935)
warningcorrelation-implies-causation
What it is
A causal claim is drawn from a merely correlational observation — "X is associated with Y, therefore X causes Y" — when the association could instead be coincidence, reverse causation (Y drives X), or a confounding third variable driving both. Acting on the supposed cause then fails, because the real mechanism was never established. This extends the domain's analysis-distortion theme but is not a specific sub-pitfall in the book.
How to spot it
In code and prose, flag causal language ("drives," "leads to," "because of," "boosts," "improves") attached to evidence that is only observational/correlational — a correlation coefficient, a regression slope, a before/after with no control — and no experiment, no adjustment for confounders, and no consideration of reverse causation. Be suspicious of recommendations to change X in order to move Y based solely on their historical correlation.
Pitfall
An analysis finds that customers who use feature X churn less and concludes "feature X reduces churn, so push everyone to use it" — but already-engaged customers (who were never going to churn) are simply the ones who try feature X, so the arrow may run the other way.
Done well
The team treats the correlation as a hypothesis and tests it causally — a randomized experiment (some users prompted to use X, some not), or at minimum controlling for engagement and considering reverse causation — before claiming X reduces churn or acting on it.
How to fix it
Don't infer causation from correlation. Establish cause with a randomized experiment where feasible; otherwise control for plausible confounders, weigh reverse causation and coincidence, and use causal-inference methods. State clearly when a relationship is only associational and shouldn't drive interventions.
Avoiding Data Pitfalls (Wiley, 2020), Pitfall 5: Analytical Aberrations (domain extension — not a specific sub-pitfall example in the book) · Judea Pearl, Causality: Models, Reasoning, and Inference (2nd ed., Cambridge University Press, 2009)
warninggoodharts-law
What it is
When a measure becomes a target, it ceases to be a good measure. Once people are rewarded or judged by a metric, they optimize the metric itself — often at the expense of the underlying goal it was meant to proxy — so the metric stops reflecting reality and starts incentivizing gaming. This extends the metric-gaming caution in Pitfall 5E (Moronic Measures), which the book raises but does not develop as its own pitfall.
How to spot it
In code and prose, flag a single metric used as a hard target or incentive (bonuses, rankings, OKRs, a model's objective) with no guard against gaming — no paired or counter-metric, no check that the proxy still tracks the true outcome once people optimize it. Be suspicious when a metric improves sharply while the real goal doesn't, when behavior visibly reshapes around the measure, or when one easily-influenced number drives rewards.
Pitfall
A support team is targeted purely on "tickets closed per day"; agents start closing tickets prematurely and splitting issues to pad the count, so the metric soars while actual customer resolution worsens — the measure, now a target, no longer measures support quality.
Done well
Performance is judged on a balanced set including an outcome metric (issues resolved and staying resolved, customer satisfaction) alongside activity, with counter-metrics that catch gaming, and the metrics are revisited as behavior adapts — so no single number can be gamed at the goal's expense.
How to fix it
Don't manage by a single proxy target. Pair each metric with a counter-metric, anchor incentives to outcome measures rather than easily-gamed activity, monitor whether the proxy still correlates with the true goal once it's targeted, and revise metrics as people adapt to them.
Avoiding Data Pitfalls (Wiley, 2020), Pitfall 5E: Moronic Measures (domain extension — generalizes the metric-gaming caution in the book) · Goodhart, C.A.E. (1975); Marilyn Strathern (1997), 'Improving ratings: audit in the British University system,' European Review
infointuition-analysis-false-dichotomy
What it is
Intuition and analytics are treated as mutually exclusive — either human judgment is discarded wholesale in favor of "letting the data or algorithm decide," or data is dismissed in favor of gut feeling. In reality the two are complementary: human intuition decides which questions matter, what the numbers mean, where to look next, when to stop, and how to communicate, while analysis supplies and quantifies the evidence. Abdicating that judgment to an algorithm (or ignoring evidence entirely) degrades decisions.
How to spot it
This pitfall lives mostly in process descriptions and framing rather than code or data. In prose, flag claims that analytics "replaces" intuition, that one should "collect everything and let the algorithm decide what's important," or conversely that data is irrelevant next to experience. Be suspicious of pipelines that surface patterns with no human step to judge which are meaningful, of metric/feature selection driven purely by an algorithm with no domain rationale, and of decisions presented as "the data says so" with no interpretive judgment.
Pitfall
A team adopts an auto-ML tool and declares "we used to use intuition, now we use analytics," feeding in every available variable and acting on whatever the algorithm flags as correlated — with no domain expert asking whether those patterns are useful or merely spurious.
Done well
The team uses domain intuition to frame the questions worth asking and to vet which discovered patterns are meaningful, then lets the analysis supply and quantify the evidence — combining the two ("intuition guides the analysis, analysis checks the intuition") rather than replacing one with the other.
How to fix it
Treat intuition and analysis as complementary. Keep a human-judgment step in the loop to choose which questions and metrics matter, to interpret results, and to separate useful patterns from noise — and use data to test and refine intuition rather than to abolish it. Document the domain rationale behind what's measured and acted on.
Avoiding Data Pitfalls (Wiley, 2020), Pitfall 5A: The Intuition/Analysis False Dichotomy
warningmisleading-interpolation
What it is
Sparse data points are connected (or averaged) with straight lines, implying a smooth path between them and fabricating intermediate values that were never measured. A low sampling rate — or a two-point slopegraph — can hide dramatic real variation that occurred between the sampled moments, telling a true-but-woefully- incomplete story.
How to spot it
In code and charts, flag visualizations or analyses that connect widely spaced samples as if continuous — a slopegraph or line between just two time points, a series plotted at coarse aggregation (yearly) when finer data exists, linear interpolation/resampling that invents between-sample values. In prose, watch for "increased from A to B" conclusions over a long span with no look at what happened in between. Ask whether the sampling rate is fine enough to capture the variation of interest.
Pitfall
A slopegraph connects each country's life expectancy in 1960 to its value in 2015, concluding "all countries increased" — hiding that Cambodia's fell below 20 in 1977-78, Iraq's stagnated after the mid-1990s, and Iceland's was far noisier than Canada's.
Done well
The full annual series is plotted between the endpoints, revealing the wars, stagnations, and year-to-year noise the two-point view erased — and the sampling rate is chosen fine enough to capture the variation that matters before drawing conclusions.
How to fix it
Choose a sampling/aggregation rate fine enough to capture the variation relevant to the question, and don't treat lines between sparse points as real data. Before concluding from endpoints, inspect the intermediate values, and be explicit that interpolated points are fabricated, not measured.
Avoiding Data Pitfalls (Wiley, 2020), Pitfall 5C: Ill-Advised Interpolations
warningmisleading-performance-metric
What it is
Performance is judged by a metric that doesn't actually capture the outcome that matters — typically an activity metric (how much someone did something) or an opinion metric (how others rated them) standing in for an output metric (what they actually achieved). A weak or absent link between the proxy and true impact produces unfair or backwards rankings, and measuring the wrong thing pushes people to game it.
How to spot it
In code and prose, flag rankings or evaluations built on a proxy metric without checking how well it correlates with the outcome of interest — activity counts (calls made, miles run, lines of code) or rating scores used as the headline measure of performance, with no output/impact metric alongside. Be suspicious when the "worst" by the metric is plainly excellent on results, when the metric's correlation with true impact is weak, or when a single easily-gamed number drives incentives.
Pitfall
A player is labeled the "slowest" and ranked dead last by average speed — an activity metric — even though he leads the league in Player Impact Estimate (output); average speed correlates barely at all with impact (R²=0.056), so the metric ranks one of the best players worst.
Done well
Players are evaluated primarily on an output/impact metric (e.g. PIE), with activity metrics like average speed treated as secondary context (useful for recovery or pacing, not for ranking contribution) — and any proxy is validated against the outcome it's meant to represent before being used to judge.
How to fix it
Measure the outcome that actually matters: prefer output/results metrics over activity or opinion proxies, and validate that any proxy correlates with true impact before ranking on it. Pair metrics so a single number can't be gamed, get stakeholder buy-in, and don't judge performance by a metric just because it's easy to collect.
Avoiding Data Pitfalls (Wiley, 2020), Pitfall 5E: Moronic Measures
warningspurious-correlation
What it is
Two variables correlate strongly by coincidence rather than any real relationship — common when both simply trend over time, or when a strong correlation is cherry-picked from many candidate pairs. Treating such a coincidence as meaningful invents relationships that don't exist and won't hold up. This extends the domain's analysis-distortion theme but is not a specific sub-pitfall in the book.
How to spot it
In code and prose, flag strong correlations reported between variables with no plausible mechanism — especially two time series that both trend (up or down) over the same period, or a high-correlation pair surfaced from a large search of many variables. Be suspicious of correlations presented without de-trending, without a mechanism, or without out-of-sample confirmation. Watch for "r = 0.99 between [unrelated things]."
Pitfall
An analyst notices that US cheese consumption and a particular cause of death both rose over a decade, reports their 0.95 correlation as a finding, and speculates about a link — when both merely trended upward over time with no real connection.
Done well
The analyst de-trends or differences the series before correlating, requires a plausible mechanism, and confirms any candidate relationship on independent data — recognizing that two trending series will correlate by construction and that one strong pair out of many is expected by chance.
How to fix it
Treat a bare correlation as weak evidence. De-trend time series before correlating, demand a plausible mechanism, correct for how many pairs were examined, and confirm on out-of-sample data. A coincidental correlation with no mechanism and no replication is not a finding.
Avoiding Data Pitfalls (Wiley, 2020), Pitfall 5: Analytical Aberrations (domain extension — not a specific sub-pitfall example in the book) · Yule, G.U. (1926), 'Why Do We Sometimes Get Nonsense-Correlations between Time-Series?', J. Royal Statistical Society; Tyler Vigen, Spurious Correlations (Hachette, 2015)
warningunvalidated-clustering
What it is
An unsupervised clustering (k-means, hierarchical, and the like) is run and its output is treated as real, meaningful structure — often with interpretive labels — without validating that the clusters reflect genuine groups rather than an arbitrary partition the algorithm always produces. k-means will split any data into k blobs; the number of clusters and the feature scaling are analyst choices, so a different k or scaling yields different "groups." Naming and reasoning about unvalidated clusters imposes a story the data may not support. This extends the domain's analysis-distortion theme but is not a specific sub-pitfall example in the book.
How to spot it
In code and prose, flag clustering (`KMeans`, `AgglomerativeClustering`, DBSCAN, etc.) whose results are interpreted, labeled, or acted on with no validation step — no silhouette/Davies-Bouldin score, no stability check across seeds or k, no comparison to domain-known groups, and a k chosen by eyeballing an elbow. Be suspicious of hard-coded human-readable names assigned to numeric cluster labels, and of cluster summaries presented as discovered "types" without caveat.
Pitfall
k-means with a hand-picked `k=5` (chosen by glancing at an elbow plot) is run on scaled features, and clusters 0–4 are hard-coded to names like "Western Pacific Shallow" — presenting an arbitrary partition as established seismic zones, with no silhouette score, stability check, or domain validation.
Done well
The analyst validates before interpreting: checks cluster quality (e.g. silhouette), tests stability across seeds and several values of k, and compares the result to known domain structure — then labels and reasons about clusters only to the extent the validation supports, framing them as provisional rather than definitive types.
How to fix it
Treat clusters as hypotheses, not findings, until validated. Justify the number of clusters with more than an eyeballed elbow (silhouette, gap statistic, stability across seeds/k), check sensitivity to feature scaling, and compare to external or domain-known groupings. Don't assign authoritative labels to clusters you haven't shown to be real and stable.
Avoiding Data Pitfalls (Wiley, 2020), Pitfall 5: Analytical Aberrations (domain extension — not a specific sub-pitfall example in the book) · Rousseeuw (1987), 'Silhouettes: a graphical aid to the interpretation and validation of cluster analysis,' J. Computational and Applied Mathematics 20
warningwishful-baseline-forecast
What it is
Forecasts default to a comfortable, "return to normal" baseline regardless of the current trajectory or the variable's known volatility — predicting a steady reversion to a typical level and never anticipating downturns or cycles. Such forecasts say more about what the forecaster hopes (or wants to communicate) than about likely outcomes, and they reliably miss turning points.
How to spot it
In code and data, flag forecasts that flatten to a constant baseline across the whole horizon for a variable known to be cyclical or volatile (unemployment, markets, demand), repeated forecasts that all converge to the same "normal" value regardless of starting conditions, and the absence of scenario/downside modeling. In prose, watch for predictions of "a return to typical levels" with no mechanism, and forecasts that never call a decline.
Pitfall
Each year the OMB forecasts unemployment returning to ~5% within a couple of years no matter the current rate — so the FY2008 forecast shows a smooth glide to 5% just before the 2009 spike, as do forecasts across every administration. The "return to normal" is assumed, not derived.
Done well
The forecast models the variable's cyclicality and uncertainty, presents scenarios (including downturns) with probabilities or bands instead of a single reassuring line, and is honest when the realistic outlook isn't a clean reversion to a comfortable baseline.
How to fix it
Build forecasts from the variable's actual dynamics (cyclicality, volatility, mean-reversion speed) rather than defaulting to a comfortable baseline. Present uncertainty bands and downside scenarios, back-test the forecasting approach against history, and be wary when every forecast conveniently predicts a return to normal.
Avoiding Data Pitfalls (Wiley, 2020), Pitfall 5D: Funky Forecasts
Data Pitfalls