warningdata-reality-gap
What it is
Data is never the phenomenon itself — it is a record collected about the phenomenon through a fallible chain of observing, reporting, and recording. The pitfall is forgetting this gap and treating the data as a perfect mirror of reality, so conclusions are stated about the world ("crime fell," "this is where meteorites strike") when the data only supports claims about what was measured and recorded. The gap always exists; the only question is how large it is.
How to spot it
Flag language that equates a measured or reported quantity with the underlying real-world phenomenon — e.g. "crime dropped" instead of "reported crime dropped," or a map of recorded events described as where the events actually occur. Look for counts, maps, or totals presented as complete coverage when they in fact reflect observation/reporting coverage (events are only in the data if someone saw, reported, and faithfully recorded them). Watch for claims that omit who collected the data and how, and for cumulative figures that behave oddly (e.g. counts that can be revised downward) because records are provisional.
Pitfall
"This map of every recorded meteorite impact shows meteorites overwhelmingly strike land near populated regions, not open ocean." The data reflects where strikes were observed and recorded — near developed, populated areas — not where meteorites are actually more likely to fall.
Done well
"This map shows where meteorite strikes have been observed and recorded since antiquity. The concentration over populated regions reflects reporting coverage, not a real tendency for meteorites to avoid the oceans, so we can't read it as a true strike-density map."
How to fix it
Name the measurement process before drawing conclusions: who collected the data, how, and what it could miss. Phrase findings as statements about the recorded data ("reported crime," "counter counts," "recorded strikes") rather than the phenomenon, and explicitly account for the gap when generalizing. Visualizing the data and investigating outliers, shifts, and trends often surfaces where the gap is hiding.
Avoiding Data Pitfalls (Wiley, 2020), Pitfall 1A: The Data-Reality Gap
warninghuman-keyed-rounding
What it is
Values recorded by humans tend to be rounded, fudged, or guesstimated, so they cluster at "round" numbers — multiples of 5 or 10, times reported at the top of the hour or at :05. Treating such data as precise, or reading meaning into the clustering itself, mistakes a fingerprint of the data-entry process for a real pattern in the phenomenon. The same variable measured by an instrument rather than a person typically shows no such heaping.
How to spot it
Inspect distributions at fine resolution for "heaping": conspicuous spikes at values divisible by 5 or 10, time values piling up at :00/:05/:15/:30, or a last-digit distribution that is far from uniform. A histogram that looks smooth at a coarse bin size (e.g. 10) can reveal heaping when re-binned at size 1. Flag any analysis that depends on a precision the human-entry process cannot support, or that interprets the spikes at round values as a real signal.
Pitfall
"Roster weights cluster sharply at 200, 210, and 220 lbs, so teams must set training targets at those round weights." The clustering is rounding by whoever keyed the roster — a 1-lb histogram shows roughly three in four listed weights are divisible by 5.
Done well
"Roster weights show heavy heaping at multiples of 5 and 10 — a fingerprint of human-entered rounding — so we treat them as approximate. By contrast, the precisely measured Combine weights show a near-uniform last-digit distribution, which is what an instrument-based measurement looks like."
How to fix it
Re-examine the distribution at fine resolution (last-digit analysis or 1-unit bins) to expose heaping before trusting precision. Treat heaped values as approximate, avoid over-interpreting spikes at round numbers, and prefer instrument-measured data when the level of precision actually matters to the conclusion.
Avoiding Data Pitfalls (Wiley, 2020), Pitfall 1B: All Too Human Data
warninginductive-overreach
What it is
Generalizing from specific observations in the data — singular statements about what was measured, when, and where — to a universal claim about what is always true, without acknowledging the inductive leap. No quantity of confirming observations proves a universal statement, so treating "it happened in this data" as "it holds in general" overreaches beyond the time, place, and conditions the data actually covers. A common tell is a verb-tense shift from the past ("crossed") to the present or future ("cross," "will cross").
How to spot it
Look for claims that slide from past-tense, scoped observations to present- or future-tense universal generalizations — "X crossed the bridge in April" becoming "X bikes cross the bridge," or "this fund outperformed last year" becoming "this fund is the best investment going forward." Flag conclusions that extend a finding beyond the sampled time, place, or population, and any "past performance predicts future results" reasoning.
Pitfall
"Eastbound bike crossings exceeded westbound in April 2014, so more cyclists cross the bridge heading east." A single month of recorded counts is generalized into an always-true claim — and later data shows westbound is typically higher.
Done well
"Eastbound crossings exceeded westbound in April 2014. That suggests a testable hypothesis — eastbound is usually higher — which I checked against other months and found false: westbound is typically higher, with a summer peak."
How to fix it
Keep singular (observed) and universal (general) statements distinct. State findings in the past tense, scoped to the data at hand. When you do generalize, frame it as an explicit, falsifiable hypothesis and test it against additional data rather than assuming the pattern holds beyond what you measured.
Avoiding Data Pitfalls (Wiley, 2020), Pitfall 1D: The Black Swan Pitfall
warningmeasurement-system-drift
What it is
A time series shows an apparent trend that is partly or wholly an artifact of changes in how the phenomenon was measured — instrumentation sensitivity, coverage, reporting requirements, or definitions — rather than a real change in the phenomenon. Because the data-reality gap itself shrinks or grows over the period, comparisons across time are not apples-to-apples, and a rise in the data can simply mean we got better at detecting or recording.
How to spot it
Examine long time series of counts or rates that span known changes in instrumentation, coverage, reporting rules, or definitions. Be suspicious when a rising trend is concentrated in the hardest-to-detect cases (small events, borderline diagnoses, minor incidents) and coincides with technological or procedural improvements. Flag any claim that reads a long-run trend as real change without controlling for, or even mentioning, detection capability.
Pitfall
"Recorded magnitude-6.0+ earthquakes have climbed steadily since 1900, so the planet is becoming far more seismically active." The increase is concentrated in the smallest, hardest-to-detect band and tracks the expansion of seismic instrumentation, not necessarily any real rise in seismic activity.
Done well
"Recorded magnitude-6.0+ earthquakes have risen since 1900, but the increase is concentrated in the 6.0–6.9 band and coincides with major improvements in seismograph coverage. We therefore cannot conclude that actual seismic activity increased; the trend may largely reflect better detection."
How to fix it
Before reading a time trend as real, ask whether the measurement system changed over the period. Hold the method constant by restricting to a consistently measured subset (e.g. only the largest earthquakes, which were always detectable), annotate known instrumentation or definition changes on the chart, and scope claims to "recorded" counts when detection capability cannot be held constant.
Avoiding Data Pitfalls (Wiley, 2020), Pitfall 1A: The Data-Reality Gap (earthquake example)
warningrater-inconsistency
What it is
Human ratings and categorizations carry measurement noise: different raters disagree (reproducibility), and even the same rater can score an identical item differently on a repeat viewing (repeatability), swayed by context and the order in which items appear. Treating subjective ratings as if they were precise, objective measurements reads real signal into what is partly rater variability.
How to spot it
Flag analyses that lean on subjective human ratings or labels — ripeness, quality, sentiment, severity, star ratings — as though they were exact, repeatable measurements. Be suspicious of single-rater labels with no measure of inter-rater agreement, of small rating differences treated as meaningful, and of rating scales whose categories are vague or undefined. Watch for ignored order or context effects, where the same item is scored differently depending on what preceded it.
Pitfall
"Our panel rated version B a 3.8 and version A a 3.6, so B is the better design." A 0.2 gap on a subjective scale can easily be rater noise; the same panelists often score an identical item differently when it appears in a different position in the sequence.
Done well
"Version B averaged 3.8 versus A's 3.6, but inter-rater spread was wide and a hidden duplicate item was scored differently by a third of the panel. We treat the gap as within noise and collect more ratings before concluding B is better."
How to fix it
Quantify rating reliability before trusting it: use multiple raters and report inter-rater agreement, embed repeated or duplicate items to measure intra-rater consistency, define each rating category operationally, and randomize presentation order to limit context effects. Treat small differences on noisy scales as inconclusive.
Avoiding Data Pitfalls (Wiley, 2020), Pitfall 1C: Inconsistent Ratings
warningunfalsifiable-claim
What it is
Working with a hypothesis that no possible observation could disprove — either because it is framed as a basic existential or untestable claim, or because the analyst shields it from disconfirmation by only ever seeking supporting evidence. A claim that cannot be proven wrong cannot be tested, so corroboration of it teaches us nothing. Genuine progress comes from honest attempts to falsify a claim, not from accumulating evidence that flatters it.
How to spot it
Flag conclusions for which no described observation would count as disconfirming, hypotheses stated so vaguely that any result appears to fit, and analyses that gather only confirming cases while never looking for the data that could refute the claim. Watch for existential framings ("X exists somewhere") presented as data-supported conclusions, and for post-hoc reasoning that explains away every contradicting result.
Pitfall
"Our redesign improves engagement." When metrics fall, the team says the benefit is long-term; when they rise, it is taken as proof. No conceivable outcome could ever count against the claim, so the data can never test it.
Done well
"Our redesign will raise 30-day retention by at least 2 points. If retention is flat or lower in the A/B test, we will treat the hypothesis as disproven and roll the change back." The disconfirming result is defined in advance.
How to fix it
State hypotheses so that a specific, observable result would falsify them, and name that disconfirming result up front. Actively seek the evidence that could prove you wrong rather than only the evidence that confirms you, and treat a failed prediction as a valuable correction rather than a threat to defend against.
Avoiding Data Pitfalls (Wiley, 2020), Pitfall 1E: Falsifiability and the God Pitfall
Data Pitfalls