Data PitfallsThe Data Pitfalls checklist
All 78 pitfalls from Avoiding Data Pitfalls by Ben Jones (Wiley, 2019), one line each. Run down the list before you share a chart, a dashboard, a query or a report.
Epistemic Errors
- Mistaking Data for Reality
Data is never the phenomenon itself — it is a record collected about the phenomenon through a fallible chain of observing, reporting, and recording. - Rounding Fingerprints in Human-Keyed Data
Values recorded by humans tend to be rounded, fudged, or guesstimated, so they cluster at "round" numbers — multiples of 5 or 10, times reported at the top of the hour or at :05. - Inductive Leap from Sample to Universal
Generalizing from specific observations in the data — singular statements about what was measured, when, and where — to a universal claim about what is always true, without acknowledging the inductive leap. - Trends Driven by a Changing Measurement System
A time series shows an apparent trend that is partly or wholly an artifact of changes in how the phenomenon was measured — instrumentation sensitivity, coverage, reporting requirements, or definitions — rather than a real change in the phenomenon. - Treating Subjective Ratings as Objective Measures
Human ratings and categorizations carry measurement noise: different raters disagree (reproducibility), and even the same rater can score an identical item differently on a repeat viewing (repeatability), swayed by context and the order in which items appear. - Unfalsifiable Hypothesis (the God Pitfall)
Working with a hypothesis that no possible observation could disprove — either because it is framed as a basic existential or untestable claim, or because the analyst shields it from disconfirmation by only ever seeking supporting evidence.
Technical Trespasses
- Fragmented Categorical Values
A single real-world category is split across many surface forms — misspellings, abbreviations, punctuation, and inconsistent capitalization ("Chevrolet" / "Chevy" / "Cheverolet", "Gray" / "Grey" / "grey") — so grouping or counting the raw field scatters one category across dozens of values. - Character Encoding Corruption (Mojibake)
Text is read with the wrong character encoding — typically a UTF-8 file decoded as Latin-1/Windows-1252 or vice versa — so multi-byte characters turn into garbled sequences ("café" → "café", "naïve" → "naïve"). - Join Fan-Out (Row Multiplication)
A join is performed on a key that is non-unique on one or both sides, so each row matches multiple partners and the result set multiplies. - Join Key Mismatch Drops Rows
Two datasets are joined on a key whose values are represented differently across sources — spelling, punctuation, word order, abbreviations, or an entity present in one source and absent from the other ("St. - Ambiguous Missing-Value Encodings
A single field encodes "missing" in several incompatible ways — an empty string, a literal "?" or "Unknown", and out-of-range numeric sentinels like 0, -1, or 99 — and these are then conflated, counted as real categories, or fed into arithmetic. - Silently Dropping Null Records
Records with null or blank values in a field are filtered out — explicitly, or by a function that skips them by default — before computing a proportion, rate, or average, and the result is then read as representative of the whole. - Silent Type Coercion
A value's data type is implicitly converted on load or during processing, changing its meaning without raising an error — leading zeros stripped when an identifier is read as an integer (ZIP "02134" → 2134), numeric-looking strings auto-cast, dates or booleans inferred from ambiguous text, or large IDs losing precision as floats. - Ambiguous Two-Digit Year Parsing
Two-digit year values (or other truncated date encodings) are stored without century information, then parsed or aggregated as though they were complete, so the century is guessed wrong or the raw 0–99 codes are treated as real years.
Mathematical Miscues
- Division by Zero or Tiny Denominator
A rate, ratio, or percent change is computed with a denominator that is zero or very small, producing an undefined result (division by zero → infinity/NaN) or an unstable, explosive value that swamps comparisons and rankings. - Dropped Missing Categories
A category or period with no records — a true zero — is silently omitted from a chart or table instead of being shown as zero. - Embedded Total Rows
A data set contains pre-aggregated subtotal or "Total" rows mixed in with the detail rows — a "Total" sex alongside Male and Female, or a state-level row alongside its counties. - Floating-Point Accumulation Error
Repeated floating-point arithmetic accumulates small representation errors, so sums of many values, running totals, or money stored as binary floating point drift from the exact result, and exact-equality comparisons (`==`) between computed floats fail unexpectedly. - Integer Division Truncation
Two integer values are divided and the result is silently truncated (floored) to an integer instead of yielding the true fractional quotient, so a rate or ratio that should be, say, 0.3 comes back as 0. - Misreading a Log-Scale Average
Values measured on a logarithmic (or otherwise nonlinear) scale — pH, decibels, earthquake magnitude, star magnitude, log-returns — are averaged arithmetically and the result is read as the average of the underlying physical quantity. - Mismatched Units of Measure
Values expressed in different units of measure are combined, compared, or fed into a calculation without converting to a common unit. - Partial-Period Aggregation
A time series is aggregated into buckets (year, month, week) where the first or — more often — the most recent bucket covers only part of its period, because the data was extracted mid-period. - Percentage Points vs Percent Change
A change between two percentages is described using the wrong kind of difference — reporting the absolute change in percentage points as a relative percent change, or vice versa. - Uneven Aggregation Coverage
Records are grouped into categories that are then compared, but the categories aggregate different amounts of underlying data, so the comparison is not apples-to-apples. - Unweighted Average of Rates
Rates, ratios, or percentages computed over groups with different denominators are combined by summing them or taking a simple (unweighted) arithmetic mean, as if the groups were equal in size.
Statistical Slip-Ups
- Mean Mistaken for Typical
A measure of central tendency — usually the mean — is reported as the "typical" value with no regard for the shape of the distribution. - Uncorrected Multiple Comparisons
Many statistical comparisons are run across many variables or subgroups, and the few that come back "significant" are reported while the rest are ignored — with no correction for the number of tests and no replication. - Nonresponse Bias
Conclusions are drawn from the subset of a sample that actually responded (or could be reached), as if it represented the whole target population. - Unchecked Normality Assumption
Tools that assume a normal (Gaussian) distribution — the standard deviation as the headline measure of spread, the 68–95–99.7 rule, Z-scores, control/spec limits, Six Sigma capability — are applied to data that isn't normal. - Omitted Uncertainty
Point estimates — rates, proportions, averages, rankings — are presented with no expression of uncertainty (confidence interval, error bar, margin of error), so readers treat noisy estimates as exact. - P-Value Misinterpretation
The p-value is misread as something it isn't — most commonly as the probability that the null hypothesis is true (or false), or a low p taken as proof of an effect and a high p as proof of no effect. - Regression to the Mean
Extreme measurements tend to be followed by less extreme ones simply because of chance variation, not because of any intervention. - Simpson's Paradox
A trend or comparison that holds in aggregate reverses (or vanishes) when the data is split by a lurking third variable. - Small-Sample Extremes
Statistics computed from small samples vary far more than those from large samples (the variance of the mean grows as the sample shrinks), so small groups disproportionately occupy the extremes of any ranking, map, or top/bottom-N list. - Statistical vs Practical Significance
A result is statistically significant (low p-value) but the effect is too small to matter, and the significance is reported as if it were importance. - Survivorship Bias
Conclusions are drawn only from the people, items, or records that "survived" some selection process, while those that dropped out are invisible and ignored. - Unrepresentative Sample Comparison
Aggregate rates or averages are compared across groups whose samples were collected without a consistent (stratified) plan, so the groups differ in composition as well as in the factor of interest. - Untested Significance Claim
A difference between groups (in means, rates, conversion, and the like) is declared real or meaningful purely because the observed numbers differ, with no test of whether the gap could arise from chance given the sample sizes and variation.
Analytical Aberrations
- Uncontrolled Confounding Variable
A comparison or model attributes an effect to one variable while ignoring a lurking third variable that influences both the supposed cause and the outcome. - Correlation Implies Causation
A causal claim is drawn from a merely correlational observation — "X is associated with Y, therefore X causes Y" — when the association could instead be coincidence, reverse causation (Y drives X), or a confounding third variable driving both. - Goodhart's Law (Metric Gaming)
When a measure becomes a target, it ceases to be a good measure. - Intuition/Analysis False Dichotomy
Intuition and analytics are treated as mutually exclusive — either human judgment is discarded wholesale in favor of "letting the data or algorithm decide," or data is dismissed in favor of gut feeling. - Misleading Interpolation
Sparse data points are connected (or averaged) with straight lines, implying a smooth path between them and fabricating intermediate values that were never measured. - Misleading Performance Metric
Performance is judged by a metric that doesn't actually capture the outcome that matters — typically an activity metric (how much someone did something) or an opinion metric (how others rated them) standing in for an output metric (what they actually achieved). - Overfit Model Extrapolation
Among several candidate models, the one with the best in-sample fit (highest R², lowest training error) is selected and used to predict, regardless of whether its form makes sense. - Spurious Correlation
Two variables correlate strongly by coincidence rather than any real relationship — common when both simply trend over time, or when a strong correlation is cherry-picked from many candidate pairs. - Trend Extrapolation Beyond the Data
A trend fitted to historical data is projected beyond the observed range as if it would continue unchanged, ignoring saturation points, natural ceilings, asymptotes, and regime changes. - Unvalidated Clustering
An unsupervised clustering (k-means, hierarchical, and the like) is run and its output is treated as real, meaningful structure — often with interpretive labels — without validating that the clusters reflect genuine groups rather than an arbitrary partition the algorithm always produces. - Wishful Baseline Forecast
Forecasts default to a comfortable, "return to normal" baseline regardless of the current trajectory or the variable's known volatility — predicting a steady reversion to a typical level and never anticipating downturns or cycles.
Graphical Gaffes
- 3D Chart Distortion
A chart adds a third dimension or perspective effect — 3D pie, 3D bars, tilted/extruded views — that distorts the very encodings the reader must judge. - Chart Omits Key Context
A chart is accurate as far as it goes but omits something more important — a "sin of omission." A static snapshot can be technically true yet miss the real story (a category's share is growing over time), and presenting data without its known caveats and exclusions (reported vs actual, excluded categories) leaves the audience with a false sense of completeness. - Chart Type Dogmatism
Chart types are judged by blanket rules — "pie charts are bad," "word clouds are evil" — and banished from consideration, rather than evaluated by fit to the objective, audience, and context. - Chart Type Message Mismatch
The chart type doesn't support the comparison the message requires. - Cherry-Picked Chart Window
A chart shows a narrow slice of the data — a short time window or restricted range — that creates an impression the full data would contradict or reverse. - Disproportionate Encoding
The size of a mark — a bar's length, a wedge's angle, a bubble's area — does not match the value it represents under any consistent scale. - Dual-Axis Distortion
A chart plots two series against two independent y-axes with different scales, so their apparent relationship — crossovers, which line is "higher," how closely they track — is an artifact of the arbitrarily chosen scales rather than the data. - EDA Noise Mistaken for Signal
During fast exploratory analysis, a pattern is taken as real without pausing to check whether it's signal or noise. - Encoding Order Mismatch
An encoding choice expresses an order that is not in the data. - Incoherent Axis Scale
An axis's tick labels are internally contradictory, so no consistent scale exists and no reader can map a position to a value. - Inconsistent Scales Across Charts
Several charts that invite comparison — small multiples, a dashboard's panels, a before/after pair — use different axis scales, ranges, or units for the same measure, so equal visual heights or lengths stand for different values. - Insufficient Precision for Comparison
A chart is asked to convey a fine distinction (one value just edges out another) using an encoding that can't show it precisely. - Optimize/Satisfice False Dichotomy
Visualization choice is framed as either finding the one "best" chart or settling for any "good enough" one — and success is often judged on comprehensibility alone. - Overloaded Confusing Chart
A chart crams in so much that the audience can't extract the intended message — too many overlapping series ("spaghetti"), a color palette recycled across more categories than it has colors, and the series of interest left in a low-salience default. - Rainbow Color Scale
A rainbow (jet) color scale is used to encode an ordered, continuous variable. - Task-Mismatched Visualization
A chart is technically fine but doesn't help the user accomplish their actual task, because the wrong variable is mapped to an axis (or the wrong encoding/aggregation is chosen) for the question at hand. - Truncated Y-Axis
A bar or column chart's value axis doesn't start at zero, so the bar lengths no longer encode the values proportionally. - Unevenly Spaced Axis
A quantitative axis gives each labeled tick an equal-width slot even though the numeric intervals between labels are unequal, so position along the axis no longer encodes the value proportionally.
Design Dangers
- Colorblind-Unsafe Palette
A color encoding relies on hue distinctions that color-vision-deficient viewers can't perceive — most notoriously red-green, which is indistinguishable to the most common forms of colorblindness (roughly 1 in 12 men). - Design-Induced User Error
When users misread or misuse a visualization, the design blames the user instead of treating the error as a design flaw to fix. - Inaccessible Contrast
Text or marks are rendered with too little contrast against their background — light gray on white, thin colored text, faint labels — so they're hard or impossible to read for users with low vision, on poor displays, or in bright light. - Inconsistent Color Encoding
Across a multi-view dashboard, the same color (or the same hue at a given saturation) is used to mean different things — encoding one variable in one chart and an unrelated variable in another, or serving as two separate sequential scales for different measures. - Inconsistent Encoding Across Charts
Across a set of charts — a dashboard, a slide sequence, a report's figures — the same visual channel means different things, or the same thing is shown different ways. - Inconsistent Formatting
Numbers, dates, units, labels, and styling are formatted inconsistently across a chart or dashboard — mixed date formats, varying decimal places, currency sometimes shown and sometimes not, different capitalization or terminology for the same thing. - Mobile Responsiveness Neglect
A visualization or dashboard is designed only for a large desktop screen and breaks on the small screens where much of the audience actually views it — tiny unreadable labels, controls that fall off-screen or can't be tapped, horizontal scrolling, a dense multi-panel layout crammed onto a phone. - Omitted Aesthetic Opportunity
A chart is clear and correct but purely utilitarian, missing the chance to engage the audience and evoke the subject — a "sin of omission" on the aesthetic side. - Too Many Color Encodings
A single dashboard carries multiple independent color schemes at once — several legends, conflicting red-green scales, categorical hues layered on top of sequential ones — so the viewer has to keep track of what color means in each view and quickly loses the thread. - Unclear Interaction Affordances
An interactive visualization doesn't make it discoverable what actions are possible or understandable what the controls and marks mean, and its controls aren't naturally mapped to the views they affect.
Biased Baseline
- The Unheard Voice
Whose voices, sources, and contributions are represented in the data work — and whose are missing — goes unexamined, so perspectives that have historically been undervalued or ignored are left out, and their contributions are under-credited or misattributed.