Texas Sharpshooter
What it is
Cherry-picking data clusters after the fact to create the appearance of a meaningful pattern — like painting a target around bullet holes.
How it works
Real-world examples
- •A cancer cluster near a factory being attributed to the factory without controlling for dozens of other factors.
- •Investment funds highlighting their best-performing period while ignoring the overall track record.
- •Conspiracy theorists connecting selected events while ignoring the thousands of events that don't fit.
Ethical guidelines
- ●Hypotheses should be formed before examining data, not reverse-engineered from it.
- ●Presenting post-hoc patterns as pre-determined predictions is intellectually fraudulent.
- ●Any pattern found in data must be tested against new, independent data to be validated.
How to defend against it
- ►Ask whether the pattern was predicted in advance or discovered after the fact.
- ►Look for all the data that DOESN'T fit the proposed pattern — not just the data that does.
- ►Demand independent replication before accepting data-mined patterns as real.
From the Defense Playbook
Before you publish, present, or press an argument, go through it as a hostile reader would, looking for the fallacies and statistical shortcuts you would attack in an opponent, and fix or cut what you find.
Before arguing over who is right, ask what observation would count against the claim, and agree on it in advance; a claim that no possible evidence could count against is not being offered as a claim about the world.
Before an A/B test or pilot starts, write down the primary metric, the harm metrics that must not get worse, the sample size or stopping rule, and the planned analysis, so that the result can tell you something you did not already want to hear.
Every playbook entry states how strong its evidence is and when not to use it. Browse the full playbook.
References
- Gilovich, T., Vallone, R., & Tversky, A. (1985). The hot hand in basketball: On the misperception of random sequences. Cognitive Psychology, 17(3), 295–314Shows that people perceive meaningful streaks and clusters in sequences that are statistically random — the entry's premise that given enough data, random clusters will appear and be read as patterns. (Note: later work by Miller and Sanjurjo, 2018, found a bias in the original analysis, so the hot-hand conclusion specifically is contested; the clustering-illusion point stands.)
- Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366Demonstrates that choosing which comparisons to report after seeing the data ('researcher degrees of freedom') can make almost any pattern look significant — the after-the-fact target-drawing the entry describes.