"What Would Count as Evidence?"
MinutesBefore arguing over who is right, ask what observation would count against the claim, and agree on it in advance; a claim that no possible evidence could count against is not being offered as a claim about the world.
How to do it
- 1Ask the question sincerely and early: "What would we expect to see if this were false?" or "What result would make you less confident?"
- 2Press for something concrete and checkable: a number, a document, an outcome by a date. "If crime rises in the two years after the change, I would take that as evidence against it."
- 3Answer the same question about your own position, first if possible. It is the price of asking, and it keeps the exercise honest.
- 4Agree on the test before looking at the data. Standards chosen afterwards tend to be chosen to fit.
- 5Notice the self-sealing patterns: evidence against the claim is reinterpreted as evidence for it (the absence of proof shows how good the cover-up is), the bar rises each time it is met, or critics are disqualified by the fact of criticizing. Name the pattern once, in neutral terms.
- 6If no answer is forthcoming, change the kind of conversation you are having. You may be dealing with a value, an identity, or a loyalty, which are legitimate things to hold but are not settled by evidence, and it is better to discuss them as what they are.
What to say
- “What would you expect to see if you were wrong about this?”
- “Here is what would change my mind. What would change yours?”
- “Before we look at the numbers, can we agree on what result would count for and against?”
When to use it
- •A disagreement has gone several rounds and each side reads every new fact as support.
- •A claim is framed so that both an outcome and its opposite would confirm it.
- •You are about to commission a test, a pilot, or an investigation and want its result to be able to settle something.
- •Someone "just asking questions" never says what answer would satisfy them.
Counters
Evidence and how strong it is
The idea descends from Popper's criterion of falsifiability (1959, 1963): a theory says something about the world only to the extent that it rules something out. Philosophers of science have since shown that falsification is not a clean demarcation test, because hypotheses are tested in bundles and scientists reasonably protect good theories from a single anomaly (Lakatos 1970), so the question works as a diagnostic, not a verdict. The practice of fixing the test in advance has strong empirical backing from research methods: undisclosed flexibility in choosing outcomes and analyses after seeing data makes it easy to find support for almost anything (Simmons, Nelson & Simonsohn 2011), which is the rationale for preregistration. On the psychological side, asking people what would change their minds is a form of consider-the-opposite, which reduces biased evaluation of evidence in experiments (Lord, Lepper & Preston 1984). Whether posing the question in a live disagreement changes anyone's mind has not been tested. Evidence strength: strong methodological consensus for agreeing on tests in advance; good experimental support for the debiasing logic; no direct test of the conversational move.
- Failure to name a falsifier on the spot is not proof of bad faith. Most people have never been asked, and well-supported beliefs can be hard to imagine refuting (what would disprove that the Earth orbits the Sun?). Give time, and offer examples.
- Not all meaningful claims are empirical. Moral, mathematical, aesthetic, and religious claims are not settled by observation, and demanding evidence for them is a category mistake.
- It can be used as a stalling tactic: demanding impossible standards of proof, or a single decisive experiment where evidence is properly cumulative. The standard you ask for should be one you would accept for your own side.
- Agreeing on a test means being bound by it. Do not ask unless you are prepared to lose.
- Popper, K. R. (1963). Conjectures and Refutations: The Growth of Scientific Knowledge. Routledge & Kegan PaulThe argument that a theory's empirical content lies in what it forbids, and that theories compatible with every outcome explain nothing.
- Lakatos, I. (1970). Falsification and the methodology of scientific research programmes. In I. Lakatos & A. Musgrave (Eds.), Criticism and the Growth of Knowledge. Cambridge University PressThe qualification that theories are not abandoned on a single falsification, which is why the question is a diagnostic and not a verdict.
- Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359-1366The demonstration that choosing standards after seeing data can produce support for almost any claim, the reason for agreeing on the test beforehand.
- Lord, C. G., Lepper, M. R., & Preston, E. (1984). Considering the opposite: A corrective strategy for social judgment. Journal of Personality and Social Psychology, 47(6), 1231-1243Experimental evidence that actively considering contrary outcomes reduces biased evaluation of evidence.