Uri Simonsohn’s Small Telescopes

I just happened to come across this paper from 2015 that makes an important point very clearly:

It is generally very difficult to prove that something does not exist; it is considerably easier to show that a tool is inadequate for studying that something. With a small-telescopes approach, instead of arriving at the conclusion that a theoretically interesting effect does not seem to exist, we arrive at the conclusion that the original evidence suggesting a theoretically interesting effect exists does not seem to be adequate.

This would’ve been a good thing for us to cite when talking about evidence vs. truth.

To put it another way, a scientific paper typically makes two claims:

1. It’s likely that X is true.

2. The data provide evidence for X.

It can be difficult to argue point 1, one way or another. But we should be able to make progress on point 2. That’s why I rarely go around saying, “X doesn’t work.” Rather, I say that I’ve seen no good evidence for X, and that maybe X works or maybe it doesn’t, and that I suspect that X has positive effects for some people in some settings and negative effects in other cases.

4 thoughts on “Uri Simonsohn’s Small Telescopes

  1. The first claim requires comparing competing explanations. I don’t think the second does. At least not when I read it as “the data is consistent with x”.

  2. There are some differences in expressed awareness of these issues in different areas.

    In clinical research, at least in my circles, statements along these lines were common back to the 1990s “there is currently no good evidence to supports this claim but it may in fact be true”.

    I used to say that all the time but now I regret it, as I feel evidence is far too strong a term. Now I would say the data and model assumptions are (adjective) compatible with there being effects of a given size. And encourage assessments of other sizes.

    • Yep. Sometimes it’s about the effect size, sometimes it’s about existence/non-existence. A 2-sided P-value is a test for the latter, but no good for the former.

  3. Would be very interested in your take on equivalence tests both broadly and as they relate to replication studies. I wish there were more established frameworks for quantifying the strength of evidence for the absence of an effect (of interesting size), but maybe I’m just oblivious. This paper seems to follow a similar line of analysis, but I’m judging off the abstract.

Leave a Reply to Matt Hlavacek Cancel reply

Your email address will not be published. Required fields are marked *