Evan Werfel writes:
I’ve been thinking about the problem of setting an appropriate prior when conducting Bayesian analysis. Do you know if anyone has done any work to quantify how much prior-belief-ness undertaking a study to test a particular hypothesis might represent? Part of me wants to say that given all of the effort to design a study, get IRB approval and collect data, there has to be some lower bound on one’s prior before doing the data analysis… because if it were any lower, presumably one would undertake a different study. At the very least, one imagines that the researcher’s prior is greater than 0. I wonder if cognitive psychological researchers studying hypothesis testing could estimate how much prior belief people need, on average, to take action on their beliefs.
My reply: It’s hard to say. For example, a researcher’s prior belief in the efficacy of a proposed new treatment might be low, but if it could benefit millions of people, then it could be worth studying because of the high potential benefit. Even a treatment with negative expected value can be worth studying: if there is a small probability that it has a consistent positive benefit, then you do the experiment to see whether to proceed further. This is basic decision-analytic reasoning. Conversely, if the prior is that the treatment is probably beneficial, it can still be a good idea to do the experiment, just in case it actually has a negative effect, in which case you’d learn that and know not to proceed.
All that is not even considering issues such as cognitive biases, financial and career incentives, and all sorts of other reasons why we would expect researchers’ priors to be wrong.
Regarding your original question: Sometimes I call this sort of thing “anthropic reasoning” by analogy to the anthropic principle in physics, whereby we can derive some properties of our world, given the information that we exist in it.
Here’s an example from a few years ago where I used anthropic reasoning to answer the question, Should we take measurements at an intermediate design point? I love that paper, and I remain bummed that it’s only been cited 3 times.
It’s only been cited three times that you know of. Chat-GPT has cited it hundreds of times, although it credits the paper to Wansink and Kanazawa (1885).
The “effect size” is only one parameter of the hypothesis (model) being tested. The number of people who actually believe their model like y = A*x + C, is vanishingly small. That A (the effect size) is also exactly zero is essentially never.
The practice amounts to comparing an arbitrary one parameter model (C) to a two parameter model (A and C). Of course it is impossible for the former to perform worse than the latter.
People also do studies because they think an intervention does not work and they are trying to convince others of the same.
Yes I think this is important to consider. I know of people who do replication studies in psychology because they believe the effect will not replicate and want to make a point. I think it would be nice if papers included a brief personal reflection paragraph as an appendix in which each author somewhat candidly explains their motivation for the paper, what they expected going in, and how the results have updated their beliefs.
Kind of the opposite to the current standard of “the authors declare no conflicts interest” statement, which always makes me laugh when we all know publication is not unrelated to career progression in most areas.
Ironically, the closest thing to what you are suggesting that I am aware of is from Brian Wansink of all people (https://www.brianwansink.com/retracted_articles.html). That link probably produces a warning, but seems safe from my tries. While Wansink will forever be notorious to readers of this blog, I have to give him credit for at least trying to explain what he did and what he would do differently (while avoiding the messiness of having messed up ethically).
+1
A huge amount of research in the biological sciences, especially in Universities and research institutes, is done in a spirit of strong investedness in either the hypothesis or the process I would say. Prior-belief-ness is high.
On the other hand, outside this rather traditional approach (“let’s see if we can find something real and interesting”) there are public health analyses, for example, in which the extent of prior-belief-ness may be quite low. The notion that ivermectin might be an effective anti-covid treatment has never been particularly well supported, but since this issue should be sorted properly, several large scale trials have been run (with unsurprisingly negative results: e.g. the Together trial; the PlatCov trial).
On the other, other, hand lots of research is done in a hypothesis-free environment, and in these cases it’s the process that the practitioners are likely invested in. For example, the discovery of the structure of DNA wasn’t really hypothesis-driven – however the scientists were invested in their expectation that the subject was ripe for discovery and in their self-belief in the possibility of finding something interesting. The Human Genome Project was pretty much a hypothesis-free endeavour stimulated by a belief in the value of the process. These studies are often driven by the invention of novel techniques and the results obtained are good for generating new hypotheses.
IMO it’s useful to support hypothesis-free approaches in science since these can broaden the scope of scientific endeavour, and it has to be said that formulating every study in the form of an investigation of a hypothesis can result in a rather stylised and self-limiting approach (“hey, we can plan the paper, with a snappy title, even before we start the work”!).
There’s also a potential problem with the simplistic “all studies are conducted by a single person” model being assumed (leaving aside the possibility that a single person’s priors may change from day to day). What if there are several people involved in a study, and they have different priors?
For example, the 1919 solar-eclipse expedition (to test the idea of gravitational lensing as predicted by Einstein’s new theory of General Relativity) was jointly organized by Arthur Eddington, who was quite keen on GR, and Frank Dyson, who was quite skeptical (but thought it made for an interesting study).
Peter:
You refer to a “simplistic ‘all studies are conducted by a single person’ model being assumed.”
Who is assuming such a model? I don’t see that assumption anywhere in the above post or comment thread.
Andrew:
From the quote from Evan Werfel: “… there has to be some lower bound on one’s prior before doing the data analysis… because if it were any lower, presumably one would undertake a different study. At the very least, one imagines that the researcher’s prior is greater than 0.”
From your part of the post: “a researcher’s prior belief in the efficacy of a proposed new treatment might be low”; “if there is a small probability that it has a consistent positive benefit, then you do the experiment to see whether to proceed further.” [emphasis added]
My point is not that you or Werfel are somehow imagining that all research is literally done by single individuals, but that “the prior belief” is posed as though that were the case: as though there is just one researcher, or one researcher who matters, or that all the researchers in a project’s team (somehow) have the same prior.
Peter:
I was assuming that “a researcher” referred to the research team and that their prior represents a model they have chosen to use. I agree that when researchers disagree, this makes things more complicated. Actually, this is an issue even with a single researcher, as an individual person will have a mix of beliefs!