Steve Stigler points to an unusual example of research being used to provide evidence that the null hypothesis is true.
Here’s the research article (“No evidence for magnetic field effects on the behaviour of Drosophila,” by Marco Bassetto, Thomas Reichl, Dmitry Kobylkov, Daniel Kattnig, Michael Winklhofer, P. J. Hore, and Henrik Mouritsen), and here’s the quick summary (“Doubt cast on magnetic sensing in flies,” by Eric Warrant).
From the article:
Under meticulously controlled conditions and with vast sample sizes, we have been unable to find evidence for magnetically sensitive behaviour in Drosophila. Moreover, after reassessment of the statistical approaches and sample sizes used in the studies that we tried to replicate, we suggest that many—if not all—of the original results were false positives. Our findings therefore cast considerable doubt on the existence of magnetic sensing in Drosophila and thus strongly suggest that night-migratory songbirds remain the organism of choice for elucidating the mechanism of light-dependent magnetoreception.
From the summary:
But do the authors definitively debunk the existence of a magnetic sense in Drosophila? Possibly, although there are now at least 15 publications reporting that this sense does exist, with many indicating a Cry-based mechanism. Can all of them be wrong? Again, possibly — and for similar reasons — but this is a serious call to make. . . . Nonetheless, Bassetto et al. have raised a major red flag over the likelihood of Drosophila having the capacity for magnetic sensing. . . .
Here are some interesting things:
1. As can be seen from the abstract of the article, the scientific models here are not just speculations (if we try X, perhaps Y will happen!); they are attempts to explain an empirical puzzle, in this case regarding animals’ navigation.
2. As is typically the case, the null hypothesis of zero effect includes a fuzzy zone of effects that are not exactly zero but show no predictable pattern.
3. With enough data, it should still be possible to reject the statistical null hypothesis, as real data will never match any “specific random number generator”.
4. The summary asks, “do the authors definitively debunk the existence . . .” You can never prove a negative (a concept that seems to have been beyond the capacity of the U.S. judiciary to understand), so I think this question is pretty much pointless.
5. The summary asks whether “at least 15 publications” can be wrong. Hey—check out out the literature on embodied cognition. Or nudging. Hey, Brian Wansink alone had more than 15 publications! I don’t really like the phrase, “this is a serious call to make,” but, sure, future work is warranted etc.
Anyway, this is a good example to point to when we want to talk about a point null hypothesis.
it’s indeed a very serious call to make when you “debunk” science.
Think of all the people who will have been harmed by the loss of their dream jobs and dream livelihoods if the Power Pose turns out to be true! Oh the Humanity!!! Think of all the lost NGO donations and never-to-be think tanks! The Perpetual Renewable Energy Machine, the Fountain of Youth and the Infinite UBI have all been lost becasue “debunkers” just don’t take things seriously enough!
“Debunking” is preventing society from achieving greatness! We’ve got to stop it!
I think we all know what it means in the context of empirical work, but the claim that you “can’t prove a negative” is an irrational peeve of mine. Mathematics does this all the time. Proving that there exist no somethings with such properties is, in principle, straightforward.
1. Suppose there is one of these things
2. …reasoning…
3. Contradiction!
Clearly we can’t make the same kind of absolute statements about a well-defined universe in the real world, but the issue is a practical impossibility, not a logical one.
This is right up there with my annoyance with every time someone says “the plural of anecdote is not data” when the original quote was the opposite and, I would argue, if you think about it, obviously correct.
Magnetic field sensing by organisms is a fascinating topic and, from what I can tell, it’s extremely hard to study. I’m impressed by the paper, and by Nature actually publishing a null result.
The flies may also need to be starved males and exposed to blue light:
https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0211826
Also, for physics folks I have some basic questions:
Seems to me theres energy being released in the parallel condition thats unaccounted for in the antiparallel (cancelled out) condition. How can there possibly not be more heat or sound or light or something emitted?
And shouldn’t they be looking at magnetic potential? Ie the gravitational field (net force) is very small at the center of the Earth, but the potential (energy required to get an object from there to outerspace) is very high. Why isn’t there a similar situation with these cancelling out magnetic fields?
“You can never prove a negative” but the next best thing should be that making experiments and collecting data should provide upper bounds on the phenomenon’s magnitude, and in many cases that should be essentially the same when the bounds get tight enough.
For instance “a photon has no mass” can’t be experimentally proven, but from the wikipedia page physicists have used so called “null” experiments to bound its mass: “If a photon did have non-zero mass, […] Coulomb’s law would be modified […] This provides a means for precision tests of Coulomb’s law. A null result of such an experiment has set a limit of m ≲ 10−14 eV/c2.”
In most cases such a result of “this effect may or may not exists but if it does, it’s smaller that some tiny value” should be a kind of result I’d like too see more often in publications, it’s still scientifically valid and useful knowledge, more so that “there was no significant difference”.
I think the main obstacle to have a better knowledge aggregation of this for something like magnetic effect on flies is that contrary to a photon’s mass, there may not be (as far as I can guess as an outsider) a clear agreed upon quantitative measurement and unit. The value reported in the paper is a “preference index” which I understand is dependent on tons of experimental details and can’t be compared readily across experiments. I guess this encourages debating without end on whether something exists and makes the threshold to make a “negative” claim high, rather than encouraging a more incremental and appeased quantitative progress.
What I find strange about this experiment (and the reason I’m taking time away from my real job (trademarked)) is that 1) it makes no attempt to disprove the causal mechanism of the quantum properties that are ascribed to how birds migrate. 2) There have been a few studies (links below) that show how sensitive this quantum mechanism is to anthropegnic electromagnetic noise.
Therefore, the possibility exists that this experiment is true and, also, that birds do have a quantum mechanism that uses the Earth’s magnetic property to orient themselves when migrating.
The links:
1) A broad, largely non-technical explanation of how birds may have radial pairs that use quantum effects to orient themselves when migrating:
https://www.scientificamerican.com/article/how-migrating-birds-use-quantum-effects-to-navigate/
2) A formal explanation of how this quantum mechanism works:
https://www.nature.com/articles/nature06834.epdf?sharing_token=nhESHo7dVVngNoskRd1vzNRgN0jAjWel9jnR3ZoTv0MYTpUBJx6lFc-6LJ3LtGvCSxXs4_BPL_2_B2oeraZQbCMWXmEfBvhO2PQjyBg0-JJo1PnI8mLmrdFzoSbHc_qmgdvho5dOTnG9ZtIMHxoDFBOIJiITfiD-upRMXsWFFEY_ckAcf1-WY5Savd6IQzCXiJAtNyv5kQEJ9d9G46c-XaC8fN_7xJFb3PfzuIoWG-k%3D&tracking_referrer=www.scientificamerican.com
3) How these radial pairs are easily disrupted by human-made electromagnetic noise:
https://www.nature.com/articles/nature13290.epdf?sharing_token=EfssOZuZ_4Absf-rS2VY-tRgN0jAjWel9jnR3ZoTv0MqMC9EAQ4anSE5cb7jvdtqIbjrL73acUFuMFz2e-Q62H_qy8Av3AutXGm3G067QIWIsWSj6WaVYypHnYtGQj-hWbJA7s69peFjc8-Onscj6elvPoOCFFJ0F0d91jcGeNd5ZDoM94mgXJsq5uf6XipEGSxohgD6u50iKE2LMDCStpNLX3pBro0znct-JKQoM_I%3D&tracking_referrer=www.scientificamerican.com
4) And how its very sensitive to the color of the light:
https://www.nature.com/articles/364525a0.epdf?sharing_token=CZwXFUuL5IaRHM44IBHMHdRgN0jAjWel9jnR3ZoTv0N96sopNBR21wCMDRkEZ1RPtHe27-ajpts5sDTo35sHF3C8hNqUDSVC5klgNXudYTJTe3Rm6RRONilP3VoiE0HhfJH1YolidkHRlBUgpdHEkzqz6jDwWJz_AbK33gKBw-L81Bi0HDJLiL2HnYCk9neE3bw5CYDbxaSA2WI7Y6s3nw4_uLGqlhyB6CPSB9JSu3M%3D&tracking_referrer=www.scientificamerican.com
In short, there could have been paths to casually disprove any of the mechanisms described.
When Andrew says “You can never prove a negative,” either his statement is false or it can’t be proven.
It’s not clear to me what constitutes “a negative.” I can say “4 is not prime” or, equivalently, I can say “4 is a composite number”. One involves a negative, one doesn’t. But then maybe you’ll say “prime” is itself a negation, so “4 is not prime” is a double negation and hence a positive. Outside of mathematics, I can prove “there is nobody 3 meters tall” (equivalently, “everyone is under 3 meters”) by inspecting all the people in the world.
By “You can never prove a negative…,” I assume Andrew is referring to trying to prove that something doesn’t exist.
Since we’re picking nits about this statement, can you prove a positive any more than a negative? Is it any easier to do one than the other? Does it depend on certain conditions?
Let’s assume the research was done without flaw. If false negatives are as equally likely as false positives, is it any harder to prove a negative than a positive?
There can be pure deductive proofs given that the premises are accepted, but once evidence is involved there is no proof or disproof.
You can’t prove anything due to affirming the consequent. Ie, there is always some other explanation (“god/aliens did it”).
https://en.wikipedia.org/wiki/Affirming_the_consequent
You can’t disprove anything due to the Duhem-Quine thesis. Ie, under-determination, there is always some auxiliary assumption to be blamed. Eg, the equipment is malfunctioning (where that “equipment” includes your mind).
https://en.wikipedia.org/wiki/Duhem%E2%80%93Quine_thesis
Yes, I think I agree with all of that.
Regarding the “You can never prove a negative” statement, I think it also relates to the difference between having two (or more) discrete/disjoint possible states, versus a continuum of possible states.
Say I could create an error-free program for drawing some random numbers, that at start up randomly (and unknown to me) selects one of two distributions to draw from, either a U(0,1) or a U(-1,0) distribution. Then I need only see one observation to know (“prove”) which distribution it was set to draw from – because each observation is incompatible with one of the states (except exactly 0, which has zero probability of happening).
However, if it is set to randomly draw from a N(m,1) distribution, where m is selected at random from a N(0,1) distribution at start up, then I can never prove that it was set to draw from, say, a N(0.5,1) distribution, no matter how many observations I have, as I will never have enough information to exclude all other potential states.
Andrew, would it be possible to spell out really explicitly what is meant here by “That’s exactly the point.” I went back and read the old blog post you linked to, but I still don’t get what the precise point is here.
> I really really like the identification of a null hypothesis with a random number generator. That’s exactly the point.
I tried to unpack the comment: Suppose the null is
H_0: \mu = 0
This means that we are assuming under the null that the sample mean is coming from a distribution
Normal(mu=0,sigma/sqrt(n))
where sigma is estimated from the data. Is this Normal distribution the “specific random number generator”?
I didn’t really understand what is meant in the original post you linked to when you say that some other test could have been done. I mean, I do understand that some other test could have been done, but the specific random number generator remains the one above regardless of what test one does. Why is it a problem that the null hypothesis is a specific random number generator, namely Normal(mu=0,sigma/sqrt(n))?
After all, once we do get the data, our best estimate (thinking like a frequentist here) of the underlying distribution is Normal(samplemean, estimatedSE). That is also a specific random number generator, but it seems like a reasonable guess given (asymptotic) results and given that I am tied for the moment into frequentist thinking.
Sharavan:
The “specific random number generator” is at the level of the data-generation process, not the estimation. So in your example the specific random number generator is that the individual data points are generated independently from this particular distribution with a mean of exactly zero. This is not a problem, exactly–it’s a mathematical model, that’s all. The problem comes when the goal is set to reject a model which we know ahead of time is a simplification, and when decisions and inferential summaries are produced in terms of how difficult it is to reject the null hypothesis.
Oh I see now; the problem is that we just set up a way to make a decision to reject a simplified model by computing some summary statistic (like the t-value).
I think what’s confusing for me is invoking a specific random number generator. If I understand this correctly, it is not the critical thing here that we set up a specific random number generator.