(an issue raised by a recent failed-science story:) The distinction between research that could be valid but just happens to be wrong, and junk science that never had a chance

Greg Meyer writes:

An item perhaps of interest in regard to post-publication review, in this case in math/logic. A paper claimimg to prove a theorem posted posted online on 25 October in Studia Logica had its proof refuted in on 28 October in an online discussion at MathOverflow by David Roberts. The editor “retracted” the paper on 30 October (along with an earlier paper by the same author; I’m not sure what the issue was with the earlier paper). (I’ve put “retracted” in quotes because it’s a print journal, and the refuted proof did not, and will not, appear in print, so it’s a retraction of a not-yet-fully-published paper.)

One of the commenters at MathOverflow, Alec Rhea, wrote, “MathOverflow seems to be taking on a role as the final stage of review.” I thought this might be an interesting case study in extended peer review.

This is indeed interesting to me. Not the math part—I could care less about the twin primes conjecture—and not even the retraction part, but something else.

There are two kinds of retractions. In the first kind, some research is done that could’ve worked, but it turns out that something was done wrong, and it didn’t really work out. In the second kind, the research never even had a chance of being correct, and the closer you look, the more you realize that the claims were bogus: not just wrong, but lacking whatever it might take to possibly work.

This doesn’t have to do with fraud. It can be simple incompetence. Or, to put it more charitably, lack of understanding of sophisticated ideas. An example is that beauty-and-sex-ratio research we discussed many years ago. Big claims, published in a legit biology journal and hyped by legit media outlets, but it never had a chance to work. The researcher who did this was, I assume, naively under the impression that statistical significance implied a good signal-to-noise ratio. He was completely wrong, just as the people making claims about elections and longevity were completely wrong—not necessarily wrong on the directions of their substantive claims (who knows?) but wrong in their belief that they’d discovered or proved or found good evidence for their claims from their data.

One characteristic of these never-had-a-chance research projects is that they can seem reasonable to casual readers, while experts can easily see that the work is hopeless.

And that brings us to this math problem. As Meyer wrote, someone claimed to prove a theorem and it turns out that he didn’t. In this case, the accurate framing of the story is not, “It looked like the Twin Prime Conjecture might have been proven but it turned out the proof was flawed.” Rather, it’s “Someone who never had a chance of proving the Twin Prime Conjecture deluded himself and some reviewers into thinking he would be able to do it.” If you follow the thread, you’ll see that the author of that paper never had a chance, any more than I’d have a chance to row a boat across the Atlantic.

I was reminded of this example from a couple of years ago of someone claiming to prove that sqrt(2) is a normal number. Again, dude never had a chance.

This is different, for example, from that disgraced primatologist. He misrepresented his data, but he was qualified, right? He could possibly have learned something important about monkeys in his experiments; he just didn’t. It just struck me that when writing about replication failures or science failures we should distinguish between the two scenarios. The distinction is not sharp, but I think the general point is relevant.

11 thoughts on “(an issue raised by a recent failed-science story:) The distinction between research that could be valid but just happens to be wrong, and junk science that never had a chance

  1. The distinction doesn’t seem so clear to me. In almost any observational study (and even small RCTs), the list of potential confounders is large. Couldn’t you always say that these studies were doomed from the beginning? At what point do you have sufficient data on relevant variables to say a study “could have worked” rather than lacked “whatever it might take to possibly work?” I feel like the dividing line may have more to do with what your prior beliefs are rather than anything intrinsic to a particular study.

    Put another way, couldn’t you say that any of the disgraced studies you have highlighted (ESP, power pose, election losses and longevity, etc.) “might have worked?” If the results had been strong enough might they have still worked, despite their limitations? “Strong enough” is a slippery concept, since it is easy to say that some of these studies were so noisy that no evidence could have been “strong enough,” but strong enough for what? Those studies would certainly not have been enough on their own to have discovered convincingly anything, but perhaps they would have found enough to be considered worthwhile evidence.

    I’m not questioning the value of these studies, only that the distinction you propose is not so clear to me.

    • Couldn’t you always say that these studies were doomed from the beginning? At what point do you have sufficient data on relevant variables to say a study “could have worked” rather than lacked “whatever it might take to possibly work?”

      Yes, the vast majority of research being published today is doomed to fail at answering the question being asked.

      That doesn’t mean nothing of value can be salvaged, but it is typically incidental to the (stated) goals of the researchers.

      “Sufficient data” is determined by theory and experience. Once you know enough to reliably predict/reproduce some phenomenon and communicate how to others, then you know how much of various data are required.

  2. In mathematics there’s another category – promising approaches that stand a chance but don’t work out.

    Years ago Marguerite Lehr , a colleague of mine at Bryn Mawr College, told me of a conversation she’d had years before that with Oscar Zariski, a brilliant algebraic geometer then at Johns Hopkins. She told him about a failed attempt to solve a particular problem. He said “you must publish this.” She asked why, since it had failed. He replied that it was a natural way to attack the problem and people should know that it wouldn’t work.

    https://en.wikipedia.org/wiki/Marguerite_Lehr

    PS I am surprised too that Andrew “could care less about the twin primes conjecture”.

  3. I think it’s refreshing that Andrew doesn’t care about the twin primes conjecture. After thinking about it for a few seconds, I realized that I also don’t care about the twin primes conjecture.

    It’s kind of interesting to think about what sorts of unanswered questions you actually care about. “Is there life on other planets?” Definitely. “What does Quantum Mechanics mean?” Totally. Twin primes, meh …

    Probably the intellectual world could do with a little less pure math worship …

    (Apologies if this starts a flame war.)

    • Chacun a son gout. No flame war (at least from me). I don’t think the intellectual world as whole worships pure math. But Andrew did mathematics before he did what he does now and I thought some of that curiosity would remain.

  4. The distinction is not pedagogical. To put it charitably, the second category comes out of hubris. There is so much alchemical thinking in statistics, and eclecticism in science more generally that allows even those who “don’t stand a chance” to try their creative, not matter how stupid or incompetent, ways. For the sake of creativity along, such efforts should be rewarded. No waste of time from a pedagogical viewpoint, the thousandth time might stick.

  5. I find it interesting that someone can care about whether sqrt(2) is a normal number but not care about the twin primes conjecture. It can’t be a pure vs applied thing, both of them seem equally devoid of real-world applications (unless I am missing something).

    • Not believing in irrational numbers is such classical crankery that the ancient Greeks were in to it; there were stories about Pythagoreans murdering someone who proved that sqrt(2) is not a ratio of two integers. People who reject irrational numbers tend to reject whole swathes of mathematics which are both fun and useful. As far as I know, the proposition “there are an infinite number of primes p such that p+2 is also prime” does not have wide-reaching implications for your ability to do math. I didn’t remember that proposition and it is fun for me!

      • Unless I’ve misread your comment, “normal” != “rational” – as I understand it a normal number is an irrational number whose decimal expansion has the digits 0-9 appearing equally often (and analogous property hold for expansions in other bases). Apparently this has some importance in computer science, but beyond that I’m not sure.

        As for not believing in irrationals, I recall coming across some videos by a mathematics professor in Australia who seemed very keen to avoid their use at all, maybe through some kind of ultra-constructivist ethos.

  6. Andrew states, “I could care less about the twin primes conjecture.” BUT, should he have used “I could NOT care less about the twin primes conjecture”?? I found this web site
    ——————————————————–
    https://www.grammarbook.com/blog/definitions/could-or-couldnt-care-less/

    “When someone uses either version of this expression, they are applying sarcasm to convey they have no concern or preference. Therefore, the technically correct version is couldn’t care less.

    It is easy to see why this has to be so when you further examine the thought. Suppose Jim has been asked what movie he would like to see. If he responds “I could not care any less than I already do,” he would be lengthy but clear. On the other hand, if he were to reply with “I have fewer cares,” the meaning would be lost.

    In this way, saying that you “couldn’t care less” is like saying you care “less than nothing.” It’s an obvious exaggeration, but the meaning and emphasis are straightforward.”
    ——————————————————
    The website concludes with the suggestion that either phrase is to be avoided in serious discussions and relegated to “the occasional blog post”:

    ################################################
    “Why We Could Care Less About This Rule

    Hopefully the explanation above clarifies why “couldn’t care less” is better than “I could care less” in conveying the same thought. If you keep using the non-preferred phrase, though, you will probably be in good company.

    The two versions of the given statement have been used so often that both are well understood. In addition, because they are based on informal slang and sarcasm, neither expression will be found in serious reporting or academic work. That means you are more apt to find these constructions in everyday speech and the occasional blog post.”
    #################################

    For more hairsplitting on the subject,

    https://www.merriam-webster.com/words-at-play/could-couldnt-care-less:

    “Both could and couldn’t care less are informal, and so you are unlikely to use either one in formal writing. If you have need of using it in some other context, and would like to avoid alienating some portion of your audience you should stick with couldn’t care less. And if you can’t get past some people continuing to use could care less, and the fact that there’s nothing you can do about it, you may console yourself with the notion that at least they are not saying ‘I could care fewer.’ “

Leave a Reply

Your email address will not be published. Required fields are marked *