When rereading this post the other day, I noticed the post that came immediately before.
I followed the link and came across the delightful story of a researcher who, after one of his papers was criticized, replied, “We can keep debating this after 11 years, but I’m sure we all have much more pressing things to do (grants? papers? family time? attacking 11-year-old papers by former classmates? guitar practice?)” One of the critics responded with appropriate disdain, writing:
This comment exemplifies the proclivity of some authors to view publication as the encasement of work in a casket, buried deeply so as to never be opened again lest the skeletons inside it escape. But is it really beneficial to science that much of the published literature has become . . . a vast graveyard of undead theories?
I agree. To put it another way: Yes, ha ha ha, let’s spend our time on guitar practice rather than exhuming 11-year-old published articles. Fine—I’ll accept that, as long as you also accept that we should not be citing 11-year-old articles.
As is so often the case, the authors of published work are happy to get unthinking positive publicity and citations, but when anything negative comes in, they pull up the drawbridge.
From the perspective of the ladder of responses to criticism, the above behavior isn’t so bad: they’re not suing their critics or using surrogates to attack them critics or labeling anybody as suicide bombers or East German secret police, they’re just trying to laugh it off. From a scientific perspective, though, it’s still pretty bad to act as there’s something wrong with discussing the flaws of a paper that’s still being cited, just cos it’s a decade old.
Putting together a list
Anyway, this made me think of a fun project, which is to list all the different ways that researchers try to avoid addressing legitimate criticism of their published work.
Here are a few responses we’ve seen. I won’t bother finding the links right now, but if we put together a good list, I can go back and provide references for all of them.
1. The corrections do not affect the main results of the paper. (Always a popular claim, even if the corrections actually do affect the main results of the paper.)
2. The criticism should be dismissed because the critics are obsessive/Stasi/terrorists, etc. (Recall the Javert paradox.)
3. The critics are jealous losers sniping at their betters. Or, if that doesn’t work, the critics are picking on unfortunate young researchers. (I don’t think it does any favors to researchers of any age to exempt their work from criticism.)
4. The criticism is illegitimate if it does not go through the peer-review process. (A hard claim to swallow given how the peer-review process is rigged against criticism of published papers.)
5. Criticism should be a discreet exchange between author and critic, with no public criticism. (But the people who claim to hold that attitude seem to have no problem when their work is cited or praised in a public way.)
The most common response to criticism seems to be to just ignore it entirely and hope it goes away. Unfortunately, that strategy often seems to work very well!
“The most common response to criticism seems to be to just ignore it entirely and hope it goes away. Unfortunately, that strategy often seems to work very well!”
I find this even more distasteful when it is a private communication to the authors. I often email authors privately with questions/corrections/concerns and often there is no response. So, it is not a matter of protecting themselves from public criticism – although that may ultimately be their motivation. What is especially bothersome is the lack of collegiality. Not responding is passive aggressive behavior: it conveys, “you don’t matter, I’m too busy to bother with you, I’m too important to recognize you, it’s already published so I’ve moved on, I really don’t care,…..” Take your pick but I find it appalling behavior.
I tend to think the non-response usually represents a response of confusion and feeling that this will be difficult. People put the query away thinking they’ll get to it later and then don’t.
I guess I tend to lean toward the incompetent rather than malicious.
As a second year grad student I found a significant maths mistake in a 5 year old paper written by one of my field’s superduperstars (NAS, APS, etc). I sent a brief note outlining the mistake, providing a correction, and discussing the implications. He sent back an even briefer note that said, more or less, “Shi*t, you are correct. We screwed up. You should turn this into a longer piece that offers the correction but focuses on implications of the correct equation. It could be good short paper. If you’d like, I’ll read and comment on your first draft.” I wrote the ms. he suggested, sent it to him, received helpful comments, and submitted my revised version to a decent specialty journal. It was my first first-authored pub.
Really cool
Great experience. I’m not sure why you didn’t mention the name of the guy; the story does him nothing but credit. You too.
“I don’t think it does any favors to researchers of any age to exempt their work from criticism.”
Agreed. It’s too bad this view seems to be waning in popularity. Not that it was ever popular, it just seems more popular these days in some academic circles to take the side that the person making even a very mild critique of a student paper is an asshole or otherwise stepping out of line.
I agree that the idea that criticism is valid and valuable “seems to be waning in popularity,” which is unfortunate. This is perhaps “item 6” for the list: criticism itself, like spitting in public, is something that decent people don’t do.
There is the unfortunate practice of refraining from overt criticism to then stab someone in the back (privately, of course). From my experience, this happens too often in academia – rather than take responsibility for criticism, “decency” dictates that nothing bad be said publicly but then the criticism is privately acted upon. I’ve seen people denied tenure this way, students left to fend for themselves rather than actively be mentored, and colleagues dismissed silently rather than confronted. My opinion of many academics (certainly not all, but too many) is that they avoid taking responsibility and avoid direct conflict, but prefer the indirect and anonymous kind.
@Jessica:
But sometimes the person making even a very mild critique of a student paper is, well, maybe not an asshole, but insensitive about their delivery of it.
https://laurakornish.com/2018/09/pecking-order/
Nice link to your own site for the SEO, but *pecking* is the very definition of peer review. The entire point of science as it has been successful is to put forth an idea and have it “pecked” over for faults and — if it survives — be added into the institutional knowledge of mankind and further built upon.
“You didn’t phrase your criticism within a compliment sandwich” is an initial step towards “criticism is bad in general mmmkay.” It’s part of what’s led us into the current mess.
Abigail:
It’s ok for people to link to relevant blog posts in their comments—I wouldn’t call that “SEO”!
I’m not cool – what does SEO stand for? I even searched and could only find Search Engine Optimization but that doesn’t seem to fit.
Dale:
Yes, Search Engine Optimization. I took Abigail’s comment to be suggesting that Laura included her link in order to boost her blog’s ranking for future Google searches.
And we do get lots of spam comments that include irrelevant links. Ummm . . . let me go into the spam folder right now . . . here’s one:
I’ve obscured the url there, but you get the idea. That’s a typical crude example; sometimes we get spam comments where the text of the comment could be legitimately contributing to the blog discussion, but it still features an irrelevant link to an online casino or shopping site or whatever. I think these may be written by real people who are willing to work that hard for the SEO, but I delete these too. On the plus side, WordPress seems to have improved its spam filtering, because now I only seem to get about 100 comments per day going into the spam filter; it used to be thousands. I didn’t need to look at these thousands, as they’d go straight into the spam folder; still, it was mildly bothersome to think that this is happening. At this point, WordPress must be trapping them even earlier in the system.
Other times people have real comments with relevant links, which is what Laura’s comment above looks like. I’m as big a fan of snark as anyone, but I didn’t think Abigail’s “nice link to our own site for the SEO” fit the bill here.
I’ve got a few for the list:
1. “The [critical] paper contains far too many egregious scientific errors and logical fallacies to review here, but let’s look at one […]”
2. “[The criticism is] blind use of [x] without understanding [y]”
3. “Is [the criticism] sheer incompetence or is it politically motivated?”
4. “The [critical] statistical analysis in fact confirms [the main conclusion!]”
And lastly, a little pearl-clutching:
5. “It is more than embarrassing that [my critics have] published this nonsense. It is a scandal. Let’s hope it was not political on the part of [their] institution, but just a bad mistake.”
Matt –
It’s pretty much covered in Andrew’s list but I was expecting to see from you “Why should I respond to your critique, you’re just going to use it to find more problems.” Or, “sometimes you have to downplay the uncertainty to get the message of risk across.”
You might have to turn on your “skeptic” ID badge.
Joshua,
How about you? Can you really just nod along with what Andrew wrote and perhaps shake your head a little bit at those silly social scientists who respond poorly to criticism?
I feel like 1 is acceptable if the fallacy is vital to the criticism and not just incidental. 2 and 4 are also acceptable if properly explicated.
Adede:
Yes, indeed, some criticism really is uninformed!
“Results don’t affect conclusions” is indeed a very popular refrain. I think it says something about how some scientists view the research process.
Here’s another interesting one: “Had the correct panel been included, it would have actually STRENGTHENED our conclusion that HIF1 was successfully inactivated…”
Written by Nobel Prize winner William G. Kaelin. It seems to miss the point of why I write about image errors, I’m usually not terribly concerned about anyone one specific error, more troubled that there is some kind of failure in the process of turning images into papers.
https://pubpeer.com/publications/C1A2B9B2AF0132551E6BCB4D64E7CC
David:
That reminds me of the common statement that a statistically significant result is even stronger if it came from a really noisy study. This is what we call the “What does not kill my statistical significance makes it stronger” fallacy, and it’s a natural, but erroneous, conclusion to take based on a naive interpretation of the “measurement error causes attenuation” result in linear regression. Eric Loken and I discuss the issue further here.
Unfortunately, it’s a very common error for researchers to say that, had their measurements been better, that their results would’ve been even stronger. In addition to the attenuation error discussed above, I suspect there’s an even cruder error that people are making, which is to suppose, had their study been different, that the point estimate would be unchanged, so that the only result of making the study more precise would be to lower the standard error, thus . . . making their result even stronger.
What you see is that these researchers are living within a very comfortable trap in which theories are flexible enough to explain any results, there are enough forking paths to always find something to report, and in which any flaws in a study make its results even stronger. All that, plus media-friendly results and academic colleagues in high places, is pretty much enough for a career!
I will never forget a journal review I did, where I commented on the low internal consistency of a measure (i.e., a Cronbachs alpha of .55 or something). I wanted them to comment on it in the limitations.
In their response, they sent an R simulation demonstrating the spearman’s attenuation formula and argued poor reliability was actually a strength of their results. Defending the position “the worse the reliability of a measure, the better evidence statistically significant results are” is a crazy position to argue, but there it was.
Fortunately, these days, you can just send a link to a good blog post on the matter. :)
Sholto David’s work regarding uncovering publications with dicey photoshopping has been prominently featured in the New York Times recently. For example regarding Columbia University,
https://www.nytimes.com/2024/03/20/science/columbia-cancer-research-retractions.html
When it comes to Harvard, he writes
https://forbetterscience.com/2024/01/02/dana-farberications-at-harvard-university/
which has a delightful pun and a great cartoon.
All in all, his stuff is not to be missed.
I shocked to see a reference to guitar practice, because as a matter of fact, I’d rather be doing guitar practice (literally) than pretty much anything else.
One comment in the context of generally critizing the closer scrutiny of published work is that one should simply trust a researcher when they report a result in a paper. One should not question a published claim or data analysis.
I’ve got you beat in spades (well, in parallelism) on the guitar practice bit. Makes my office a tad noisy, though…
https://pbase.com/davidjl/image/160763279
Cool assembly of you playing all over the place with I think two guitars. It seems we have a lot more in common besides guitar (violin, I used to play it; photography, briefly obsessed with it; and Japan, spent five years there). You have a very nice office, especially for someone presumably living in Tokyo :)
Ah, the convenience of an L-shaped office. You can go off into a corner and practice guitar without disturbing yourself.
Do you play Go? If so, we’re twins.
While the guitars are all different, I admit that the two closest to the camera are seriously similar (except in age: a ’52 Gibson and a 2007 Heritage). But the blonde ones are a folky thing and another jazz archtop, and the electric is an electric.
I’ve added three more since then, so I have lots of things to do other than criticizing old papers. (This is ON TOPIC. Really. It is.)
We inherited a wood-frame earthquake death-trap (of the sort* that made the Kobe earthquake such a disaster (as well as the recent Noto Peninsula one)) and renovated it. Most of the second floor became the office. The ward (oops, “city”) told us we could do either A or B to bring it up to local earthquake standards. We did both.
*: The standard older wood-frame house in Japan has a heavy tile roof and inadequate diagonal bracing. So when an earthquake hits, the bottom moves and the roof doesn’t, so the shear rips the verticals apart. Then the roof crushes everything under it. Ours had a zillion 4×4 verticals** (every 90 cm or so throughout the whole building), so was fundamentally sound. And now has a tin roof (A) and a LOT of diagonal bracing (B).
**: https://pbase.com/davidjl/image/154933843
No unfortunately no Go (but I did Iaido for several decades).
Wow, what an amazing house—in Tokyo? That amount of space I associate with rural Japan. I was a ppor student or a サライマン while living in Japan, so I never could even imagine living in such a space.
We lived in Nishinomiya, and left for India just before the earthquake struck. We were living in a similar death-trap; it was completely destroyed in the earthquake. A student from Africa has taken over the apartment from us and the entire structure collapsed on him, but he was a very fit guy and survived.
“That amount of space I associate with rural Japan.”
Yes. Most units in Tokyo are condos, but single-family homes are common, too. But Japan has a problem with abandoned houses, so fixer-uppers are a dime a dozen. If you are good at darts and aim at the center of the Yamanote line, you’d hit me (that is, I’m smack dab in the middle of dead downtown Tokyo). But there are at least 3 other such houses (one larger, two smaller) in this immediate neighborhood. So if you are willing to renovate, you can, for less than what condos half the size are going for.
[My apologies to everyone else who will find this boring and off-topic. Probably this conversation should be continued in private but I don’t know who David is and how to email him.]
That’s amazing, David. But this renovation looks to me like it would cost somewhere in the vicinity of 10-20 million yen, at a lower bound. And my primary concern would be getting insurance for if an earthquake does strike. My memory is that several people I knew in Nishinomiya just lost their homes, period, and had to start over with housing (although I don’t know whether they had insurance and the insurance just refused to pay). But I focus too much on the worst case in general.
David, Shravan… may I suggest moving this conversation to Mastodon? Not because I think it’s inappropriate here but just because I think Mastodon is a great public venue for general conversations about cultural stuff.
I know Shravan is already there, perhaps David would be interested enough to sign up somewhere. If so, I can recommend @bayes.club as a relevant instance for people interested in Bayes (Aki and Shravan are both already there for example) and also @sdf.org as the instance I use (which is run by a public nonprofit group of computer nerds who have been around since before the invention of the world wide web, so I expect they know how to keep public resources online)
FWIW, I’m at @djl on the MIT Mastadon thing. (I’m new to that sort of thing, though, never having done Facebook or Twitter…)
And I’m also djl who is an alum of MIT which exists in the edu part of the email universe.
(Confession: I couldn’t figure out Shravan’s email from his version of the above disembling.)
Daniel, I am too stupid to figure out how to find David’s mastodon handle on his particular server (in my defence, I am sick right now with the flu). David, I am easy to find, just google me, or click on my home page link. Did you mean it was too hard to unpack this: vasishth squiggle uni hyphen potsdam dot de ? Or did you mean something else? I only put that obscured email address up as a half-hearted entrance test for prospective PhD students (I did have one call me from France, complaining they couldn’t work out my email; didn’t seem like a good fit).
Anyway, my Mastodon handle is: https://bayes.club/@ShravanVasishth
Daniel and I while away many a pleasant hour there complaining about how pretty much everything and everyone sucks. A huge improvement over twitter.
Oh, I missed your email David. I just emailed you.
This post and the comments are spot on. It reminds me that the same thing occurs in classrooms. We’re all capable of making a mistake in class — we might misspeak, we might provide a statistical description that no longer applies (e.g., the % of survey respondents identifying as non-binary), or mangle an equation on the board, to name just a few I’ve done myself. Once in a while, a student will catch it. Rather than admit the error or promise to look into it, I’ve heard plenty of stories of faculty getting defensive, or lashing out that their degree makes it inappropriate for students to question their authority (and recount this lashing out to colleagues expecting sympathy!). Paraphrasing, “I don’t think it does any favors to TEACHERS of any age to exempt their work from criticism.”
+1
The university lecturers I have corrected during lectures have always taken it well when I have corrected them*. One time I approached a lecturer during the ‘five minute break’, a short break he would hold mid-lecture so that students could drink some water or come forward to ask individual questions.
Me (quietly): “Excuse me, you missed a button when you buttoned your shirt this morning…”
Teacher (looks down, buttons up and replies in a thunderous voice so that the last person in the room hears it): “Huh! I’m standing half-naked in front of you and nobody says a word!”
He was my favourite lecturer, and the humour he applied to situations where he made mistakes was admirable.
*Usually I would point out errors in equations.
Yes! In my wife’s huge intro stats classes she offered bonus points to anyone who caught her mistakes. Oddly, none of her colleagues picked up the practice.
Responding to post-publication comments by accusing the commentor of ethics violations and putting them through a time-consuming investigation is a fun little response.
Another common response, (maybe a subtype of #2) “If you are so smart, go ahead and write a better paper instead of bugging me!”
Sometimes, authors do not share their arguments with the reviewer but go behind their back and send them directly to an editor .. like below (a recent case I dealt with).
“The reason for this email is to raise what we think is a more serious concern over one reviewer of the manuscript, who we believe reveals a clear personal conflict of interest in their review, and then the handling of that review by the journal, once it was submitted. We cannot confirm with 100% certainty the identity of this reviewer, but we are very confident that we know the person based on a number of public tweets that they have posted about XXX, the organisation that funded the research described in the manuscript. However, even without this confirmation of their identity, and regardless of whether our suspicions are correct, we believe that the reviewer reveals a clear conflict of interest that spills over into personal animosity towards XXX that should immediately disqualify them from the review process.”
I have seen this argument a lot lately though .. “this reviewer can not say anything about my work, because he or she is biased in opinion.”
Anon:
I’ve seen this in both directions. As you say, sometimes an author will use this tactic to shake off an annoying but relevant review. Other times, as an author I’ve seen reviewers make negative arguments that they do not justify, and sometimes it does seem that the reviewer has a personal ax to grind. Conflicts of interests do exist.
I think a lot of the problem arises from journal publication becoming this high-stakes thing, which can make everything into a political struggle. I recommend that reviewers see their job not as gatekeeping but rather as providing useful information to the journal editors who can then decide what to do with the papers being submitted. I remember a few years ago receiving a paper to review that had some ok stuff but most of which was just terrible. I gave some specific comments, a few months later the editors sent me a revised version that did not address my concerns, this went back and forth one or two more times and finally I just told the editors that if they wanted to publish the article, they should do so—they didn’t need my permission! I did not know any of the people involved, either personally or professionally; it was not a political thing at all, I just felt I’d wasted a bunch of time, that’s all!
Re “ 1. The corrections do not affect the main results of the paper.” I had the publisher (Wiley) take it upon themselves to add that statement to a correction I submitted. That phrase was added immediately after text we wrote saying ‘this was our original conclusion, and here is our changed conclusion that follows our corrected calculations.’ The publisher obviously had that ‘conclusions unaffected’ excuse prewritten as part of their template boilerplate. Begs the question why the publisher would encourage such excuse making.
Excellent post. Please add gaslighting. I was troubled by a private email exchange with a leading scientist where the answer to each of my legitimate criticisms of his C/S/N paper was to tell me I didn’t know what I was talking about, even though he had dipped his toe into my field.
Here’s another, more general-purpose one, from a Response to a Comment Jamie Robins and I published a couple of days ago on major statistical errors in a physics education research paper:
“…a positivistic, narrowly deterministic, and reductive way of interpreting results undermines the complexity and nuance of the social world.”
The paper we commented on is the sole published justification for a $3M education project.
Michael,
Wow—do you have a reference for this one?
Also, it’s funny for them to criticize a statistical critique as being “narrowly deterministic” and undermining “complexity and nuance,” give that a central point of most statistical critiques of published work is that they present claimed effects as universal (thus ignoring complexity and nuance) and that they downplay variation (thus being narrowly deterministic).
For an amusing example of deterministic reasoning in social science, see the article discussed here. Scroll down to the bit on Accepting uncertainty and embracing variation.
Their Response, which includes links to our Comment:
https://journals.aps.org/prper/abstract/10.1103/PhysRevPhysEducRes.20.018002
They do respond to one of our points, about enhanced social desirability bias, by saying that their surveys were pseudonymous, which they forgot to mention in the paper. Also they say that the training video on their site of a teacher urging students who were going to fill out the survey to “show your love for physics” was posted after their research.
On an inappropriate MAR assumption, they say at length that they think it was ok.
But the most amazing thing was their response about failing to use cluster stats rather than individual stats. They say at great length that the problem with cluster stats would be that the error bars are too big! They decline to give what we asked for, a list of results for 13 teachers to allow readers to judge uncertainty for themselves. Maybe there wasn’t space for a list of 13 numbers.
Michael:
Interesting. In your comment, you recommended the data be analyzed to account for clustering. In their response, they write, “the number of teachers in our study does not reasonably enable using cluster methods, as Ref. [1] explains . . .”
I went to Ref. [1], and this was your comment! I think this is the relevant bit from your comment that they’re referring to: “The need for considering cluster effects can be seen in a limiting case. Say that the treatment group consisted of one teacher’s classes at one school and the control group consisted of another teacher’s classes at another school. . . . in this extreme case, treatment effects could not be identified. With multiple teachers, identification is possible but with enhanced uncertainty. With only three teachers in the control group, the actual study [1] was not far from the limiting case.” Your comment continues with more discussion of the challenges of multilevel modeling in this sort of case where the number of groups is low.
The message they seem to take from your discussion is that accounting for clustering is difficult and so the best analysis is to ignore the clustering entirely! They do not seem to realize that the result of this decision is to make the even stronger assumption of zero between-cluster variance.
They also mention that some of your references are to the medical research literature and suggest that such methods will not be appropriate for education research. At best, that reaction of theirs reflects a stunning ignorance of the education statistics literature, given that much of the literature on multilevel analysis comes from education research.
I guess the take-home message is that statistics is hard. Even people with Ph.D.’s in physics can get confused by statistics. Physics is a lot harder than statistics, but that doesn’t mean that in statistics you can just wing it.
It’s disappointing that the authors of this paper didn’t take the opportunity to learn from your thoughtful criticism. I’m reminded of our general discussion of how we can best set up our workflows to learn from our mistakes. The authors of that particular paper have a looooong way to go on this account. Maybe if they continue along their never-learn-from-mistakes path, they’ll reach the heights of worldly success attained by the authors of Nudge.
For you to have put in all that effort, first to read their paper, then to find the problems, then to work with the journal to publish your comment, and then for them to reply with this sort of ignorant doubletalk, that’s gotta be so frustrating. I’d say, “I can’t imagine” how frustrating that would be, but of course it echoes many of my experiences over the years.
Interesting post, with the usual causal inference issues mixed in with group treatment level etc.
The paper we were commenting on at least had a nominal RCT, although oddly vague about how it was randomized. It had another interesting feature. Their initial study of the effect of their revised lessons had no control group! Looking at the submission/revision dates, I guess someone at the journal noticed so the publication was delayed a year.
Michael:
Yeah, also disappointing in your case that the journal editors didn’t step in and add some sanity to the proceedings. Instead they just allowed that rebuttal, which just sits there along with Einstein (1905) and the rest of the physics literature.
Andrew- Thanks. It’s par for the course. It wasn’t all that much work. The disappointment is that the journal now has a Statistics Review Committee that’s supposed to catch this stuff. Some committee members are very smart, but they’re fighting an uphill battle. Mostly losing.
Excellent – will someone collect these and publish a paper.
Economists, accountants and police officers will tell you that if you catch someone red-handed with no possible defence, they will accuse you of lying, producing no evidence to support the claim. And, yes, academics do the same: I can give references. And supporters of the offender and their research programme will repeat the claim, again without evidence.
My recent paper ‘Famine falsehoods and publication ethics: rejoinder to Daoud and the Journal of International Development’ https://mpra.ub.uni-muenchen.de/120082/ documents a case where a Reply, a refutation, documented multiple falsehoods and myths in a paper by Daoud. His Response did not challenge these. Instead, he produced new falsehoods and personal abuse. Nearly all his Response was on a subject not mentioned in the reply : it was a diversion, a distraction from his falsehoods. He says ‘my reply will primarily focus on the core principles of my theory of famines, … That strategy of replying will hopefully be more illuminating for the readership of the Journal of International Development than showing how Bowbrick is misstating historical facts and empirical data.’ [He produces no evidence of me misstating anything.] Readers disagree: my Reply was the Journal’s most read paper for that year. Indeed Daoud cited my 1986 paper ‘A refutation of Sen’s theory of famine’ showing that a Nobel prize winner had adopted a similar strategy when his frauds were exposed. This paper is still being cited, and is still on reading lists at universities around the world 38 years later.
Clearly Daoud’s Response is ethically unacceptable, and contrary to COPE (Committee on Publication Ethics) guidelines. We may ask why The Journal of International Development and Wiley, fully aware of these falsehoods and of COPE chose to publish Daoud’s Response, and to suppress the criticisms. This is arguably an offence under EU and UK law.
Suppression is a powerful form of falsehood. If you suppress enough evidence, the flat-earth theory is clearly right. People believe what they want to, so they ignore inconvenient facts.
AOb One thing missing from the list: the original paper “shows” that the moon is made of green cheese. The critic points to fatal flaws in that analysis. The response is “if you cannot tell us what the moon is, in fact, made of, then our results stand.” There are otherwise reputable journals that actually make this the core of their policy for accepting critical comments.
What puzzles about this response: suppose two people wrote equally-flawed papers at the same time. One concluded the moon was made of paper mache and the other concluded it is made of sponge cake. If critics could not tell us the actual composition of the moon, how would the public know whether it was made of paper mache or sponge cake? Presumably astronauts would want to know whether the moon was edible.
Am:
Indeed. This is something we discuss in our piranha paper.
Consider the people who think that political attitudes and behavior are determined by the outcomes of college football games, the people who think that political attitudes and behavior are determined by hormones, the people who think that voting political attitudes and behavior are determined by shark attacks, the people who think that political attitudes and behavior are determined by subliminal messages, . . . These different groups of researchers should all be at each others’ throats, as at most one or two of these effects can realistically exist with anything like the magnitude and stability implied in their research papers. But, no, they all get along just fine; they’re all in the happy tent of social/evolutionary psychology. It’s kind of like how fundamentalists of different religions can see themselves as allies against the secular world. I can’t really speak to the religious thing—that one’s complicated as it also connects to ethnicity, nationality, social roles, etc.—; for the people who work in the sillier forms of social/evolutionary, it’s hard for me to believe that they really believe everything they are saying.
I keep thinking back to my email exchange several years ago with the psychology professor who was angry at me for dissing the ovulation-and-clothing study (see here for the story). I kept trying to get him to commit to whether he actually believed that women were 3 times more likely to wear red or pink shirts during certain times of the month (jeez, just describing this “finding” reveals the many many layers of forking paths required to get to that conclusion!), and he kept not answering. My impression was, no, he didn’t believe it—for example, if asked to bet on the proposition, he would not lay money on it—but he liked the general area of research and he also liked the general idea of psychology professors getting positive press—so he just avoided thinking too hard about the specifics. I think this is how the authors of flawed papers can see themselves as allies: they focus on the process and on social roles, not on the purported scientific conclusions of the papers.
Another part of that correspondence made a lasting impression on me. You (or someone) had pointed out that the numbers of days they were using for before and after ovulation couldn’t be right (they had omitted the ovulation day or something … whatever). The response was, “This approach is common in the literature.” No substantive defense or admission. At that point, for me, the game — publishing, rather than adding to a store of knowledge — was out in the open.
Kyle,
Oh yes, and also there’s some implicit theory underlying that psychology professor’s mistaken claim.
It goes like this:
1. A study is done and yields a statistically significant result. (Remember, these people don’t understand the idea of forking paths. They think that “p-hacking” is a sort of moral problem, and they are of good morals so they are not “guilty” of it.)
2. An outsider suggests that the measurement of the outcome y or the predictor x is noisy.
3. If the problem is that the outcome measurement is noisy, the argument is that noise in the outcome induces variance but no bias, and so the criticism is that the study was too noisy, but the statistically significant result retroactively shows that the study was just fine all along. The argument of bad measurement is dismissed because of the apparent success of the result.
4. If the problem is that the predictor measurement is noisy (as in the example of the days of ovulation), the argument is that if you estimate an effect with a noisy predictor, this will “attenuate” the coefficient, bringing its estimate closer to zero. Therefore, the argument goes, had the measurement been better, the estimate would’ve been even stronger. The fact that the result showed up in the data, “despite” the bad measurement, is taken to make it even more impressive. As Loken and I discuss, that reasoning is wrong—we call it the “backpack fallacy.” But it’s an insidious argument because it appears to have theoretical and mathematical support.
5. The fallback position is that, ok, sure, we haven’t discovered exactly what we said we discovered, but we discovered something. For example, the result went in in the opposite direction as the preregistered hypothesis, but, no problem, we have a new theory for that, and hey! it’s statistically significant at the 10% level. Or, the dates of ovulation were off, but that just means we discovered something about the pre-ovulation period, and we can come up with a theory about that too!
The key assumptions are: (a) statistical significance is taken to imply generalizable truth, and (b) rejecting straw-man null hypothesis A is taken as evidence in favor of preferred alternative hypothesis B.
I want to write all this up somehow, with some title like, “The folk statistics theories that have been destroying the social sciences.” OK, maybe something pithier than that. Whatever. Putting it in a thousand blog posts isn’t enough. It’s necessary to convey the totalizing nature of the argument, the way in which it can swallow up all objections.
My point here is that it’s not just about people manipulating and playing the game. Yes, they’re playing the game, but it’s a game that they think leads to scientific discovery.
Let me put it this way: I play the game too. I work to get papers published and to get my ideas out there. But I’m playing the game for the ultimate reason that, in my view, this will advance science. I think that people such as that confused and aggressive psychology professor who emailed me thinks that his game-playing is advancing science too. He’s wrong about that, but he’s not wrong that if you’re in a game-like situation, it’s good to know the rules. Science is social.
Excellent post. I have been wondering what response people get after criticizing an article and this made my day :)
I can add the following to the list, which is partly response from authors but also advice that I received.
1. Consider it noise in the literature and it will be corrected with more studies.
2. Don’t criticize a single paper as that can be taken personally.
3. As an additional point to 2 above: collect similar papers and write a response/review in general.