“Why I blog about apparent problems in science”

Nick Brown writes:

In this post I want to discuss why I blog directly about what I see as errors or other problems in scientific articles. . . . I have seen criticism of the “blog first” approach because it “drops stuff on the authors out of the blue” or “doesn’t give them a right to defend themselves”. People have suggested that it would be better to approach the authors first and discuss the problems. That seems obvious, and it was how I used to approach things too, but over time I have changed my mind, for a couple of reasons.

People ask me this question too! So I’m curious to see Nick’s take on it. Here’s what he says:

First, I believe that, in principle, science should be conducted with radical transparency. . . . The whole reason for posting a preprint is to allow direct feedback on it, which anyone can take part in. . . .

The second reason is much more personal. If I write to an author and say “Umm, I think I’ve found these problems in your article”, it feels to me as if I’m entering into a process of negotiation with them. I worry that maybe it feels to them like there is something they can say or do that will persuade me not to share what I’ve found. . . . I really hate that feeling. Some people seem to have no problem with that kind of implicit, low-level conflict, but it really doesn’t sit well with me.

I feel the same way. It’s kind of weird. In a way you could say that Nick or I or other data thugs are very confrontational: we go around criticizing people’s published work, and that makes people feel bad! Sometimes we even make jokes while we’re doing it. But I agree with Nick: to me, an intellectual confrontation does not feel like a personal confrontation, and I too find personal conflict to be unpleasant.

Many channels for promotion, many channels for criticism

I’m not saying that was Nick does, or what I do, is “the right way” to do scientific criticism. I don’t think there is a right way. Different approaches will work for different purposes.

To put it another way, consider all the different ways to positively promote research. You can quietly publish journal articles and books, you can promote them on blogs and twitter, you can send personal emails to people, you can do interviews and make claims like, “we think we’ve discovered a new continent,” you can contact well-known scholars and hope they’ll write blurbs like, “A genuinely new idea,” “unique and novel insights,” “a previously unknown . . . problem as large as an elephant,” etc.

All these are legitimate promotional strategies. And, believe me, when I have something to promote, I’ll push every button I can.

So here’s my point. Just as there are many channels for promotion, and that’s a good thing, it’s also a good thing that there are many challenges for mixed or negative criticism. It would be silly to say that we should only promote our work through journal articles, or books, or op-eds, or personal communications with colleagues, or twitter, etc. And, similarly, I bristle at a suggestion that critics should only operate through internal peer review, or through personal communications, etc. Let’s use all the channels we have, and let’s do what we’re best at. Nick is excellent at open-source detective work; he’s the I. F. Stone of research evaluation. So that’s what he should do. Other people can work well within institutional settings as journal editors etc., and that’s good too.

And sometimes the positive and negative come in the same package, as when a book is described as “a humbling lesson in inaccuracy”—and this ends up as a promotional quote!

36 thoughts on ““Why I blog about apparent problems in science”

  1. Thanks Andrew and Nick for taking on the challenge of fostering high quality discussions b/c there are so few social media formats that are all that successful. Twitter format has some advantages in that we can witness audiences responses to expert opinions. The responses do seem to follow principles of social networks dynamics and convey some of the common cognitive and analytic biases that have been amply elaborated in decision literature.

    Coincidentally, I tweeted the following post this morning.

    https://twitter.com/JadePinkSameera/status/1400080997346123781

    It aligns with the subject matter of Andrew’s post. In fact, even earlier Nassim Taleb tweeted the following. I can’t express to you how strongly I disagree with Nassim’s take.

    https://twitter.com/nntaleb/status/1400076853935841284

    I disagree with it b/c such a stance ends up shutting off the query over which there is stark disagreement. In fact, one’s adversaries in debate may be using emotional blackmail to really get under your skin. People can be very clever in distracting us from substantive discussion. I think that is what occurred between Serge Lang and those he critiqued. Basically lost his cool sometimes. And that just furthers the rifts among experts.

    So I welcome any fora that can carry on with great humor, satire, and substantive argumentation. I have real appreciation for debate. My Dad won several All-India debates before partition. However, he did not like it when I debated him. LOL

    Sander Greenland and Andrew Gelman have strong debating skills. And I get goosbumps when someone displays strong debate skills. Better than dating cool looking guys. JUST KIDDING. Well maybe.

    I have to say that the COVID debates on Twitter have been a bit disappointing. I’m glad that Vinay Prasad is trying to keep the conversation going.

        • Sameera:

          Yeah, I’m down on the whole concept of “enemies” in this setting. For example, I’ve thrown a lot of criticism in the direction of Sunstein, but I don’t consider him an “enemy” and I hope he doesn’t think I’m his enemy either. I’d like him to stop making some of the mistakes he makes, and he doesn’t seem very interested in avoiding these errors, so we’re in opposition, but I don’t think it makes sense to think of us as enemies.

        • It does strike me that certain people cross a line which you’re okay with throwing out a personal insult or two. Is that something you think about–the point at which you move from criticizing a book or article or claim and feel comfortable inferring things about its author?

        • Somebody:

          I’m pretty much always comfortable inferring things about authors—as long as I’m clear that I’m inferring or speculating. We had a discussion once on this blog where someone said I should stick entirely to the facts and not speculate about authors’ motivations, but I don’t see why not speculate. I agree that it’s not good to speculate without labeling the speculations as such. This is related to “story time” in scientific articles. I have no problem with story time as long as it’s clear where the data end and the story begins.

        • Hi Andrew,

          I wasn’t thinking of your attitude toward Cass Sunstein or your critiques of his errors. Your blog hasn’t the feel of Twitter either. I was hoping that Cass Sunstein would respond to you on this blog. At least Daniel Kahneman did.

          Twitter is the Wild Wild West.

          Now and then, Nassim Taleb will tweet his disdain or angst about a specific academic. And one prominent academic blocked Nassim. It seemed that Nassim didn’t expect that even though Nassim has blocked some who have criticized Nassim’s own viewpoint.

        • Sameera:

          Given that Sunstein’s on record as labeling critics of published research as “Stasi,” and given that he doesn’t generally seem to engage with criticism—not just with me, but more generally, as with the blowback to his much-criticized coronavirus articles from early 2020—I’m not surprised that he doesn’t go on to this blog and engage with the criticism here. I mean, sure, he could respond, something like, “Yeah, to me it was a new continent but in retrospect I regret that we did not engage with the work of W. E. Deming and Fischer Black”—but, if he was going to do this sort of thing, I guess he wouldn’t have written that Noise book the way he did in the first place etc.

          I respect Sunstein’s enthusiasm for ideas such as nudging and his willingness and ability to write prolifically and get out there in the marketplace of ideas. To me, a key part of this is engaging directly with disagreement—I do it on the blog, Rachel Meager does it on twitter, others do it through academic articles. But I recognize that’s not his style. It seems to me that Sunstein’s style is much more to push forward and let others engage with disagreement. That’s fine! I’m a big proponent of the division of labor, and there are lots of ways to make contributions. I wish Sunstein did not denigrate those of us who publicly criticize, as well as publicly celebrate, published work, but I respect that the back-and-forth of criticism and response is not something he chooses to do.

          Kahneman is different: he has a long history of engaging with criticism, so it makes sense that he’d comment on the blog so that we could clarify our points of disagreement. I think that worked well.

        • I’m reminded of Robert Nozick, who said it wasn’t productive for him to reply to his critics rather than moving on to other work (in contrast the argumentativeness of contemporary libertarian political philosopher & Nozick-critic Murray Rothbard), and described his own method as “non-coercive”.

        • Wonks:

          Not responding to critics is a risky move if the goal is scientific progress. If you’re doing good work, you can keep doing it and let other people follow up with fixes, and everybody wins. On the other hand, if you’re doing bad work, for example building general theories off work such as that of Brian Wansink, or garbling an idea that was already well understood in economics and statistics, then dismissing or ignoring criticism can leave you just wasting your time and others.

        • Andrew,

          These are random thoughts. In essence, I don’t think you are unfair in your critiques.

          I do not follow Cass Sunstein’s Twitter feed unless one of his tweets ends up in my own feed. Usually the tweet announces a new article or book. I hadn’t come across the ‘stasi’ tweet either.

          Actually I reviewed earlier Andrew’s blog postings about Cass Sunstein’s endorsement of a few case studies. It pays to revisit earlier discussions. I am quite surprised that Cass Sunstein would characterize the proponents of replication as ‘stasi’. I speculate that Cass didn’t mean to use the characterization in its most literal sense. Two other posters on an earlier blog arrived to that same hypothesis. Nearly everyone i know, including me, has made faux pas and held false information. Sometimes we are late in admitting this.

          Nevertheless, once an expert resorts to an unpopular or offensive characterization, the expert can expect pushback. On Twitter, the reactions of subsets can be strident. As I have mentioned before, I don’t get to excited by someone’s surly or strident disposition or comments. But others do get excited and angry. Some will ruminate over them for years. Some of them are extremely intelligent and insightful too.

          Bottom line, subsets hold grudges. Being mindful of this grudge mentality is a big plus. And can save one needless angst and derision. The main point is that in many contexts such conflicts are stifled and stifling science particularly when it is likely that with some further reflection, a better approach to communicating criticism can be mapped.

          People skills do matter. That doesn’t mean that authentic reactions should be suppressed. Heck they can informative too depending on context. It’s much in how you word your reactions. Maybe that comes with a respect for wisdom.

          As I noted before, you are a master debater. Simply expressing your critique signals attention from many audiences across different fields. I’m sure many look at your blog. I have no experience as to whether Cass can or can’t debate well. The forums where he has appeared are not set up for debate. They are set up for commentaries.

          Censorship and dissent have to be on the front burner. Cass Sunstein has made excellent case for doing so. Even Daniel Kahneman expressed concern about the seeming increased censorship.

        • Sameera:

          It’s hard to know. I think it’s good to have a distribution of approaches. My approach to commenting on published research is valuable (I think), but it’s also valuable to have other approaches, including Nick Brown’s and including other approaches that are more insidery. I would not want to argue that the way I do things is the best, nor would I want everyone to go about it my way!

        • RE: “It’s hard to know. I think it’s good to have a distribution of approaches. My approach to commenting on published research is valuable (I think), but it’s also valuable to have other approaches, including Nick Brown’s and including other approaches that are more insidery. I would not want to argue that the way I do things is the best, nor would I want everyone to go about it my way!”
          ———-
          Your blog is unique and necessary for the advancement of science. To clarify Typically, I have come at much of expertise as an observer, recognizing that there is much I don’t know outside of a few areas. Moreover, I am an enthusiast of intellectual freedoms, as history has shown how brilliant thinkers can be marginalized and the consequences of suppressing diverse perspectives.

  2. “you can contact well-known scholars and hope they’ll write blurbs like, “A genuinely new idea,” “unique and novel insights,” “a previously unknown . . . problem as large as an elephant,” etc.”

    …with the implicit understanding that they will do the same for you.

    • Matt:

      Maybe so. But when I blurb books, it’s not in expectation of anything in return. I blurb because it’s kinda fun to write a blurb, and because I want to be nice to whoever asked me, and because if I like the book I’d like to help promote it. By saying all this, I’m not trying to imply that I’m above concerns of self-promotion—when I write books, I ask well-known people for blurbs, and I want them to be positive!—it’s just not an exchange. I don’t know that I’ve ever blurbed a book by someone who’s blurbed for me. Not that I wouldn’t do so; it just hasn’t come up.

      The “previously unknown . . . problem as large as an elephant” thing, though, that really bothers me. Maybe I’ll have another post just on that one. I have no reason to think that blurb is insincere; I just think it reflects ignorance. But ignorance of an interesting kind.

  3. When I was in undergraduate school I had a co-op experience alternating semesters of classes and work experience in industry. One of my coworkers/mentors was a metallurgist who earned his PhD in Germany, guess it would have been in the 60’s or early 70’s. He attended a dissertation defense where a PhD candidate had sent out pro forma invitations to a number of well-known experts in his specific area. Somewhat surprisingly, one of the experts showed up in person.

    After the presentation he asked for questions and the expert stood up and said, “I wish you had asked me to review your thesis before it was completed. In particular you failed to understand my own work which you cited extensively”. Then he proceeded to completely disembowel the poor guy’s methods and results. According to my coworker, the unfortunate candidate spent two more years starting basically from scratch on his research.

    Compared to that story, the idea of being offended that someone’s blog post points out errors in your published research seems pretty trivial.

  4. Although I had that blog post “kinda in the pipeline” for a while, what pushed me to write it was a situation where I was perhaps too critical of an article that was not all that bad (although it still had several clear problems, but there was no question of misconduct), and people had questioned whether in that case I should have discussed the issues with the authors of the article first.

    The answer came down to “Well, if I’d known then what I know now, then I would have done, but I didn’t, so I couldn’t have”. Basically, if you make a policy of generally not engaging with authors before writing about their papers, because you know from experience, that 85-90% of the time this is a waste of effort and may even help them to cover up what they have done, then the other 10-15% of the time you will appear to have been too harsh. But I think this is defensible, provided that you (a) are up-front about it, and (b) try not to be too much of a dick about it. These are not easy to reconcile; just as I am averse to getting into conflicts I am also averse to even quite mild criticism, especially when I know I’ve not done the best job that I (in some universe) “could have”.

    • >… because you know from experience, that 85-90% of the time this is a waste of effort and may even help them to cover up what they have done, then the other 10-15% of the time you will appear to have been too harsh.

      Consider this from another angle. What approach is likely to have the best scientific outcome – IOW, less than a 85-90% chance of no serous engagement with your critique.

      Is it possible that approaching a scientist and saying “I think I see an error in your analysis and I wanted to run my thoughts by you to see what you have to say. How about if we correspond further to resolve this difference of view?”

      Would be more likely to encourage someone to respond with openness and transparency about their error as opposed you to writing a blog post pointing their error.

      No guarantees, of course. It’s entirely possible that your correspondent would tell you to fuck off or just ignore you in response to the first approach. But I suspect it would have a better chance of being getting a less sub-optimal response.

      • Joshua:

        Most of the time I’m not so interested in the particular scientist in question but rather in more general issues. This might have to do with my background in statistics, because in statistics we’re always developing general methods. Even when we’re focused on an applied problem, we’re using methods that we’d like to use again in other settings.

        The relevance to science reform is that when pointing out problems in published research, my primary goal is not to get some particular researcher to clean up his act but rather to think more generally about what goes wrong. For this sort of general thinking, I think a published exchange is best.

      • “Would be more likely to encourage someone to respond with openness and transparency about their error as opposed you to writing a blog post pointing their error.”

        That sounds like a practical approach on the one hand. But on the other hand, science is *supposed* to be about debate. When a scientist publishes a paper, s/he should *expect* the paper to be closely scrutinized and criticized. S/he should be anticipating criticism and anticipating how to respond to it. That’s *is* the game. So to the extent people aren’t willing to face challenges to their work, they shouldn’t be involved in science.

        • > science is *supposed* to be about debate

          To be pedantic, science is “about” building human understanding of the world. Debate is a tool that can be used to serve that goal, but science isn’t “about” debate any more than it is “about” experiments or “about” p-values (or Bayes factors or what have you).

          Although I’m being pedantic, I think this is important because many examples of bad science arise from conflating the tools of science with the goals of science. This was Feynman’s point in his “Cargo Cult Science” piece, but we see it today with ESP, power pose, etc.

          As a tool, debate serves the goals of science when the parties involved have shared goals (e.g., to come to a mutual understanding of the causes of a phenomenon) and shared premises (e.g., a mutual acknowledgement of the relevant data).

          For that reason, I agree with the general theme that if you are trying to serve the goals of science, you should be as open and transparent as possible and expect your work to be scrutinized, critiqued, and eventually supplanted. Even if science is not “about” debate, a good scientist should supply all the material for the tool of debate to work as intended.

  5. The last time this came up here (Sunstein was involved but I don’t recall it perfectly), I checked Amazon and everyone involved had traded blurbs on each other’s books, all glowing. Glad – but not surprised – you are not doing that. These Amazon reviews have largely replaced serious literary criticism.

    The “elephant” thing also struck me as particularly vacuous, as if the authors had finally discovered the elusive reason as to why science is hard.

  6. In this post I want to discuss why I blog directly about what I see as errors or other problems in scientific articles… I have seen criticism of the “blog first” approach because it “drops stuff on the authors out of the blue” or “doesn’t give them a right to defend themselves”. People have suggested that it would be better to approach the authors first and discuss the problems.

    I think this inadvertently skips past a more important question, which is “Should I contact the authors to tell them I have just published a blog post criticizing their work?”

    If scientific progress is the goal of the criticism, then it seems to me that the answer to this question is clearly, “Yes”. Regardless of whether one should contact authors before posting, one should contact them when posting.

    The way blogs work, study authors often won’t hear about your critical blog post at all unless you inform them yourself. Or if they do hear about it, it will often be too late for them to respond because blogs move very quickly.

    So there is less opportunity for scientific progress in terms of the authors’ learning from their mistakes or you and your readers learning from your own mistakes (if you have made any).

    It would take only a handful of minutes to send a short polite email to the authors when posting, so at least they are aware of it.

    • Fogpine:

      There’s nothing wrong with sending such an email but I don’t see why it should be required or expected. My papers get cited all the time but people don’t always contact me to tell me so. I have a feeling that my papers sometimes get cited in contexts where the authors disagree with me, and they typically don’t cite me either. So I don’t see why bloggers should be expected to initiate contact more than others.

      The longer answer is that, yes, sometimes I do contact people, but often I don’t, for the same reason that Nick often doesn’t contact people . . . it’s awkward! I mean, sure, I could find the university email of that sleep scientist and tell him that I’ve posted a blog with criticisms of his work, but realistically I just don’t see this leading anywhere useful.

      That said, sometimes I do contact authors whose work I’m criticizing. For example, in writing the gaydar paper, I contacted the author of one of the papers we were discussing. I don’t think he agreed with most of what we wrote, but he sent us some helpful comments on our paper that allowed us to avoid some errors or points of confusion.

      So I guess the practice I follow is to contact people if I think this might lead to something useful, but not to do it by default. Which I guess is the same policy used by people who write research articles or news articles commenting on published work.

  7. Thanks for your response Andrew!

    It still seems best for scientific progress to contact the authors at the time of blogging detailed criticism. That would be so quick — an epsilon of additional time expenditure! What is the downside for the blogger besides the (usually small) chance the original authors show the criticism to be incorrect?

    One reason Nick avoids reaching out to authors in advance is that there usually isn’t a fruitful reaction. However, chances are high that authors won’t even hear about a detailed blog criticism unless the blogger takes a couple minutes to inform them. If the blogger doesn’t inform them and the authors aren’t aware of the post, then it is the blogger who is forcing the absence of fruitful reaction.

    Regarding your analogy with citations, from my perspective it’s a thoughtful argument but doesn’t fully work.

    Citations usually amount to a sentence or paragraph, and so aren’t really comparable to a detailed post. A closer analogy would be with Letters to the Editor in a journal, which original authors are specifically invited to respond to.

    Moreover, the best parts of blogging and post-publication review are meant avoid the problems with traditional publication, not repeat them! What’s better — the original authors being aware that someone put work into finding the problems with their research, or the original authors not even hearing about it?

    Imagine a faulty study is of cancer, for example. If authors are made aware of the fault, they may not acknowledge it, but at least will be incentivized to fix the problem in their next study (lest the criticism worsen), with real implications for patients.

    • Fogpine:

      You write that it would be a downside if the original authors show the criticism to be incorrect. No, that would be an upside! If I’m wrong, I’d like to know as soon as possible. The downside is the potential for an unpleasant social interaction, for example getting a nasty email in reply. I’m not saying it’s rational for me to want to avoid such a downside; it’s just the way it is. It’s my impression from reading Nick’s post that he feels the same way.

      Again, I think it’s good that different people have different approaches. We here have an active comments section and I’ll acknowledge errors that I make and I’ll engage with disagreement. In contrast, when Uri Simonsohn blogs, he contacts ahead of time the people who he’s writing about, but he does not always engage with disagreement, nor does he have a comments section. That’s fine; it’s how he does things. I think my approach works for me and his approach works for him. I have no doubt that there are things I could change to make my blogging more effective—but I do all of it as an extra thing, and when it starts getting unpleasant, I won’t be motivated to continue.

      Again, I’m not saying your wrong here! Maybe I should be sending out all these emails. Also maybe the people who write articles disputing my published claims should contact me about it. All this could be a good idea. I’m explaining why I haven’t done it so far, but that doesn’t mean that I’ve been making the right decision here.

      • Andrew, Thanks for your candor.

        It’s worth putting authors in awkward situations to correct errors and advance science. In that context, it’s difficult for me to understand how those issuing the criticism are comfortable avoiding the awkwardness for themselves.

        On the other hand, I’m totally comfortable hiding behind an anonymous name to avoid negative career repercussions from scientific discussions online. No doubt many would disagree with my choice more than with yours!

    • This seems naive to me. For some alleged “errors” what you suggest could work – people genuinely want to do the right thing, to improve and/or correct their analysis. However, in many cases their research is highly politicized. Some of the areas I’ve worked in involve research designed to promote or support a particular view in a contested regulatory proceeding. In other cases, the research is designed (or at least used) to support particular monetary interests (true for too much medical research). In such cases, the authors rarely want to improve their analysis – their interest may only be so they can invent defenses against the critique. Providing the authors with the critique in such cases is simply helping them protect the view they were expounding.

      I realize this is a very skeptical view of research. And I don’t intend it to characterize the majority (nor do I claim how sizeable the portion is). However, I have seen too many high profile cases where the authors of research have no intention of altering their positions. When errors are pointed out, they simply correct the errors, change their model, and continue to support the same position they originally took.

      • Dale, Thanks for the response.

        My view is probably naive, yes. However, I don’t think scientific progress is possible without buying into this sort of naivety to some degree, even if the buy-in is as minimal as, “I’m criticizing this invalid method so the authors have to find a new invalid method to rely on for their show-trial studies, slowing them down for a while.”

        Besides, making authors aware of criticisms isn’t only about incentivizing better research, it is also about incentivizing accuracy in criticisms. A blogger who knows the authors’ are aware of their criticisms also knows they are more likely to get informed rebuttals. Often, authors are the only ones who can provide an informed rebuttal on a blog because, besides authors and the blogger, none of the blog commenters have read the study in question.

        The options here aren’t just “good study + bad criticism” vs. “bad study + good criticism”. It’s all too easy to end up with “bad study + bad criticism”, just like in everyday disputes it’s very easy to have “bad action + bad response”.

  8. I’m curious another aspect of science and if and where there is a boundary that can be crossed in this area with regard to misconduct. Is it “misconduct” to say one thing in a paper and another thing in the press? On the one hand, obviously, scientists are entitled to their opinions, so in one sense it’s reasonable for them to limit their comments in published papers to the available data, but expand upon those comments in the press with their opinions. On the other hand, if there’s any group of people that are supposed to know that the exact reason that opinions that are too weekly supported to be expressed in published research probably ought not to be hyped as a likely outcome in the press, it should be scientists.

Leave a Reply to Andrew Cancel reply

Your email address will not be published. Required fields are marked *