We should all routinely criticize our own work.

Kerim Kavakli points me to this blog by psychology researcher Todd Kashdan who goes over his recent career and points to concerns he has with his own published work.

I agree with Kavakli that this is a great thing that all of us should be doing. I did not read Kashdan’s examples in detail so I have no comments on the research or whatever mistakes were made, but in general we should regularly engage in self-criticism.

I had only one concern with Kashdan’s post, which is where he seems to associate bad research with bad journals (“There are so many journals that exist. Any time a research study gets rejected, it can be resubmitted somewhere else, and if it happens again and again, scientists can keep on resubmitting until some horrendous outlet takes it.”) The trouble is that top journals publish crap too.

18 thoughts on “We should all routinely criticize our own work.

  1. Being able to resubmit over and over to different journals isn’t necessarily a bad thing. Often I have found that the reason for rejection is simply that ‘your paper wasn’t cool enough for our journal’. The coolness factor seems to be what gets crap into top journals too.

    I have also found this phenomenon in both grants and journals – you try to publish or submit something; it gets reviewed and the reviewers want changes; you don’t really like all these but make the changes as best you can anyway; it gets rejected; you try again with the updated version from the last submission thinking it is improved; the reviewers in submission 2 directly contradict the reviewers in submission 1 and you have to change your submission back the way it was originally.

    So maybe sometimes people are submitting bad work over and over until someone takes it. And sometimes people are submitting decent work over and over until they get a decent review. The reviewers may also be the ones submitting the bad work over and over on the flip side. And vice versa. Its all the same pool of people in the subject matter area, after all.

  2. Obviously toward the end of an academic career, it is easier to select the worse of one’s work and critique it. One can be assured though that the environment is such that one will find a considerable number who will critique the work regardless of whether one acknowledges the errors/crappiness of the work selected.

    On Twitter, I am astonished by some of the bullying tweets. In reading Ellen Condliffe Lagemann’s book Elusive Science, The Troubling History of Education Research, I am even more aware of the sociology of expertise has held back some of the best thinking in education. Relationships can determine who is held in esteem or not.

    • I see frequent references in comments here to Twitter bullying or just Twitter whatever. Is it really true that established, successful academics engage in tit-for-tat exchanges on Twitter concerning their work? I find that mind-blowing, frankly. What possible end can that serve other than treating ones own professional publications with contempt?

      • Yes, subsets of academics engage in tit-for-tat exchanges. Hypercompetitiveness defines the academic environment and social media. It intensified when universities/colleges made changes to their criteria for promotion and tenure: largely in the early 70s and expansion of adjunct faculty positions. Then, my speculation is that the evidence based medicine movement had significant impact on not simply the medical field but also in the social sciences. The movement raised the issue of accountability in many scientific endeavors/projects. Since the early 80s, the requirements for tenure and promotion have expanded. Publish or perish still drives the career tract. I’m sure you know all this.

        Some academics have a terse manner. More broadly lack people skills. There is little use lecturing these academics though. Shaming only complicates the relations. Some just go around shaming others and can’t take the heat when the shaming is directed at them.

        • Sometimes I wish that I still retained, ‘I’ll kick your ass’ disposition [back in high-school]. I exhibited it on Linkedin fora effectively. But I think to adopt that posture on Twitter wouldn’t be so effective. Some of the terse ones waddle off and complain to their cliques on the one or two occasions I tried it. . So darn juvenile. Right? right

          Heck I’ll get a script out of maybe. lol

      • ” Is it really true that established, successful academics engage in tit-for-tat exchanges”

        Seriously? :) That tradition that predates Twitter by about five centuries!! :) It goes back at least to Leibniz’ and Newton’s fight over who invented calculus. In which, according to one book I read, Newton anonymously organized an anonymous commission, lead by Newton, which submitted a report to the Royal Society – written by Newton – that concluded that Newton had invented calculus. :)

        There’s a modern idea afloat that academia is populated only by serious straight-minded professionals, but sorry to say nothing close to that has ever existed.

        • I have met some really nice academics too. I got caught up in the case of my Dad’s promotion, a project that occupied far too much of my life. The experience explains my career ambivalences at certain points in my life, even though I have been presented with graduate school opportunities.

          Fortunately or unfortunately, I had not developed a sufficient ambition for academic life. There are rewards to it, I am sure. But I also think that much of what we learn is not of great quality. Richard Posner’s work revived my interest in certain fields.

          I’m reading about the academic rivalries in the education field since the 19th century. So you are right probably that collegiality is either non-existent or rare. Tversky & Kahneman had quite a rare friendship. I admire that. My observation is that there is more of that collegiality among and respect for academics.

          We have begun to distrust expertise for a variety of reasons.

        • “So you are right probably that collegiality is either non-existent or rare. ”

          Apologies, I didn’t mean to say or imply that. Collegiality is common – but it’s just one side of the coin or half the show. The other half isn’t as…(choosing the word carefully)…presentable.

          “We have begun to distrust expertise for a variety of reasons”

          I argue with the word “begun”. A favorite aphorism when I was growing up was “you can always trust a fishing expert to tell you why they didn’t catch any fish”. The saying isn’t just a dis on experts. It captures the essence of expertise: a fishing expert is only as good as the fish they can catch. If the fish are somewhere over the rainbow, “expertise” is just gas.

          The truth is that humanity has long experience with people claiming to be “experts” who are “experts” in claim only – meaning they supposedly know all about catching fish, but for whatever reason they haven’t caught one yet. :)

          could be that there is more respect for academia in east Asia. That would be good if there were more respectable academics there. ;)

        • Jim

          Thanks for your perspective. After I posted my response, I thought to revise it. But had an obligation.

          I meant to suggest that with increased professionalization of many fields has come increased scrutiny by subsets of the public, investigative journalists, experts, some government bodies.

          The burgeoning of litigation also has been instrumental toward corporate and governmental accountability; of course resulting in varying degrees of successes and failures. We can all recite a string of tort liability cases that have labored for years toward a settlement.

          I think Paul Starr’s Social Transformation of American Medicine is illustrative of the changes to medicine as a consequence of increased professionaliation and commercialization.

          https://archive.nytimes.com/www.nytimes.com/books/98/12/06/specials/starr-medicine.html

          Again, it is with the evidence based medicine movement that some accelerated need for accountability became more noticeable, as for example to the design, method, and review of clinical trials. However the movement has been compromised as has been pointed out by some academic physicians and investigative journalists.

          Lastly, re: East Asian academia. Here in US, we frame so much as arguments [debates]; whereas in East Asia, conversational culture prevails. East Asians have more respect for academics. I saw that in Singapore, as one example.

        • Hi Joe,

          The content of the following article that I posted below goes more to the point I was trying to get across in the earlier post.

          I made the claim that “East Asians have more respect for academics”.

          The article contains the following claims, which I think align, partially, with the claim I posted above.

          1. “The problem in many places starts with the fact that academic appointments are sometimes not based on merit but rather on personal connections (guanxi), inbreeding (appointing academic staff from among the graduates of the university rather than searching for the best possible candidate in a legitimate and broader search), and in some cases corruption in admissions, testing, and other areas.”

          2.”….And the emphasis on seniority and respect for authority run very deep in many Asian societies. Universities have not yet figured out how to wed the standard norms of western higher education with Asian values and traditions. This might be impossible — the western paradigm for academic culture and university organization and structure has served as the international benchmark until now but this model may not be suited to all countries or all institutions.”

          https://www.insidehighered.com/blogs/world-view/academic-fraud-and-academic-culture-china-%E2%80%94-and-asia

          The characterization of corruption in research is described differently by different authors in different fields. I have followed Sheldon Krimsky’s work, from time to time. Particularly when I was in Boston. He authored Science in the Private Interest; the theme of whether conflicts of interest undermine science. And then there is Raymon Hubbard’s Corrupt Research- The Case for Reconceptualizing Empirical Management and Social Science, which quite a bold take.

          I’m not sure whether one can say that respect is absent when an academic culture is deemed corrupt.

  3. Andrew:

    Sure, everyone should routinely criticize their own work. But Kashdan’s post falls short as a model of how this should be done for 2 reasons.

    First, Harvest is the wrong time to learn you planted the wrong seeds. Self-criticism should be proactive, not retrospective, and should permeate every major design decision in the research process—decisions about the kinds of questions worth pursuing, how those questions are framed, the most appropriate study design, subject population, research methods, data collection strategies, and analytic techniques for addressing those questions, etc.

    Second, the substance of Kashdan’s retrospective assessment looks a lot more like virtue signaling than a serious self-critical evaluation of the studies he cites. Consider, for example, his 2006 article entitled “Social anxiety, positive outcome expectancies, and risk-taking behavioral intentions”. Looking back, Kashdan concludes that “the study is nonsense” because he relied on subjects’ estimates of risky behaviors they would engage in over the next 6 months instead of measures of their actual behaviors. He expresses no regrets or concerns, however, about the article’s reliance on non-experimental, cross-sectional questionnaire data from 84 college undergraduates to (1) draw causal inferences about study variables, and (2) recommend that “targeted prevention and intervention efforts should be considered” if these study findings are replicated.

    Or consider his 2012 article entitled ”The interactive effects of emotional clarity and cognitive reappraisal in Posttraumatic Stress Disorder”. Looking back, Kashdan again finds fault only with deficiencies of measurement— lack of fidelity between theoretical constructs and the questionnaires they used to measure those constructs. Notably absent from this retrospective evaluation, however, is any evidence that Kashdan has any misgivings the fact that the article relied on non-experimental, cross-sectional questionnaire data from a convenience sample of 75 military veteran patients to recommend how the “benefits of (existing) interventions might be increased by also targeting emotional clarity for improvement, or by tailoring interventions to improve cognitive reappraisal and similar emotion regulation strategies among those who already have high levels of emotional clarity.”

    Maybe I’m being too critical here, but I don’t see a lot of self-criticism.

    • “Harvest is the wrong time to learn you planted the wrong seeds.”

      OK, but better late than never. Sometimes during the frenzy of putting together a grant proposal, you just can’t see clearly some of the limitations of what you are proposing, or better ways to do them. You’re just too focused on getting the damn thing out the door before the submission deadline. I often find that in calmer times I think more clearly and see more broadly.

      • Clyde:

        I agree. In an imperfect world of limited time, unforgiving grant submission deadlines, fierce competition for diminishing resources, conservative peer review pressures, and institutional incentive structures pointed in the wrong direction— the world we live in, my proactive ideal is aspirational at best. Better late than never is exactly right and often the best anyone can reasonably aspire to in contemporary science. As a model of retrospective self-criticism, though, Kashdan’s post falls into the category of late and never!!

        John

Leave a Reply to jd Cancel reply

Your email address will not be published. Required fields are marked *