A message came in my inbox from “The RetractoBot Team, University of Oxford,” with subject line, “RetractoBot: You cited a retracted paper”:
That’s funny! When we cited that paper by Lacour and Green, we already knew it was no good. Indeed, that’s why we cited it. Here’s the relevant paragraph from our article:
In political science, the term “replication” has traditionally been applied to the simple act of reproducing a published result using the identical data and code as used in the original analysis. Anyone who works with real data will realize that this exercise is valuable and can catch problems with sloppy data analysis (e.g., the Excel error of Reinhart and Rogoff 2010, or the “gremlins” article of Tol 2009, which required nearly as many corrections as the number of points in its dataset; see Gelman 2014). Reexamination of raw data can also expose mistakes, such as the survey data of LaCour and Green (2014); see Gelman (2015).
We also cited two other notorious papers, Reinhart and Rogoff (2010) and Tol (2009), both of which should have been retracted but are still out there in the literature. According to Google scholar, Reinhard and Rogoff (2010) has been cited more than 5000 times! I guess that many of these citations are from articles such as mine, using it as an example of poor workflow, but still. Meanwhile, Tol (2009) has been cited over 1500 times. It does have a “correction and update” from 2014, but that hardly covers its many errors and inconsistencies.
Anyway, I can’t blame RetractoBot for not noticing the sense of my citation; it’s just funny how they sent that message.
This RetractoBot seems pretty cool, but they must be working through quite a backlog if they are emailing you about something you published in 2017, citing a 2014 paper that was retracted in 2015.
Also, I take issue with the last sentence of the paragraph you cited. The problem with the LaCour and Green study wasn’t so much about making “mistakes” as it was about making stuff up.
Nah, Andrew just has a seven-year backlog of blog posts.
I don’t see any problem with citing a retracted paper but you should indicate that it has been retracted when you cite it (assuming it was retracted before you submitted you paper).
For example, if you read a retracted paper and some description in the introduction or discussion or wherever gives you an idea for some experiments, you really should cite the retracted paper as an influence on your study (even if the retracted paper turns out to be an entire fabrication).
scite.ai would pick up on the nature (type) of citation, but I’m not sure whether even the paid version would offer auto-alerts like Retractobot. The Reference Check feature comes close.
” I guess that many of these citations are from articles such as mine, using it as an example of poor workflow, but still.” According to scite, there are 57 “supporting” citations, 1310 “mentioning” and only 14 “contrasting” citations.
Is there any standard way of labeling *references* as “supporting”, “mentioning”, “contrasting” (“criticizing”)?
I sometimes do a quick assessment of a paper by first checking the references, and papers:
1) May only have references I consider credible. Easy.
2) May have many I consider not credible or recognize instantly as familiar & bad. Easy.
3) Have many credible references, but some not.
3a) the low-credibility references may being debunked,
OR
3b) the low-credibility ones are those actually relied on, others are deprecated or just bibliography-pad, may not even be cited. (For example, the Wegman Report only cited 40 of 80 references.)
I once wrote a report where I felt I had to label the references to avoid reader cringe.
“Some are examples of anti-science (*), a few are science or other reasonable sources (+), and the unmarked ones describe anti-science activities.”
God, just about everything Tol, and for that matter Nordhouse too, have ever published is total garbage. Those two have more blood on their hands than just about anyone else.
https://www.tandfonline.com/doi/full/10.1080/14747731.2020.1807856
https://theintercept.com/2023/10/29/william-nordhaus-climate-economics/
Also, Tol claiming that the collapse of the gulf stream would benefit the economy, which is literally insane.
https://iai.tv/articles/what-economists-get-wrong-about-climate-change-auid-1970
Sadly the vast majority of economics is the study of what would happen if the world were held to an impossible bunch of simplifying assumptions that have no basis in reality, and are often just false. Mainstream economics does not understand how banks work, which is sad because of how trivally easy it is to prove definitively with a basic knowledge of double entry bookkeeping.
https://www.sciencedirect.com/science/article/pii/S1057521915001477
I actually would take issue with your interpretation. It is entirely possible that the collapse of the gulf stream might benefit the economy. The insanity is not with that statement, but with economics. Economics has always been a partial view of the world. In my view, it is fundamentally inconsistent with ecological principles. The measured economy only measures part of what happens – even if externalities are included and valued, it is still a partial view which excludes many resources. Resources valued by humans are included. Resources have no value in themselves – they must be of value to humans, at least indirectly. So, an ecological disaster may well benefit the economy. And, if that disaster has potentially catastrophic consequences, then economic theory will apply probabilities and discounting to valuing the impacts. There is much to be said in defense of the economic approach (but I’ll leave that to others, as I only believe some of it and strongly disagree with other parts). If you find this insane, I think most of your issue is with economic methodology itself rather than the actual application of that methodology (e.g., Nordhouse, and to a lesser extent Tol, may be practicing sound economic methodology as defined by economists).
This may be what you mean, so I’m not necessarily disagreeing with you. But I wanted to draw a distinction between bad applications of economic theory and limitations of that theory itself.