Paul von Hippel writes:
Stuart Buck noticed your recent post on A WestLaw for Science. This is something that Stuart and I started talking about last year, and Stuart, who trained as an attorney, believes it was first suggested by a law professor about 15 years ago.
Since the 19th century, the legal profession has had citation indices that do far more than count citations and match keywords. Resources like Shepard’s Citations—first printed in 1873 and now published online along with competing tools such as JustCite, KeyCite, BCite, and SmartCite—do not just find relevant cases and statutes; they show lawyers whether a case or statute is still “good law.” Legal citation indexes show lawyers which cases have been affirmed or cited approvingly, and which have been criticized, reversed, or overruled by later courts.
Although Shepard’s Citations inspired the first Science Citation Index in 1960, which in turn inspired tools like Google Scholar, today’s academic search engine still rely primarily on citation counts and keywords. As a result, many scientists are like lawyers who walk into the courtroom unaware that a case central to their argument has been overruled.
Kind of, but not quite. A key difference is that in the courtroom there is some reasonable chance that the opposing lawyer or the judge will notice that the key case has been overruled, so that your argument that hinges on that case will fail. You have a clear incentive to not rely on overruled cases. In science, however, there’s no opposing lawyer and no judge: you can build an entire career on studies that fail to replicate, and no problem at all, as long as you don’t pull any really ridiculous stunts.
Hippel continues:
Let me share a couple of relevant articles that we recently published.
One, titled “Is Psychological Science Self-Correcting?, reports that replication studies, whether successful or unsuccessful, rarely have much effect on citations to the studies being replicated. When a finding fails to replicate, most influential studies sail on, continuing to gather citations at a similar rate for years, as though the replication had never been tried. The issue is not limited to psychology and raises serious questions about how quickly the scientific community corrects itself, and whether replication studies are having the correcting influence that we would like them to have. I considered several possible reasons for the persistent influence on studies that failed to replicate, and concluded that academic search engines like Google Scholar may well be part of the problem, since they prioritize highly cited articles, replicable or not, perpetuating the influence of questionable findings.
The finding that replications don’t affect citations has itself replicated pretty well. A recent blog post by Bob Reed at the University of Canterbury, New Zealand, summarized five recent papers that showed more or less the same thing in psychology, economics, and Nature/Science publications.
In a second article, published just last week in Nature Human Behaviour, Stuart Buck and I suggest ways to Improve academic search engines to reduce scholars’ biases. We suggest that the next generation of academic search engines should do more than count citations, but should help scholars assess studies’ rigor and reliability. We also suggest that future engines should be transparent, responsive and open source.
This seems like a reasonable proposal. The good news is that it’s not necessary for their hypothetical new search engine to dominate or replace existing products. People can use Google Scholar to find the most cited papers and use this new thing to inform about rigor and reliability. A nudge in the right direction, you might say.
Google Scholar also lists the articles that cite the given article, thus telling you where the citation counts come from. If a reference is important to your own work, it makes sense to look at those. You may uncover a replication failure. I suppose someone could automate this search by looking for keywords like “replicat…” and “faill…” But then there is the problem of whether the failed replication is itself trustworthy. I often decide to ignore the reported failure. That part cannot be automated.*
As it is now, Google Scholar is an amazing service, provided for free. I use it for all sorts of things, especially looking for reviewers. For that purpose it is much better than the “Web of Science Reviewer Locator”, which uses key words to find warm bodies who might be relevant to the paper under consideration.
I’m told by an inside source that Google Scholar has a very small staff (especially compare to other Google services), and my biggest concern about it is that it will someday be discontinued.
* An example, somewhat hypothetical but only because I don’t want to bother looking up the articles in question. Many experiments get a result that “works” because it finds an effect in one direction despite an obvious mechanism that works in the opposite direction. And example is the classic study of belief polarization by Lord, Ross and Lepper (1979). They found polarization of opinion from evidence that was on the whole neutral. According to most theories of rational belief formation, neutral evidence should either do nothing or move all opinions toward neutrality. The effect of interest must be strong enough to overcome this effect. Thus, “failure to replicate” the polarization effect just means that your manipulation wasn’t quite strong enough, although it may well have reduced the effect of rational belief formation. Thus, in my textbook, I ignore these “failures”, although I do now mention a very convincing alternative explanation of the effect (Jern et al.).
Jonathan Baron wrote:
“The effect of interest must be strong enough to overcome this effect. Thus, “failure to replicate” the polarization effect just means that your manipulation wasn’t quite strong enough.”
This is depressingly close to a general dismissal of the concept of replication. It is also pretty close to saying that we just have to accept these studies as true. Psychology has embraced a paradigm of “if it’s true anywhere, it is true” as opposed to the “hard” science approach of “if it’s true everywhere, it is true.”
IMO, this sets the falsification bar way too high. There is essentially no way to claim that an effect shown in a paper is buried in noise, because maybe the replication was conducted with more noise than the original study and there are no tools available to show otherwise. If it’s not falsifiable it is not science.
“It is also pretty close to saying that we just have to accept these studies as true. ”
Matt! Of course they’re really true!! Real socialism has never been tried, Ehrlich will eventually be right, and the massive spike in crime since 2020 has nothing to do with defunding the police!
Oh – and don’t forget! – it’s *impossible* to measure anything useful about intelligence with a standardized test, even though standardized testing works for everything else on the planet.
Academics know things that other people just don’t understand!
Jonathan –
> According to most theories of rational belief formation, neutral evidence should either do nothing or move all opinions toward neutrality.
Why? Wouldn’t neutral evidence just be used to reinforce existing beliefs?
Or is the key there in the world “formation” – which indictes maybe that there was no prior belief?
It seems like newer search engines like consensus are actually moving in this direction, although consensus doesn’t seem to have a feature for connecting a study with replications yet.
There’s many interesting ideas to improve search for specialized cases. A big problem, however, is how to fund the cost of all the infrastructure and the developers. I’ve followed one particular project, better news search. It was a great idea, the people running it were very well-connected academics – but keeping it running and improving it just cost too much for what they could get even with good grant funding. In general, efforts of this sort seem to hit a funding wall in terms of getting past proof-of-concept and prototypes. The ability is there, it’s that nobody has solved the money issue yet.
Seth,
Yes, good point. We’re so used to getting useful software for free—Google, R, Python, Stan, Latex, etc. There’s this disconnect between the delivery of the service and the point of payment. Back in the old case this was the case with TV—you get the shows for free but you have to pay for the ads in the form of higher prices for the advertised products—but at least we were directly seeing the ads. Free software is created as some sort of byproduct of all the extra money that’s sloshing around in academia and industry, just as people are getting this blog for free as some sort of indirect consequence of Ivy League colleges charging high tuition and spending it on faculty to do whatever they want.
It all kinda works well until the maintainers of software don’t feel like putting in the effort to keeping it up with the times, and then we wonder what can be done about it!
I downloaded the PubPeer extension hoping it did similar, but no dice. A web plugin would probably be a good intermediary step. I don’t doubt Google *could* easily make the requisite modifications but it doesn’t really suit them to do so.
scite.ai is an academic search engine worth using. It shows supporting, mentioning, and contrasting citations like in the legal world.
The References check tool is useful with its ability to scan for retractions and editorial notices. I use the scite Zotero extension to see previews of this kind of data right in my library.
The only downside: it’s pricey if you want full functionality.
Also, still doesn’t take notice of PubPeer comments last time I checked.
We actually discussed scite.ai in our NHB article. Unfortunately It didn’t distinguish accurately between positive and negative cinations, at least in the example we looked at. Have a look at the NHB article — it’s only 800 words:
https://www.nature.com/articles/s41562-022-01518-0.epdf?sharing_token=jfhCcDFJeqjWARy5BuZLbNRgN0jAjWel9jnR3ZoTv0MdudtminWTgyERkjbHcOPyVCm9jnAdqynfMhvx_76xVIuV02luOlpJnj5e_qGaKa-WiQwvyPwjz72PsKNcXNwOItggpovwLQOYmIvgE3UGCT0l-jEKW2VuJZemNt_DpOQ%3D
Nice work. There is still a lot to improve with tools like scite.ai clearly. I wrote a critical response to a paper a few years ago, yet scite.ai just classifies it as mentioning the critiqued paper. Reason being that after the first citation of the original paper we used an abbreviation, so, the main point of the paper being a critique was lost, as all critical “citations” were missed by scite.ai