Yesterday we discussed a news article on science journals being overwhelmed by fake research. Here was one quote from that article:
International biotechnology consultant Glenn Begley, who has been campaigning for more meaningful links between academia and industry, said research fraud was a story of perverse incentives. He wants researchers to be banned from producing more than two or three papers per year, to ensure the focus remained on quality rather than quantity.
At first I thought that was a horrible idea. Some of us have more than two or three things to say in a year! Is Begley trying to silence us??
But then I thought, sure, it all just depends on how you define “journal articles.” Instead of publishing 20 journal articles in a year, I could put 20 articles on Arxiv and just choose 3 of them to publish in journals. That would be kinda cool, actually, as it would save my collaborators and me huge amounts of effort in dealing with review reports, paperwork from journals, and so forth. My collaborators and I could write as much as we always do, just much more efficiently.
So I’m on board with Begley’s proposal. I’m not sure how it would be enforced, and I’m not planning to do it unilaterally but I’m starting to warm to it it as a general policy.
So, again, you’d be able to continue to publish 20 or more articles per year, but there’s a logic to saying that only 2 or 3 can be in “journals.” There are still a couple of details that need to be worked out, such as how to count coauthored articles and how to think about non-journal publications such as arxiv. The next step would be to replace journals by recommender systems, an idea I’ve proposed before (see also here). This would move us toward an equilibrium in which we publish 0 journal articles per year. Or a system where journals continue to exist but have no special status, in the same way that private schools such as Columbia continue to have some value but are not considered intrinsically better than public schools.
I said in 2011
“I’d suggest scientists should limit themselves to an average of two original papers a year.”
https://www.theguardian.com/science/2011/sep/05/publish-perish-peer-review-science?INTCMP=SRCH
and several times since then, e.g.
https://www.dcscience.net/2011/12/16/open-access-peer-review-grants-and-other-academic-conundrums/
It still makes sense to me.
David:
I disagree. I write something like 20 original papers a year, and I think that’s a good thing! However, as discussed in my above post, I think it’s fine if only 2 or 3 are published in journals, with the rest appearing on Arxiv.
That sounds like a very good compromise.
I was thinking primarily about papers that involve experimental work. I have written only one paper (1985) that has been nominated as a classic. It was 57 printed pages and took 5 years of work. It could have been split into 10 papers, but the results would have been far inferior.
Even theoretical work can take a long time. Our 1982 paper on the stochastic theory of single molecules was 59 printed pages.
I think that anyone would be doing well if they published even one such paper a year. They certainly had a lot more than 2 or 3 weeks work in each of them.
Incidentally, it was undoubtedly these two papers that got me elected as a Fellow of the Royal Society. I’ve certainly never regretted failing to split them into a lot of smaller papers.
References
1985 https://www.onemol.org.uk/colquhoun%20&%20sakmann%201985.pdf
1982 https://www.onemol.org.uk/Colquhoun%20&%20Hawkes-1982-ocr.pdf
Donald Geman wrote essentially the same thing in 2007 — “My own (half-serious) suggestion is to limit everybody to twenty lifetime papers. That way we would think twice about spending a chip”
(Though he said this in the context of abolishing conference papers https://www.cis.jhu.edu/people/faculty/geman/publications/pdf/Ten_Reasons.pdf )
The status of Columbia is definitely not what it used to be (like, a couple of weeks ago).
It seems to me that the idea has merit. Not only would it improve the value of what is published, but it would free up the time of reviewers and editors.
On the other hand, some valuable ideas might be less widely circulated. For example, Google Scholar credits one well-known economist, Jean Tirole, with 696 publications. Some of them are books but most are journal articles. See https://scholar.google.com/citations?user=ZEDUm5UAAAAJ&hl=en . I think maybe there should be an exception for Tirole and those like him. Set his cap at 50 papers per year.
Regarding recommender systems. We have established recommender systems for books—NYT Book Review, Publishers Weekly, Amazon Reader Reviews, and so on. They are imperfect but I believe many use them in deciding what to read.
What could be done to create a recommender process for NBER Reports, SSRN, Arxiv, etc.? Specifically, how could one create positive incentives for reviewers to participate and to participate honestly? How could one avoid false recommendations? I understand false recommendations pollute Amazon reviews.
SSRN allows one to rank papers, including search results, by downloads. It seems that is a start. Perhaps some sort of thumbs-up voting mechanism on archives would help?
Bob76
50 papers per year implies one week’s thought and data generation per paper.
When I served my turn on the committee that selects fellows of the Royal Society, I saw candidates rejected because they had published such large numbers of papers.
Even weighing papers would be better than counting them.
That ‘anonymous’ reply was from me.
David:
I disagree with your statement, “50 papers per year implies one week’s thought and data generation per paper.” In 2021 I published 23 papers. Suppose I work 46 weeks per year, then your calculation would imply two week’s thought and data generation per paper. But that’s not right! I spend much more than two weeks on each paper. What you’re missing is that I can work on many projects at once.
As best I can tell Professor Tirole is not a member of the Royal Society. He is a fellow of the Royal Society of Edinburgh and the Royal Swedish Academy of Science awarded him the Nobel Prize in economics.
One cannot please everybody.
Bob76
I don’t see any need for this. The journals should just set standards and stick to them. It does seem like there’s an incentive on the part of the publishing industry to push as many papers out as possible, since the more that’s published and the more widely it’s spread, the more journals libraries will have to buy. What’s a market mechanism to control that?
“Isn’t it pretty to think so?” (Hemingway (The Sun Also Rises)
Andrew:
The virtues of limiting researchers to 2 or 3 published papers per year as a strategy for increasing the quality of published research doesn’t commend itself to the untutored intellect. What am I missing here?
John
John:
For the reasons discussed in the above post, the “You’re only allowed to publish 2 or 3 journal articles per year” rule would make me happy. That’s something right there!
I don’t see the value in limiting publications to two or three a year – publication rates (I wouldn’t necessarily call it “productivity”) is highly dependent on lots of factors; e.g. if you run a big group you may feel it important that each of your students/postdocs manage to achieve 2-3 papers every three years and that could end up with authoring 10 or more papers a year.
In the biomedical/physical science, and if one is goal-directed towards something substantial and participating in doing the research oneself (as opposed to managing a research group), two to three papers a year seems about right (I’m with David Colquhoun on that), unless your approach is superficial or (perhaps) you have a technique that is in demand by other researches and can be applied without much effort through lots of collaborations. Otherwise, it’s not easy to see how research focus can be dedicated to more than a few lines of attack. Of course, science is a very personal pursuit and how one pursues it varies.
There’s another aspect tho and this relates to the fact that science increasingly has two public interfaces. In terms of the progression of scientific fields (I’m thinking of biomedical/physical sciences where there is an external, fixed reality which may not apply to some aspects of social science and especially economic “science”), it doesn’t matter a damn how one gets there since we expect the “truth” will eventually out. So if rubbish, mendacious crap and science misrepresentations are posted on arxiv it may be annoying but doesn’t really hinder the finding out of reality (I’m quite partial to Fayerabend on that) by researchers acting in good faith.
But the developing edges of scientific fields may overlap with societal concerns that are potentially contentious (global warming; response to pandemics; evolution etc.) and giving stuff posted to arxiv and the like equal weight to papers that have gone through a journal publication process is asking for trouble IMO because it then becomes easy to provide a sort-of parallel body of “publications” to support some agenda or to maintain the pretence of uncertainty where this may not really exist (“teach the debate” as the Intelligent Designers would have it pre-Dover!).
It may be an old-fashioned viewpoint but publishing in scientific journals still provides a number of checks; it forces the submitter to take care that the research is valid and methodologically sound as far as they’re able (it helps if the researchers have internalised some standards of quality); it puts the paper through a review process that will at the very least find the agenda-led rubbish (it helps if the journal is of sufficient quality to have robust review procedures). It provides a forum that is slow, but thorough, in refuting rubbish.
“it forces the submitter to take care that the research is valid and methodologically sound as far as they’re able”
No, it doesn’t.
It’s pretty amazing you’d even write that. That has been a major concern of this blog and Andrew’s advocacy for decades.
It does force the submitter to take care that other people will support the type of or particular research they do, but “valid and methodologically sound” research is not a requirement of the current publication system. Ultimately, truly bad research frequently gets shouted down in the end. But that can take years or even decades.
Its a kind of funny thing to say also if you know the history of a lot of “pre-revolution” research – for example the “geosyncline” theory that preceded plate tectonics, and was really just a lot of wild imagining with little basis in anything that could be called “verified”.
Or – to smack down what’s apparently your field – what about “junk DNA”?? A concept completely ridiculous in evolutionary terms but widely claimed to be true for generations by biochem types studying DNA.
The same could be said about numerous social science claims that are in wide circulation today – little more than guesswork or hope underlain by shoddy – if any – fundamentals.
Genes mutate. Without mutation there would be no evolution, which is trail and error plus memory. Most mutations fail to produce anything beneficial, i.e., they produce junk. The junk can later mutate into something beneficial, but in the meantime it is junk, a completely non-ridiculous, in fact, essential part of evolution.
The argument as I understand it, is about whether specific genes are junk or have undiscovered functions. Plus the semantics of whether having a large collection of junk might be useful in a dumpster-diving way. No expert that I am aware of is claiming all genes are doing beneficial work that couldn’t be removed without harming the organism. In fact there are known to be some genes which are producing active harm in the unlucky people who have them.
As in evolution, trial and error is how science progresses. It is useless and harmful to blame scientists from making mistakes as long as they remain willing to consider contrary evidence.
Out of the millions of known genes, not a single one is “junk”. The supposed junk is the other 99% of DNA with unknown function.
If all you do is brute force trial and error, that isn’t science. That is what “evidence based medicine” and similar amount to though. In science you synthesize information into a theory/model that lets you predict what will happen in new circumstances. This is far more efficient than trial and error.
chipmunk:
1. re your quibble with (I’ve added back the bit you omitted) “it forces the submitter to take care that the research is valid and methodologically sound as far as they’re able (it helps if the researchers have internalised some standards of quality) ”
I’m referring to the large majority of researchers at least in biomed/physical sciences who are trying to find stuff out in good faith through their research (these generally have internalised standards of quality). Of course there have always been a few cheaters and rogues and we are unfortunately seeing a pretty substantial underbelly of rubbish in low grade (and quite ignorable) journals nowadays.
But standards of quality do apply certainly in the biomed/physical sciences and IMO maintaining and improving these should have the effect of raising the game at least in the upper sector of quality journals (not much one can do about the rubbish journals except ignore these). Casting these aside in favour of dumping everything on arxiv and equivalents would not be good especially for contentious areas of societal importance where the misrepresenters, conspiracy theorists and various other agendas would have a field day.
2. Not sure what you mean with the rest of your post. Sure, where observational and experimental evidence is underdetermined there are bound to be competing theories some of which may appear rather dubious in hindsight. But that’s science! In your example, out of that uncertainty, evidence for plate tectonics grew (especially the rather serendipitous observations of midocean ridges and the mapping of magnetic anomalies and their stunning implications). As I said in my post we expect the truth will eventually out – it may take time though. Not sure what you’re referring to with your “verified” sentence – I didn’t use that term in my post.
What’s your problem with “junk DNA”? – it seems a pretty useful hypothesis and is still supported by lots of evidence. I can’t think of a reason why this should be considered “completely ridiculous in evolutionary terms” – have you got a simple description of why you think that’s the case?
Maybe because I got an undergraduate degree in History before I switched to science, I’ve read a fair amount of the early literature on various topics, when society journals published papers that had been read at their annual meetings. Some of the societies had a system I liked. The journal editors would recruit two or three ‘discussants’ who would comment on the paper, and then publish the comments with a response by the author along with the paper. That system could work if people only get to publish two papers a year.
This is not unusual in statistics (see Gelman, 2007, Struggles with survey weighting for example). I really like this as a model, and I wish more disciplines would pick this up.
In industry, a resume over 1 page is considered to be too long. What if, similarly, you could only list 10 of your published papers to represent yourself on your CV (and department website, academic bio, other related documents), and awards committees, hiring, tenure-making provosts only solicited and looked at 10 of your publications?
Addendum: what would be your 10 (or 5, if you’re too lazy)?
That applies to the UK Research Excellence Framework (REF) process on which basis around £2 billion of public money is allocated to Universities for research. Outputs per academic staff is limited to their top 5 papers in the REF assessment period (e.g. your 5 “best” papers in the last 7 years). This presumably has some impact in promoting quality over quantity in publishing outputs … although some might say that a significant aim is to maintain the status quo in terms of allocation of research funds.
Somebody:
If I had to list only 5 published papers, it would depend on the audience which 5 I’d list. If the topic is political science, that’s one list; if it’s statistical theory, that’s another; if it’s statistical computing, yet another; if causal inference, a different set of 5; etc.
For NSF funding proposals, you’re restricted to listing at most ten of your articles (or the like) to establish your credentials.
Similarly, in NIH Biosketches, one can list five “Contributions to Science” with up to 4 papers each, so a limit of 20.
The NIH does something like this now for grant applications. On your “biosketch” (that’s what they call a CV) you are permitted to list up to 5 publications that speak directly to your most relevant experience in connection with the proposal under review, and then for each of four areas in which you have made “contributions to science” you can list up to 5 publications. So that’s a maximum of 25 altogether.
As a reviewer, it sure beats the old days when people would list hundreds of publications!
I’ve thought about this from time to time, and my initial sympathy for a (completely unrealistic) cap on papers-per-person-per-year has waned. Basically, it addresses the symptom (too many papers) rather than the causes (too many scientists, too-large research groups, and incentives for publishing quantity over quality).
How about half of the projects that get funded are direct replications? That will transfer a big chunk out of the wasteful category into useful.
We can combine the two ideas: you can write original research papers every time you have reported a replication. That way there is no explicit cap on activity. m=1 seems most obvious. As this will raise questions about what it means to be an author, either apply the rule by author number (have fun with accounting), or using the contribution statements (ugh). Everyone gets one paper-opportunity for free to make it more fair for junior researchers.
The main appeal to me in limiting the number of papers per year is that it would put a stop to the increasing expectations on number of papers published (it doesn’t matter whether this expectation is true or perceived). There was a time when few or no PhD students would publish before submitting their theses, now the norm is by-publication and the number of publications for this norm seems to grow. People now recommend bachelor and master students to publish their “research project” as a way to get a PhD stipend. There are rare geniuses that write important papers as undergraduates but that is not the norm, not the 10th percentile, and recommending undergraduates to publish is a way to poison the mindset from the start. “You, with no prior experience or knowledge of the current status of the field, should be able to learn how to do research, decide on a topic, do the research, find something interesting – so you better chose your topic wisely – write it up and publish it, all within half a year.” That message completely undermines that research is a slow process, one that involves trying things that may fail. Multiple students expressed the worry that if p > 0.05 in their project, they would not graduate. That is wrong in many ways! Students are not at fault for picking up this nonsense and being worried about it. It means they are getting the vibe and figuring out how funding works; they are not entirely wrong.
I loathe the recommendation that students publish a literature review. They should definitely read the literature and attempt to summarize it as a way to focus their thoughts and understanding. Should we expect a randomly selected student to review the entire literature in 6 months and describe it in a novel way, something that is worthy for anyone to read besides supervisors and reviewers? Why do many automatically suggest or insist that students publish their review? When I went through this, there were on average two new publications per day, today it is probably three per day. Someone with 20 years of experience can read the titles or abstracts and have a good ratio of dismissing the less important publications with limited effort. With 0 years of experience you will be forced to take shortcuts, or give up on the idea that your summary is important beyond a learning experience.
Instead of “write original research papers”, I think it should be “get funded for original research projects”. Otherwise it makes sense to try it out. It would be fascinating to see how people eventually figure out how to game the rules, and until then it would generate more useful info anyway.
Over in Computer Science, Turing Award winner Michael Stonebraker talks about the “diarrhea of papers” and its consequences: https://www.youtube.com/watch?v=DJFKl_5JTnA&t=862s
In relation to this, this position paper from the DFG (the main research funder in Germany) may be of interest https://www.dfg.de/en/research_funding/announcements_proposals/2022/info_wissenschaft_22_61/index.html
Systemic problems aside, how specifically do other people here blog-readers consume academic literature these days? Search engines? Regular journal reading? Twitter? Word-of-mouth? Some other mechanism?
I think the extent that stuff comes down to me, it’s effectively word of mouth, but I’m not in academia.
This sounds like a pill for the symptoms, not the cause, of the illness.
Then we would need to return to citing individual chapters or subchapters, just to keep some fidelity when referring to an 80 page paper