Econ Journal Watch asked me and some others to contribute to an article, “What are your most underappreciated works?,” where each of us wrote 200 words or less about an article of ours that had received few citations.
What happens when you drop a rock into a pond and it produces no ripples?
My 2004 article, Treatment Effects in Before-After Data, has only 23 citations and this goes down to 16 after removing duplicates and citations from me. But it’s one of my favorite papers. What happened?
It is standard practice to fit regressions using an indicator variable for treatment or control; the coefficient represents the causal effect, which can be elaborated using interactions. My article from 2004 argues that this default class of models is fundamentally flawed in considering treatment and control conditions symmetrically. To the extent that a treatment “does something” and the control “leaves you alone,” we should expect before-after correlation to be higher in the control group than in the treatment group. But this is not implied by the usual models.
My article presents three empirical examples from political science and policy analysis demonstrating the point. The article also proposes some statistical models. Unfortunately, these models are complicated and can be noisy to fit with small datasets. It would help to have robust tools for fitting them, along with evidence from theory or simulation of improved statistical properties. I still hope to do such work in the future, in which case perhaps this work will have the influence I hope it deserves.
Here’s the whole collection. The other contributors were Robert Kaestner, Robert A. Lawson, George Selgin, Ilya Somin, and Alexander Tabarrok.
My contribution got edited! I prefer my original version shown above; if you’re curious about the edited version, just follow the link and you can compare for yourself.
Others of my barely-noticed articles
Most of my published articles have very few citations; it’s your usual Zipf or long-tailed thing. Some of those have narrow appeal and so, even if I personally like the work, it is understandable that they haven’t been cited much. For example, “Bayesian hierarchical classes analysis” (16 citations) took a lot of effort on our part and appeared in a good journal, but ultimately it’s on a topic that not many researchers are interested in. For another example, I enjoyed writing “Teaching Bayes to Graduate Students in Political Science, Sociology, Public Health, Education, Economics, . . .” (17 citations) and I think if it reached the right audience of educators it could have a real influence, but it’s not the kind of paper that gets built upon or cited very often. A couple of my ethics and statistics papers from my Chance column only have 14 citations each; no surprise given that nobody reads Chance. At one point I was thinking of collecting them into a book, as this could get more notice.
Some papers are great but only take you part of the way there. I really like my morphing paper with Cavan and Phil, “Using image and curve registration for measuring the goodness of fit of spatial and temporal predictions” (12 citations) and, again, it appeared in a solid journal, but it was more of a start than a finish to a research project. We didn’t follow it up, and it seems that nobody else did either.
Sometimes we go to the trouble of writing a paper and going through the review process, but then it gets so little notice that I ask myself in retrospect, why did we bother? For example, “Objective Randomised Blinded Investigation With Optimal Medical Therapy of Angioplasty in Stable Angina (ORBITA) and coronary stents: A case study in the analysis and reporting of clinical trials” has been cited only 5 times since its publication in 2019—and three of those citations were from me. It seems safe to say that this particular dropped rock produced few ripples.
What happened? That paper had a good statistical message and a good applied story, but we didn’t frame it in a general-enough way. Or . . . it wasn’t quite that, exactly. It’s not a problem of framing so much as of context.
Here’s what would’ve made the ORBITA paper work, in the sense of being impactful (i.e., useful): either a substantive recommendation regarding heart stents or a general recommendation (a “method”) regarding summarizing and reporting clinical studies. We didn’t have either of these. Rather than just getting the paper published, we should’ve done the hard work to more forward in one of those two directions. Or, maybe our strategy was ok if we can use this example in some future article. The article presented a great self-contained story that could be part of larger recommendations. But the story on its own didn’t have impact.
This is a good reminder that what typically makes a paper useful is if it can get used by people. A starting point is the title. We should figure out who might find the contents of the article useful and design the title from there.
Or, for another example, consider “Extension of the Isobolographic Approach to Interactions Studies Between More than Two Drugs: Illustration with the Convulsant Interaction between Pefloxacin, Norfloxacin, and Theophylline in Rats” (5 citations). I don’t remember this one at all, and maybe it doesn’t deserve to be read—but if it does, maybe it should’ve be more focused on the general approach so it could’ve been more directly useful to people working in that field.
“Information, incentives, and goals in election forecasts” (21 citations). I don’t know what to say about this one. I like the article, it’s on a topic that lots of people care about, the title seems fine, but not much impact. Maybe more people will look at it in 2024? “Accounting for uncertainty during a pandemic” is another one with only 21 citations. For that one, maybe people are just sick of reading about the goddam pandemic. I dunno; I think uncertainty is an important topic.
The other issue with citations is that people have to find your paper before they would consider citing it. I guess that many people in the target audiences for our articles never even knew they existed. From that perspective, it’s impressive that anything new ever gets cited at all.
Here’s an example of a good title: “A simple explanation for declining temperature sensitivity with warming.” Only 25 citations so far, but I have some hopes for this one: the title really nails the message, so once enough people happen to come across this article one way or another, I think they’ll read it and get the point, and this will eventually show up in citations.
“Tables as graphs: The Ramanujan principle” (4 citations). OK, I love this paper too, but realistically it’s not useful to anyone! So, fair enough. Similarly with “‘How many zombies do you know?’ Using indirect survey methods to measure alien attacks and outbreaks of the undead” (6 citations). An inspired, hilarious effort in my opinion, truly a modern classic, but there’s no real reason for anyone to actually cite it.
“Should we take measurements at an intermediate design point?” (3 citations). This is the one that really bugs me. Crisp title, clean example, innovative ideas . . . it’s got it all. But it’s sunk nearly without a trace. I think the only thing to do here is to pursue the researcher further, get new results, and publish those. Maybe also set up the procedure more explicitly as a method, rather than just the solution to a particular applied problem.
“Objective Randomised Blinded Investigation With Optimal Medical Therapy of Angioplasty in Stable Angina (ORBITA) and coronary stents: A case study in the analysis and reporting of clinical trials”
Your title is almost as long as some recently mentioned papers (those 2 paragraph ones). I’d suggest that this title gets in the way of it being cited. But you list a number of more succinct titles with few citations, so I’m not sure title length is a good predictor of citations, even though this one might be.
I would prefer a title that says something about what the answer is rather than just giving the question. I hope the paper did more than just ask a question. The title should summarize the paper.
That only works if the paper can be summarized in a sentence or less. I happen to have been reading recently a paper entitled “Relationship between migration and latitude among west European birds.” There’s a pretty strong relationship, with, as you might expect, northern birds being more migratory. But there are differences among various ecological categories of birds, the relationship is not a simple straight line, there are model selection issues, and a bunch of other things to think about. The title, I think, is really good, because it tells you exactly what the paper is about.
Titles announcing a result– say, “Birds in western Europe often show a quadratic relationship between migration and latitude”– overstates the confidence one can place in the result, leaves out many things actually discussed in the paper, and glosses over the authors’ reasonable approach to how much can be said based on the data they were able to gather. Most scientific papers, in my experience, don’t come to some simply stated conclusion that can be announced in the title as a declarative claim, such as “X regulates Y”. Such claims don’t properly reflect the uncertainties of scientific investigation, or, in my opinion, the humility that authors should have about their own results; such titles are hubristic.
David:
To avoid the question mark, I could’ve changed the title of “Should we take measurements at an intermediate design point?” to “Conditions under when it is appropriate to take measurements at an intermediate design point,” but that seems too jargony to me. I guess I could’ve tried some gimmick title such as “Take measurements at an intermediate design point only if gamma > 0,” and then defined “gamma” appropriately in the paper.
Going forward, I do think it’s possible to do more work on the topic. Maybe the next iteration will get more attention.
I confess to once having written a paper with the title “Relationships between …”. I suspect almost no one has read it (nor is it a very good paper). If I have written a paper, I hope there is one declarative sentence I can write that is consistent with the paper. The title need not cover everything that is in the paper. But, I would like to learn something from reading the title. Similarly, I want to learn something from reading an abstract. Don’t just tell me what questions you had. Also, tell me some of the answers.
David:
This discussion is helpful. In the case of “Should we take measurements at an intermediate design point?”, I guess a better title would have been, “Using prior information about dose-response to decide whether to take measurements at an intermediate design point,” which is a little bit awkward but has the advantage of conveying more of the content.
Yeah, I can see some advantage to a declarative statement in a title but here I much prefer the original title/question.
Maybe “Use dose-response prior to decide whether to take measurements at an intermediate design point”. In sixth grade, we got a book out of the library each week. After reading it, we had to write a one-sentence summary of the book.
David:
Thanks—that works!
The main paper from my PhD I tried to title something like “Fluid flow and grain motion cause the earthquake liquefaction of sands”.
This would have been a great title because it describes succinctly the result, and also the standard textbook description said specifically that “undrained” conditions = no fluid flow was the cause of earthquake liquefaction.
The reviewers demanded that I change the title. It was too declarative and simple and (in their opinion) over-confident. The truth was the result destroyed half a century of research and they didn’t want it to call the standard theories out too directly.
Yes that’s too bad – it’s an excellent title IMO. Of course some people can get away with declarative titles even if their “declaration” is untrue rubbish. e.g.:
“Why Most Published Research Findings Are False”
Well, here’s a question to be answered: Why would Wordsworth be on my mind this morning, when I looked out my back window?
I read “A simple explanation for declining temperature sensitivity with
warming,” but was unable to follow the logic. And I don’t think the title is right.
Bullet points from the paper:
– “Most studies attribute changes in temperature sensitivity to shifts in underlying biological processes.”
– “Yet, despite an increase in studies reporting declining or shifting temperature sensitivities, none have provided strong evidence of the biological mechanisms underlying these changes…”
– “The missing mechanisms may be hidden in the data: Environmental factors…”
And now the main point of the paper:
– “Here, we propose a simpler alternative explanation: the use of linear models for nonlinear responses to temperature. Researchers generally use methods with assumptions of linearity to calculate temperature sensitivities, often relying on some form of linear regression to compute a change in a quantity—days to leafout or carbon sequestered over a fixed time, for example—per ℃, thus
ignoring that many biological responses to temperature, especially events, are nonlinear (Figure S1).”
There is no argument in the paper that the declining response to temperature is not real, only that it is modeled as linear when it is really nonlinear. So what does the paper provide “a simpler explanation” for? The way I read it, it is a simpler explanation for why linear models diverge from the data at higher temperatures, but it is not an explanation for why the temperature divergence occurs, which is what the title claims.
My guess is researchers might come upon this paper wanting to cite it but end up just scratching their heads because they can’t really use it the way it is written. Perhaps:
“A simpler explanation for why models fail to adequately account for declining temperature sensitivity”
Is this a typo?
“we should’ve done the hard work to more forward in one of those two directions.”
My guess is this should be
“we should’ve done the hard work to move forward in one of those two directions.”
I guess it is, and possibly a frequent one, when fingers fly; notice the position of the respective keys, r, f, and v.
The latest on the “if you don’t like your lack of citations, do something about it” front:
https://www.science.org/content/article/vendor-offering-citations-purchase-latest-bad-actor-scholarly-publishing
“From there, it was relatively easy to purchase additional citations. Using the name of the fictional scientist, the research team contacted the vendor through WhatsApp and purchased the “50 citations [for $300] package.” Within 40 days, five papers were published that each included 10 citations to the fake news researcher’s work. Four of the five appeared in a single chemistry journal.”
Andrew, I hope your mention of the ‘morphing’ paper will prompt some people to read it. (“Using image and curve registration for measuring the goodness of fit of spatial and temporal predictions”). About once a year I think ‘I should pick up that work again’, I really think there’s something there.
For other readers of this blog: the idea behind the paper is that for some kinds of data, such as time series or maps, the difference between prediction and data can best be thought of as multi-dimensional: you have the magnitude wrong and you have the location wrong. Summarizing with something like RMS error of the value might be unhelpful or misleading. For example, if there’s a process that occasionally produces a sharp spike in some parameter, and you try to forecast the magnitude and timing of the spikes, a forecast that the parameter value is constant can produce an RMS error smaller than one that gets that magnitudes exactly right and has the locations just a little bit wrong. But: imagine printing your forecast on a rubber sheet, and then locally distorting the sheet to reduce the error between the data and the distorted forecast; now you can quantify the error in terms of both the magnitude of the remaining error and the amount of distortion you had to do to achieve that. You can make distortion cheap or expensive by changing the stiffness of the rubber sheet. I love the idea and I really like the paper, although I regret that we didn’t use a title like “A Day Late and a Dollar Short: a method for quantifying multidimensional error” or something like that. A day late and a dollar short, that’s such a great way of pointing out that there are errors in both dimensions.
Anyway we made a really good start in that paper, but we have quite a few unresolved issues so we don’t have something we can really trust. I was hoping/expecting we would end up with a method so straightforward (even if not computationally simple) that it would become a standard way of summarizing errors, especially time series errors: the RMS error is such-and-such, and this decomposes into (this much error in magnitude, that much error in timing). We aren’t there.
By the way, after publishing the paper we learned that the concept behind the paper had been used before, I think for analysis of aerial photography way back in WWII or something like that. There is nothing new under the sun. Or at least not so much.
I’ll also mention another paper that I really like, that both Andrew and I co-authored (along with Lin and Krantz): “Analysis of Local Decisions Using Hierarchical Modeling, Applied to Home Radon Measurement and Remediation.” That’s not a great title but since my work on it was funded by a radon grant we really had to get Radon into the title, not just the body of the paper, even though the main point of it really is more general, about how to combine a lot of uncertain continuous numbers in order to get a discrete decision. https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi=fa98fb7009382d3ddd4b5b8b9e22c76b3e435bd7
I don’t think there’s any single thing in the paper that was new at the time, but we got some emails (‘some’ meaning ‘three’, I think) from college professors who said it’s the cleanest example they’ve seen of decision-making under uncertainty in a complicated situation. So that was nice. Probably there are plenty of other good examples now though.
“(‘some’ meaning ‘three’, I think) ”
which is how many people typically take the time to actually read thoroughly and understand *any* paper, regardless of how many times it has been cited. (kudos to those people both for their effort and for complementing you on your work and effort)
IMO an interesting study would be to choose a widely cited paper, then create a test on the contents of the paper and have all the people who cited it take the test…
Ha, yeah, that would probably not go very well.
I once made the mistake of reading (or at least skimming) a bunch of papers that cited my work, to see what influence I had had. It was pretty depressing. For every citation that based on an actual use of something from one of my papers, there were about five that were something like “Other people have worked on this paper too (Schmuck 1988, Schmoe 1992, Groucho 1997, Price 2002)” where it wasn’t even clear that they had read so much as the abstract.
(This comment appears down below too, because I put it there by mistake.)
I really like the “Analysis of Local Decisions…” paper and recommend it a lot as an example of the full Bayesian loop, including individual decision-making and policy recommendations. Unfortunately, you don’t get any citation credit from my recommendations.
Thanks Eric! I don’t care about the citations at all, except inasmuch as a lack of citations usually indicates that nobody is reading the paper. I’m very happy to hear that all these years later you are still recommending the paper, I think that’s great!
Ha, yeah, that would probably not go very well.
I once made the mistake of reading (or at least skimming) a bunch of papers that cited my work, to see what influence I had had. It was pretty depressing. For every citation that based on an actual use of something from one of my papers, there were about five that were something like “Other people have worked on this paper too (Schmuck 1988, Schmoe 1992, Groucho 1997, Price 2002)” where it wasn’t even clear that they had read so much as the abstract.
The link to treatment effects in before-after data does not work. still gated at the publisher.
OK, I added links to all the articles.
Not so long ago, word had it that a sure way to get noticed was to have a title with the following phrase appended:
“… and the Jewish Question.”
Today, of course, that phrase is too hot to handle and even an MOT would get in trouble. For a possibly outmoded disquisition on the subject, see
https://contendingmodernities.nd.edu/theorizing-modernities/why-is-the-jewish-question-different-from-all-similar-questions/
A few ideas to resolve this titling dilemma:
1. Pretesting and piloting titles with relevant audiences may help, taking inspiration from survey methodology.
2. Clickbait techniques, used carefully, could pull in more readers. This means:
a. positive framining
b. results/claims are at the center
c. succinct (5-20 words considered optimal for citations)
d. no word play, but use other interesting angles like questions, quotes, surprising or colloquial words
e. Create a curiousity gap, keeping a bit of important info in the abstract.
Finally, I’d advise keeping the four Cs in mind: credibility, communicability, creativity, and connectivity (keywords for relevant scholars to latch on to).
Overall, it remains a pretty difficult problem but at least there are some constraints to work with.