Last week in class we read and then rewrote the title and abstract of a paper. We did it again yesterday, this time with one of my recent unpublished papers.
Here’s what I had originally:
title: Unifying design-based and model-based sampling inference by estimating a joint population distribution for weights and outcomes
abstract: A well-known rule in practical survey research is to include weights when estimating a population average but not to use weights when fitting a regression model—as long as the regression includes as predictors all the information that went into the sampling weights. But it is not clear how to apply this advice when fitting regressions that include only some of the weighting information, nor does it tell us what to do when analyzing already-collected surveys where the weighting procedure has not been clearly explained or where the weights depend in part on information that is not available in the data. It is also not clear how one is supposed to account for clustering in such analyses. We propose a quasi-Bayesian approach using a joint regression of the outcome and the sampling weight, followed by poststratifcation on the two variables, thus using design information within a model-based context to obtain inferences for small-area estimates, regressions, and other population quantities of interest.
Not terrible, but we can do better. Here’s the new version:
title: MRP using sampling weights
abstract: A well-known rule in practical survey research is to include weights when estimating a population average but not to use weights when fitting a regression model—as long as the regression includes as predictors all the information that went into the sampling weights. But what if you don’t know where the weights come from? We propose a quasi-Bayesian approach using a joint regression of the outcome and the sampling weight, followed by poststratifcation on the two variables, thus using design information within a model-based context to obtain inferences for small-area estimates, regressions, and other population quantities of interest.
How did we get there?
The title. The original title was fine—it starts with some advertising (“Unifying design-based and model-based sampling inference”) and follows up with a description of how the method works (“estimating a joint population distribution for weights and outcomes”).
But the main point of the title is to get the notice of potential readers, people who might find the paper useful or interesting (or both!).
This pushes the question back one step: Who would find this paper useful or interesting? Anyone who works with sampling weights. Anyone who uses public survey data or, more generally, surveys collected by others, which typically contain sampling weights. And anyone who’d like to follow my path in survey analysis, which would be all the people out there who use MRP (multilevel regression and poststratification). Hence the new title, which is crisp, clear, and focused.
My only problem with the new title, “MRP using sampling weights,” is that it doesn’t clearly convey that the paper involves new research. It makes it look like a review article. But that’s not so horrible; people often like to learn from review articles.
The abstract. If you look carefully, you’ll see that the new abstract is the same as the original abstract, except that we replaced the middle part:
But it is not clear how to apply this advice when fitting regressions that include only some of the weighting information, nor does it tell us what to do when analyzing already-collected surveys where the weighting procedure has not been clearly explained or where the weights depend in part on information that is not available in the data. It is also not clear how one is supposed to account for clustering in such analyses.
with this:
But what if you don’t know where the weights come from?
Here’s what happened. We started by rereading the original abstract carefully. That abstract has some long sentences that are hard to follow. The first sentence is already kinda complicated, but I decided to keep it, because it clearly lays out the problem, and also I think the reader of an abstract will be willing to work a bit when reading the first sentence. Getting to the abstract at all is a kind of commitment.
The second sentence, though, that’s another tangle, and at this point the reader is tempted to give up and just skate along to the end—which I don’t want! The third sentence isn’t horrible, but it’s still a little bit long (starting with the nearly-contentless “It is also not clear how one is supposed to account for” and the ending with the unnecessary “in such analyses”). Also, we don’t even really talk much about clustering in the paper! So it was a no-brainer to collapse these into a sentence that was much more snappy and direct.
Finally, yeah, the final sentence of the abstract is kinda technical, but (a) the paper’s technical, and we want to convey some of its content in the abstract!, and (b) after that new, crisp, replacement second sentence, I think the reader is ready to take a breath and hear what the paper is all about.
General principles
Here’s a general template for a research paper:
1. What is the goal or general problem?
2. Why is it important?
3. What is the challenge?
4. What is the solution? What must be done to implement this solution?
5. If the idea in this paper is so great, why wasn’t the problem already solved by someone else?
6. What are the limitations of the proposed solution? What is its domain of applicability?
We used these principles in our rewriting of my title and abstract. The first step was for me to answer the above 6 questions:
1. Goal is to do survey inference with sampling weights.
2. It’s important for zillions of researchers who use existing surveys which come with weights.
3. The challenge is that if you don’t know where the weights come from, you can’t just follow the recommended approach to condition in the regression model on the information that is predictive of inclusion into the sample.
4. The solution is to condition on the weights themselves, which involves the additional step of estimating a joint population distribution for the weights and other predictors in the model.
5. The problem involves a new concept (imagining a population distribution for weights, which is not a coherent assumption, because, in the real world, weights are constructed based on the data) and some new mathematical steps (not inherently sophisticated as mathematics, but new work from a statistical perspective). Also, the idea of modeling the weights is not completely new; there is some related literature, and one of our contributions is to take weights (which are typically constructed from a non-Bayesian design-based perspective) and use them in a Bayesian analysis.
6. Survey weights do not include all design information, so the solution offered in the paper can only be approximate. In addition the method requires distributional assumptions on the weights; also it’s a new method so who knows how useful it will be in practice.
We can’t put all of that in the abstract, but we were able to include some versions of the answers to questions 1, 3, and 4. Questions 5 and 6 are important, but it’s ok to leave them to the paper, as this is where readers will typically search for limitations and connections to the literature.
Maybe we should include the answer to question 2 in the abstract, though. Perhaps we could replace “But what if you don’t know where the weights come from?” with “But what if you don’t know where the weights come from? This is often a problem when analyzing surveys collected by others.”
Summary
By thinking carefully about goals and audience, we improved the title and abstract of a scientific paper. You should be able to do this in your own work!
I like your 6 general principles. I think they could be applied to more than just papers – presentations and class lectures, for example. I don’t claim to be successful, but I find it useful to think along those lines for almost everything I do professionally. In particular, I was never taught how to approach a class session – often just following the text or a traditional course outline seemed like the obvious thing to do. But if I stop and ask myself the questions you list, then I have usually found that I should be doing something different than they way I was taught or the way textbooks are organized. I do believe almost everything can be improved by asking such questions.
Good advice once again. However, the motivating questions beg a follow-up question: Should we consider SEO in this age and day? I am neither sure nor found any papers on how academic databases rank “popular” content, but I’d guess there are similar things at play as with Internet search engines, provided that you don’t customize your searches. Of course, the follow-up to the follow-up would involve research ethics.
Acronyms generally, and especially in titles, are best avoided.
Godeofry,
In general I agree with you. But some acronyms are in such common use within their subfields (LLM for large language model, MRP for multilevel regression and poststratification) that using the acronym can make the title more accessible than using the longer phrase.
Andrew,
In general, though I consider myself a textual pragmatist, I don’t agree with you. Some people, even when familiar with the acronym, find the reading experience much smoother if it’s spelled out (particularly in titles). And, it could convey the message “We’re only interested in readers who already are immersed in the field”. And finally, if someone is searching for recent papers about “multilevel regression” by title, they won’t find this one.
I understand almost nothing concerning MRP or weights (except the kind I just used for my workout), but I do like thinking about titles. I browsed through the manuscript, but could not see new research in there, but I may be missing something because I don’t understand any of it. Should you want to “(…) convey that the paper involves new research”, I think that might not be too hard to somehow try and incorporate in the title. Perhaps something like:
“A proposal and application concerning MRP using sampling weights”
“A proposal and application concerning the use of sampling weights in MRP”
“MRP using sampling weights: a proposal and an application”
Anon:
The paper definitely has new research—that’s what’s in its sections 2 and 3. The novelty might not be so clear because I have a writing style in which I like to present each idea as a natural consequence of what came before, to the extent that the reasoning can seem so natural that a reader could just assume it’s all been done.
Regarding the idea of the title conveying novelty: Yes, you make a good point. On the other hand, there may be more interest in a review article than in new research, so it might not be so terrible to present new work in the form of a review article.
I’m reminded of a problem we had with our Red State Blue State book. We presented it and titled it and wrote it to be friendly and accessible; an unfortunate consequence was that many reviewers didn’t catch that the book was chock-full of original research! So, yeah, too much accessibility can be a bad thing.
The general principles seem to be good for a *methods* paper– how to estimate, measure, observe, etc. something– but less applicable to a paper that reports actual results. If I wanted to know, for example, how many species of soil arthropods occur in a field, I could write a methods paper on how to gather data that addresses the question and how to analyze the data. But if I wanted to compare the species richness of soil arthropods of, say, wheat fields vs corn fields using available methods, there is no “solution” to the estimation problem, just the estimates derived from available methods, and a comparison of the estimates.
Of course some papers present new or improved methods and then demonstrate them on an actual implementation, but most research papers, I wager, do not present new methods. In some future paper in which an author *uses* your conditioning on the weights, it will just cite your methods paper, and not deal much with principles 3-6.
Gregory:
I think my principles apply to substantive papers too, not just to methods papers.
1. What is the goal or general problem?
2. Why is it important?
3. What is the challenge?
4. What is the solution? What must be done to implement this solution?
5. If the idea in this paper is so great, why wasn’t the problem already solved by someone else?
6. What are the limitations of the proposed solution? What is its domain of applicability?
1. For a substantive paper, there’s still a goal (in your case, estimating the number of soil arthropods in the field, and more generally understanding something about the local variation of these bugs).
2. For a substantive paper, you’ll still want to explain why this is important—not necessarily important in a global sense, but important to somebody, somewhere.
3. The challenge could be as simple as: we want to compare the species richness of soil arthropods in wheat fields vs. corn fields, and that information is not easily accessible.
4. The solution could be as simple as: we did a survey, collected some data, and used standard analysis methods.
5. The answer to “why wasn’t the problem already solved by someone else?” . . . that’s kind of interesting, actually! If the question of richness of soil arthropods is important, and it wasn’t hard to answer, why wasn’t it answered already? It could be that these ecological variables change over time, and so new data were needed to learn about trends.
6. What are the limitations? That’s an interesting question too. Maybe the limitations have to do with potential future time changes, or maybe the fields that were measured in the study are not completely representative of fields of interest, or maybe there are imperfections with the data collection, etc.
One thing that this comment made me realize, though, is that I’m missing one important thing that should go in any paper, methodological or substantive. So I’ll add it here:
7. What’s the take-home point?
For a method, this could be that we now have a method that can be used on problem X. For your substantive paper, it could be that the average wheat field has 300 species of soil arthropods and the average corn field has 400. The point is that, in my above post, there was all this discussion of process but no discussion of the main result, which is the most important part of any research paper!
Here are some things Knut Schmidt-Nielsen wrote in his autobiography, The Camel’s Nose. Everything below is a quote.
If you produce a long title with difficult words and acronyms, you are likely to lose a potential reader. You are also apt to lose readers if you use lots of unnecessary words and put them up front, where you should put the most important information. Consider the title “Further Comparative Studies of Food Competition among Grazing Herbivores.” Those first five words say very little and tend to keep the subject of the study less obvious. Also, although “Herbivores” perhaps sounds more scientific, “Animals” is simpler. Why not say “Food Competition Among Grazing Animals”?
To show how this actually works, I hand out photocopies of tables of contents from scientific journals and ask the students to scan the titles and see if anything interests them. They are invariably stop at short titles such as “Freeze Tolerance in Turtles” or the even simpler “Bloat in Sheep.”
I know the post is about rewriting titles and abstracts, but are you (or someone else in the comments) able to explain a little more about the approach in the paper itself? – it seems very interesting/promising. I’ve read through it but I don’t quite grasp what exactly is being done/what the joint model is. I’m struggling to follow if you are running two separate models and combining them, or one model that includes both the weights and the outcome in some way.
Very concretely, if I have data with some outcome ‘Health Score’, a set of demographic predictors for use in MRP like Age, Income, Sex, State, and then also some weights that are derived from additional sources of information, what one or two regression models would I set up exactly for this approach?
Jamie:
Yes, the linked paper is incomplete, as can be seen from some of its later sections. I’m currently working on adding to it, including some sample code, which should help.
Thanks, it would be great to see the final implementation. Does it involve running 2 separate models – like 1 to predict the outcome, and 1 to predict the weights, and then combining these together in some way?
Jamie:
We have a couple different versions. The simplest version involves running a regression of log(weight) on X, then running a regression of y on log(weight) and X, then adjusting that second regression based on the results of the first regression, and then poststratifying. There’s a more elaborate version using mixture models (i.e., using a mixture of normals for the error term in the first regression), and an even more elaborate version using importance sampling.
Responding to this comment is making me want to write the code and finish the damn article! I was working on it in Aug and then got busy, or, should I say, some combination of lazy and busy. Right now it’s #2 on my research to-do list, after finishing the “Understanding posterior calibration with a simple example” paper.
The suggestions made in this following article on how to write an abstract are well worth following. The thing to remember is that the abstract is not an introduction or an advertisement for the paper but it _is the whole paper_ for people who will read it and skip the rest.
https://www.geo.utexas.edu/faculty/marrett/AdvStructure/Abstracts.pdf
Harsha:
Thanks for sending. The interesting question is, given that this advice is so simple, why is it not usually followed? Why are so many abstracts bad, or at least not everything they could be? I think one reason is that when we write papers, we don’t fully think about questions 1-7 listed above. That is, one reason it’s not so easy to write the abstract that summarizes our paper is that we’re not so clear about what our paper says!
I think it’s partly because it is usually the last thing that people do, after have slogged away at the paper for more time than they care to count – at which point they’d rather see the back of it! It often comes in as just an afterthought, and we often feel like the real work is done and it is superficial to focus on polishing the abstract
How has nobody pointed out the unintentional irony in a post about rewriting and improving titles?
Blog post title: “Hey! Here’s now to rewrite and improve the title and abstract of a scientific paper:”
Notice the typo? :)
Typo fixed, thanks.
I like the emphasis on reporting and clarity here. I suggest connecting this kind of exercise with reporting guidelines (e.g. APA Jars for Psychology: https://psycnet.apa.org/fulltext/2018-00750-002.pdf) and seeing if there’s overlap. I’ve been compiling a spreadsheet with this sort of data and it’s been interesting to use that as a writing template.
P.S. For the general template list (1-6) above, I prefer the order:
1. What is the context?
2. What is the problem arising from the context?
3. Why is this problem important?
4. Challenges to solving problem?
5. Set of possible and proposed solutions?
6. What are the limitations of the proposed solution? What is its domain of applicability?
Jay,
I’d add this one to your list: “If the idea in this paper is so great, why wasn’t the problem already solved by someone else?”
A title should stand the test of time. A few months ago, looking at Harvard and Stanford, I chose for the title of an upcoming talk, “Coast to Coast Data Fraud, 2023.” As of now, Johns Hopkins and USC have emerged as well; fortunately, each of the two is also symmetrically coastal so my title still applies. I hope the New Year arrives before the University of Chicago or Kansas State University enters the retraction field and ruins my coastal imagery.
Paul:
Ohio State has had some issues too.
Andrew,
Do you mean Carlo Croce? That was way back in 2022 and if it weren’t for Pennsylvania, Ohio would be on the Eastern Seaboard. From Wikipedia
https://en.wikipedia.org/wiki/Carlo_M._Croce
“Croce has said Columbus, Ohio lacks culture, motivating him to spend more of his time traveling than on campus.[2] Croce privately collects Italian Renaissance and Baroque paintings, with a claimed ability to identify and purchase genuine masters for a fraction of their worth.”
Apparently, Croce has been around for a while
https://www.nytimes.com/2017/03/08/science/cancer-carlo-croce.html
This 2017 Nytimes article illustrates your tax dollars at work.
“Over the last several years, Dr. Croce has been fending off a tide of allegations of data falsification and other scientific misconduct, according to federal and state records, whistle-blower complaints and correspondence with scientific journals obtained by The New York Times.”
Also this guy. A real letdown from the days of James Thurber and Jesse Owens.
I don’t see the problem. “Coast to Coast” includes everything in between; otherwise it would be “Coast and Coast.” The dictionary definition of “Coast to Coast” is “extending all the way across an island or continent.” So you’re fine no matter what.
This is a great post, and the followup discussion / replies flesh out some interesting thoughts about the back and forth of thinking/writing towards a better title and abstract.
But as others have noted, your revised title: ” MRP using sampling weights” is neither generally understandable (MRP ??? WFT???), nor does it connect to the the content of revised abstract.
As AE, I would strongly suggest you just change the title to
Multilevel Regression Using Sampling Weights
The postratification part is important in the paper, but I don’t think in the title
Michael:
Hmmm . . . my problem with the title, “Multilevel Regression Using Sampling Weights,” is that it seems to imply that this paper is intended for the user who (a) wants to do multilevel regression while (b) using sampling weights. Actually, though, multilevel regression (more fully, MRP) is the tool that we’re using, not the goal in itself. The intended audience is users who want to do regression modeling with survey weights. A more descriptive title would be, “Including survey weights in regression modeling using poststratification.” But that seems kinda confusing too!
So I’m stuck. Any other ideas, AE?
‘How to get the best of both worlds by combining MRP and sampling weights’ – ‘Have your cake and eat it too: Multilevel regression and poststratification with sampling weights’ :-p
These are obviously a bit of a joke but I think it helps to convey what the bonus is for the reader/that there is an advantage, and not just that it is a new technique for no clear reason – you are (I think) trying to get the advantages of both approaches by combining them. E.g., you get small area estimation etc., but you can additionally incorporate more ‘corrective’ information with the weights that you would not be able to incorporate into full MRP
Solon, Haider and Wooldridge have a paper on a related topic with a funny and descriptive title: “What are we weighting for?”
https://www.nber.org/system/files/working_papers/w18859/w18859.pdf
Excellent stuff(!), except for the appearance of an acronym in the title. Working in an intrinsically inter-disciplinary field (biomedical informatics), I would never use an acronym in the title, except for something like “DNA.”