I followed this link from Chris Blattman to an article by economist Roland Fryer, who writes:
I [Fryer] find no evidence that teacher incentives increase student performance, attendance, or graduation, nor do I find any evidence that the incentives change student or teacher behavior.
What struck me were not the findings (which, as Fryer notes in his article, are plausible enough) but the use of the word “I” rather than “we.” A field experiment is a big deal, and I was surprised to read that Fryer did it all by himself!
Here’s the note of acknowledgments (on the first page of the article):
This project would not have been possible without the leadership and support of Joel Klein. I am also grateful to Jennifer Bell-Ellwanger, Joanna Cannon, and Dominique West for their cooperation in collecting the data necessary for this project, and to my colleagues Edward Glaeser, Richard Holden, and Lawrence Katz for helpful comments and discussions. Vilsa E. Curto, Meghan L. Howard, Won Hee Park, Jörg Spenkuch, David Toniatti, Rucha Vankudre, and Martha Woerner provided excellent research assistance.
Joel Klein was the schools chancellor so I assume he wasn’t deeply involved in the study; his role was presumably to give it his OK. I’m surprised that none of the other people ended up as coauthors on the paper. But I guess it makes sense: My colleagues and I will write a paper based on survey data without involving the data collectors as coauthors, so why not do this with experimental data too? I guess I just find field experiments so intimidating that I can’t imagine writing an applied paper on the topic without a lot of serious collaboration. (And, yes, I feel bad that it was only my name on the cover of Red State, Blue State, given that the book had five authors.) Perhaps the implicit rules about coauthorship are different in economics than in political science.
P.S. I was confused by one other thing in Fryer’s article. On page 1, it says:
Despite these reforms to increase achievement, Figure 1 demonstrates that test scores have been largely constant over the past thirty years.
Here’s Figure 1:
Once you get around the confusingly-labeled lines and the mass of white space on the top and bottom of each graph, you see that math scores have improved a lot! Since 1978, fourth-grade math scores have gone up so much that they’re halfway to where eighth grade scores were in 1978. Eighth grade scores also have increased substantially, and twelfth-grade scores have gone up too (although not by as much). Nothing much has happened with reading scores, though. Perhaps Fryer just forgot to add the word “reading” in the sentence above. Or maybe something else is going on in Figure 1 that I missed. I only wish that he’d presented the rest of his results graphically. Even a sloppy graph is a lot easier for me to follow than a table full of numbers presented to three decimal places. I know Fryer can do better; his previous papers had excellent graphs (see here and here).
I remember noticing an interesting difference between co-authorship norms in much of physics, on the one hand, and math and (I believe) string theory, on the other. In the latter, the author list is alphabetical; in the former, it implies an assessment of who put in the work — so e.g., it is assumed that the first and second authors did most of the actual work, and the very last author was the primary supervisor. This appears to have implications for one's willingness to add peripherally involved collaborators as co-authors on a paper: the middle of an author list in condensed matter physics — my research area — can be enlarged almost arbitrarily at no real cost to author 1.
Actually, Fryer is breaking with economics' social norms by not having any of his RAs as coauthors. It is standard for even simple field experiments to include the RAs as coauthors – they often do all the heavy lifting and on the ground organization!
To spread a little gossip, people often grumble about this exact issue after Fryer's seminars. He often answers questions by saying "I'm not sure – I'll have to ask the RA that ran that regression" while using first person in the rest of the presentation and claiming sole authorship of the paper being presented.
@Andy
It is still generally the case that research assistants are rarely listed as coauthors; and that the number of authors on economic articles is typically very low (I'd say — from casual estimation from my memories — that two is the median, and less than 10% have four or more.)
I think a reason for this MIGHT be that in economics, authors are cited alphabetically (people with A as first letter of last name are cited first); and professors are probably reluctant to having their names cited after the name of their PhD student. This probably contributed to this low coauthor count equilibrium. Where this alphabetical convention comes from, I know not.
Based on this last comment I propose whether a PhD student's last name is before or after their advisor's alphabetically as an instrument for number of journal articles published. (Of course, the exclusion restriction isn't quite satisfied…)
Hey–nobody commented on the "largely constant" math test scores!
@Tim:
You're right in general. I'm a soon-to-be-minted PhD economist and so would like to think that I know about social norms in econ. Low author counts are definitely the norm. I would say it's partly because of the alphabetical naming convention and partly because we're just in an equilibrium where everyone gets suspicious if we see a lot of authors – people just suspect that some of the listed people didn't really contribute and thus discount the value of any of the authors.
What my point was is that Fryer widely perceived as violating these already conservative norms within economics by omitting contributing RAs. Mostly that's because experimental economics papers – especially field experiments – require a ton of organization and hard work from RAs and the lead RA is almost always given a coauthorship. And then Fryer admitting that the RAs ran all the regressions and know the details… The sole author should know better!
@Andrew – you're absolutely right about the math test scores. I would put a large sum of money that he says that because there isn't a statistical difference between the math test scores at the beginning and end of the sample. Even though the trend is obvious to the eye and you could probably find a significant trend coefficient with a simple regression.
Andy:
NAEP has a big sample so I think the differences are statistically significant. I think there must be something else going on. For example, maybe Fryer was aware of a more sophisticated analysis that shows little improvement in test scores, and then he just pointed to the graph without realizing that the point wasn't obvious by looking at the raw data.
I agree with Tim that the paper is not (obviously anyways) breaking norms in Economics.
I also agree that this is likely an artifact of the convention in Economics of listing authors alphabetically, which means that there is no way to signal relative contributions through author order.
"Heavy lifting and ground organization" during data collection are not substantial intellectual contributions and rate acknowledgements, not coauthor status. That distinction is not just a norm but rather explicitly imposed at many journals. For example, the (flagrantly violated) ethical rules for coauthorship at most medical journals are clear that "acquisition of funding, collection of data, or general supervision of the research group alone does not constitute authorship."
That is an attempt to suppress "coauthorship inflation," the tendency to list a bazillion "coauthors" on every paper. The lack of a transparent way to signal relative contributions in econ journals is sometimes frustrating, but it does have the benefit of not generating incentives to list everyone from the RAs to the lead author's cat as coauthors.
Chris:
Yes, I'm not saying Fryer was doing anything wrong, just that he's clearly following much different rules than I do, in using the first person to describe collaborative work.
I just watched the film "Wating For Superman" and they also said there were no improvement on scores. But I didn't notice anything wrong as well and just noticed in the graphic you pointed after reading what you said.
Is it possible that he saw what he wanted to saw and didn't properly test this claim? It seems to be common sense now, and maybe he just saw what he wanted see.
I know, this would be a bit silly, but who knows?
I think this paper is likely to be misinterpreted by people who read just the abstract.
In the abstract, Fryer writes: "I find no evidence that teacher incentives increase student performance."
Later, he reveals that "treatment teachers were not significantly more likely to fill out school surveys" that would increase their chances of receiving a bonus.
If teachers eligible for a possible bonus couldn't even be bothered to fill out a survey, it seems unlikely that this incentive affected their behavior in other ways.
I suspect that each teacher felt that his or her individual efforts could not increase the likelihood of a bonus, because bonuses were paid or not paid to a teacher depending on the performance of the school as a whole, not on the performance of the teacher's particular class.
So the reason that the incentives had no effect is that the teachers perceived no incentive.
Figure 3 in the paper says percent, means fraction.
"Heavy lifting and ground organization" during data collection are not substantial intellectual contributions and rate acknowledgements, not coauthor status.
The reason the biomedical field ignores this requirement or works around it is partly because of training. It takes a ton of training and long-term commitment to do experimental work; you'd like the person doing the experiment to be able to adapt and understand rather than be a lab drone who bothers you every day. As a result, that work is done by grad students who need to publish.
The work around is that the people who did the experiment are allowed to write the paper. That's really how it should be; if you did not heavily involve the person who was on the ground doing the experiment in the planning, analysis, and write-up you have probably done a bad job.
In all the list-of-people-who-are-authors requirements that I've actually read "did the analysis" qualifies you. Not including the person who knows the details of the analysis as an author is strange.
These matters do vary enormously between fields.
I've encountered the complete opposite in which the sheer extended labour (even discomfort and danger) entailed by field data collection is always regarded as worthy of co-authorship, but seriously contributing to data analysis is sometimes regarded as minor. The tacit argument is that the time commitment is often slight, but that ignores the time spent acquiring the expertise to give good advice on what to do and what not to do.
It occurred to me that demonstrating these social norms in experimental economics might be useful. If you look at
Esther Duflo – <a href="http://econ -www.mit.edu/faculty/eduflo/papers” target=”_blank”>http://econ <a href="http://-www.mit.edu/faculty/…” target=”_blank”>-www.mit.edu/faculty/eduflo/papers
John List – http://home.uchicago.edu/~jlist/research2/researc… (you have to click the topics to see the papers)
who are 2 of the biggest names in experimental economics, you can see that virtually ALL of their papers include a graduate student coauthor. That coauthor was more than just an RA – I'm positive they did most of the organizational heavy-lifting, which left them prepared to add the most value in data analysis. That's the way it works in experimental econ.
In biology, the system's three tiered, with "first authors" being the ones making the "scientific contribution", "middle authors" doing the lab work, and "last authors" being lab directors (i.e., PIs/fundraisers). They're not alphabetical within these sublists, either. Some journals require descriptive text laying out what each author contributed.
A paper Mitzi was just on had around 100 authors, including over a dozen "first" authors. There were around eleven institutions involved, and this was only the nematode side of a larger sequencing project.
@Andy and Ryan: I agree it is probably a good idea to have the people who did the grunt work also contribute heavily to the analysis, but that doesn't necessarily happen. If Fryer's RAs were really just RAs and did not contribute to the design, analysis, nor write-up, then they ought not be listed as coauthors regardless of how much hard work they did. In economics, including experimental, they typically would not be so listed because the cost of adding more names to the paper is large. For example, the first Duflo paper she lists is coauthored with one former grad student, but seven more are listed in the acknowledgements, not as coauthors.
I'm skeptical about what "largely constant" math scores from 1978 to 2005 would even mean. Wikipedia says the NAEP assessment "stays essentially the same from year to year, with only carefully documented changes." But how relevant in 2005 is a math assessment tool from 1978? Haven't math educational goals changed out from under the NAEP, for example an increased emphasis on probability & statistics?
You have some nerve complaining about coautorship norms of others!
Wow:
The trolls never seem to respond, but just in case you're being sincere . . . I'm not complaining, just commenting on differences between Fryer's field and my own.
As long as you remain within your field, and therefore publish under the same authorship rules consistently, then you should be okay. In my experience, it is when you switch fields that you run into problems. For example, when I was an undergraduate, I spend many, many hours working on data collection for several research projects that ultimately ended up as manuscripts: I got an 'acknowledgment', whereas many of my classmates who did essentially the same thing — but in a medical lab — were awarded authorship (eg., 6th author out of a list of 12 authors). I published my undergraduate thesis with my supervisor in an economics journal. We did the usual shopping around at economics seminars. Along the way, we picked up another author. I was reluctant to do this because her last name placed her in front of me alphabetically, so now instead of being 2nd of 2, I was 3rd of 3. In economics, a 3-author paper has slightly less value than a 2-author paper, but the value doesn't drop off dramatically. But by this time, I knew I was headed into medicine– and the value of 2nd, and then 3rd, authorship drops off dramatically after 1st.
Now I am in a situation where the old rules (economics) of authorship do not apply. In medicine, the grunt work of data collection generally results in automatic and substantial authorship awards. And not just for experimental studies. If a senior researcher directs the collection of data on a population-based cross-sectional sample, or on a cohort, then she often gets authorship irrespective of whether she actually contributes to the paper. Thus, cohort studies like the Nurses Health Study (NHS) are a gold mine for the Principal Investigators, because they get authorship on virtually every paper that comes out of it even if they have nothing to do with the actual paper. In most cases, this gift authorship is not so egregious, because the analyses are not that complicated (eg., Cox regression with time-invariant covariates, for which NHS maintains a database of old SAS code that you can just modify and re-run whenever you have a new study idea (eg., does aspirin influence mortality risk; does diet influence diabetes risk; etc). But when the statistical method is far beyond the reach of the PI, for example a marginal structural model, then it really does rankle me to no end because I can't possibly see how the data collectors could have contributed. But that is how the rules are.
If you are first author, then in medicine/epidemiology it is relatively costless to add a long train of authors behind yourself. But if you switch over to economics, then it is costly to have 3 authors on a paper when you are listed as 'third'. So not only do you end up as 'third author' (which means next to nothing from the perspective of a medicine department chair), now I have to also explain to my chair why I am bothering to publish in a piddly journal like the American Economic Review that has an impact factors of 3.