I’m postponing today’s scheduled post (“Empirical implications of Empirical Implications of Theoretical Models”) to continue the lively discussion from yesterday, What if I were to stop publishing in journals?.
An example: my papers with Basbøll
Thomas Basbøll and I got into a long discussion on our blogs about business school professor Karl Weick and other cases of
plagiarism copying text without attribution. We felt it useful to take our ideas to the next level and write them up as a manuscript, which ended up being logical to split into two papers. At that point I put some effort into getting these papers published, which I eventually did: To throw away data: Plagiarism as a statistical crime went into American Scientist and When do stories work? Evidence and illustration in the social sciences will appear in Sociological Methods and Research. The second paper, in particular, took some effort to place; I got some advice from colleagues in sociology as to where to send it (and I think it got rejected from a couple other places first, but now I can’t remember if I’m confusing that with some other paper).
Anyway, here’s the question: Why did I go to the (nontrivial) effort of publication? Why wasn’t it enough to post the papers on my website and blog them?
In this case I think it comes down to audience and impact. American Scientist has a circulation of 73,000. Not all its readers read every article, but I’ve gotta be reaching some people who don’t see the blog, indeed have never heard of it.
The circulation of Sociological Methods and Research has to be a lot smaller, so this one’s a tougher call. Somehow I feel the article is more “real” if it has appeared in a journal. Is this a reasonable attitude for me to hold? I’m not sure. There’s something that would bother me about having the article in limbo, not published anywhere. Ultimately I could stick it in a book, I guess, but I always worry that when I put original ideas in a book that they might not be noticed. (I think that happened a bit with our Red State Blue State book, which contained tons of original research that I fear was ignored because people just assumed the book was a popularization of existing material.)
So I don’t know, especially about the SMR paper. The paper will get some additional readers this way, but I don’t know if it was worth the effort (indeed, we are still in the copyediting phase; there was some glitch and we haven’t heard back from the copy editors who I think are in India). The ultimate win will be if scholars in this area ultimately encounter our work and start thinking differently about the ways in which stories fit into our social science understanding. For this, publication has the advantage, I think, that people will be more likely to take our arguments seriously once the paper appears “in print.”
I could go on. I’ve written many papers that I love but have been a struggle to get published. For example, my three articles on the philosophy of Bayesian statistics. These papers express some ideas that have been bugging me for about 20 years. (I remember yammering on to someone about Bayesian model checking and falsification around 1995, and at that time I’d already been thinking about these issues for awhile, motivated in large part by disillusioning encounters I’d had in the early 1990s with Bayesians who’d just refused to check the fit of their models as a matter of principle.)
For a long time I’d wanted to write an article on the topic—I’m in the habit of considering “the article” as the basic unit of communication—but I couldn’t figure out how to get started.
Then one day in 2009 I was contacted by Harold Kincaid, the editor of the Oxford Handbook of the Philosophy of the Social Sciences, and asked to write an article on Bayesian approaches. This seemed like a great opportunity to put my ideas down and get them out there. In particular, writing for a journal is annoying because you have referees looking over your shoulder, whereas edited volumes tend to accept pretty much whatever you send them. (Not always; last year I was asked to write a chapter for an edited volume (on a completely different topic), and what I sent them wasn’t quite what they were looking for, and my collaborator and I didn’t feel like making major revisions. So that does happen sometimes.)
Anyway, I wrote this philosophy article and I really liked it, enough so that it seemed worth submitting something to a journal. I felt the article needed something more, though, as I hadn’t read anything in the philosophy literature after Lakatos (1970). So I approached Cosma Shalizi, a person I’d never met (I think) but who I knew from his writings was interested in the philosophy of statistics. Cosma added some useful things and we put together a paper, which we got several firm rejections after submission to several statistics journals. At that point we really weren’t sure what to do with the paper, but then I was contacted by Thom Baguley, who’d seen me moan about this on the blog and suggested we submit it to the British Journal of Mathematical and Statistical Psychology, a journal he edited. And then it eventually appeared with discussion. I do think all this got our article more attention and gave it some legitimacy, to the extent that maybe our ideas will have broader influence.
Two papers with Guido
Next I’ll discuss two articles that are currently in limbo. The articles concern econometrics and are in collaboration with Guido Imbens (in both cases because I wrote something that seemed to me to need something more, and I contacted Guido asking if he’d be interested in collaborating).
Why ask why? Forward causal inference and reverse causal questions arose from some things I’ve been thinking about for many years, concerning how to integrate “why” questions into statistical frameworks for causal inference. The conceptual breakthrough came when I realized that the resolution of a Why question did not have to take the form of an answer to the question. Guido and I passed this paper back and forth, fixing it up in various ways, then I posted it and it got a lot of attention. A publisher even contacted Guido and me to see if we’d like to write a book on the topic. So the paper got attention. But, for similar reasons as discussed in the example above, I wanted to publish it in a journal. We’re still in the midst of this process. Responding to reviewer and editor suggestions is definitely making this a better paper, mostly in that we’re learning ways in which our argument can be misinterpreted, and we’re getting the opportunity to better connect our ideas to the many literatures on causal inference and statistics.
The other paper, Evidence on the deleterious impact of sustained use of polynomial regression on causal inference, derives directly from a blog post. I think Guido’s forthcoming contribution on this paper will have a big impact. As it is, I have an example and some impressions. I’m hoping Guido will be able to add the theory that makes this all fit together. And, again, I don’t really feel this paper will have its full impact until after it is published in a journal.
A world without journals?
Now consider a world in which research journals have disappeared, having been replaced by some sort of beefed-up Arxiv or SSRN (or maybe Plos-One is a better model in that authors would then have to spend a few hundred bucks to defray the cost of organizing review reports).
Where then would these projects stand? I’d write these papers and post them and blog on them, just as before. I imagine that review would happen much more in a “post-publication” or “during-publication” sense, in that I’d post, then get comments, then revise, etc., but the location of the paper would stand in place, rather than moving from preprint to published article.
What about the larger benefits of exposure and indexing? In the super-SSRN world, all posted papers would get listed on Google scholar the way published articles do now, so indexing would be no problem.
But what about exposure? I’m on record as hoping that, in the future, organizations such as the Journal of the American Statistical Association or the American Journal of Sociology, will repurpose themselves, changing from publishers to recommendation systems. Instead of publishing X articles every quarter, or even Y articles every week, the volunteer editorial board of JASA (along with its volunteer referees) will choose X (or maybe 2X) articles every week to promote, out of all the articles appearing that period in this Arxiv/SSRN/Plos-One space. There could even be a mechanism for “submitting” to JASA so they’d consider promoting the paper.
I’ll conclude by discussing how things work now in some fields.
As some commenters noted, statistics is a somewhat decentralized field, which means that applied statisticians such as myself don’t have much difficulty publishing in journals, as we can choose among various outlets in statistics and applied areas. Even within statistics, there’s no clear ranking. The top journal is JASA, I think—but others would give the nod to the Annals of Statistics, or maybe Biometrika (although those latter two journals lean toward a no-applied-content rule which restricts their scope). Then there’s Biometrics, and the Royal Statistical Society, and also statistics journals in neighboring fields such as Psychometrika, Econometrica, and JMLR. Not to mention all the journals in probability theory. So statistics is not one of those fields where there’s a clear ranking of top journals.
On the other side, the transmission of statistics papers is still centered on journal publication. Various people (including me and, perhaps most notably, Radford Neal) post papers on their websites. Some things go on Arxiv but that’s a bit of a mixed bag with a lot of good stuff never getting in there.
And then there’s books. I don’t know so much about other fields, but within statistics there’s been a long history of influential books that fall somewhere between textbooks and research monographs. Perhaps the best description would be, graduate textbooks containing innovative restructuring of ideas. Books in this tradition include those of Snedecor, Cochran, Feller, Kish, Cox, McCullagh and Nelder, Tukey, Hastie and Tibshirani, Rasmussen and Williams, and, well, us. I’m sure I’ve missed a few, but you get the idea. For ideas to get disseminated, they just have to get into the influential books, somehow or another.
The other way our ideas get out there is through software. Think of all the Bayesian methods being spread through Bugs (or, now, Stan). Or the influence of lowess, or lasso. Theory helps too (there are lots of reasons for a method to be trusted), but my point here is that software implementation is a kind of publication, indeed, one of the best kinds.
Economics and political science
What about other fields? Economics seems much more well organized (for better and for worse) than statistics. They have a few journals that are almost universally agreed to be the best, and it seems that papers published in these places automatically get attention.
Parallel to this is a network of popular economics bloggers that has no real parallel in any other academic field. I mean, sure, if I happen to blog a new paper, people will read it, but I’m just one guy and I don’t try to cover the field or even all of Bayesian statistics. In contrast, economics has a bunch of blogs that are a lot more popular than ours and which regularly plug and argue about new published work.
Political science doesn’t have this density of blogging but it does have a recognized set of top journals.
The other thing that helps is that economists and political scientists typically publish less than statisticians, or at least it seems that way to me. In these social science fields, a paper gets workshopped for awhile before submission, then there can be a grueling review process. This is a pain in the ass but it does have the effect of reducing the rate at which papers get published.
Still, a lot of work in social science, especially in policy analysis, never gets published in journals. There’s lots of information in Gallup reports, Pew reports, various documents prepared by organizations doing studies in different countries, etc etc. This stuff gets emailed around but can be hard to find if you don’t know where to look.
I get the impression that developmental psychology (the subfield in which my sister works) has a somewhat old-fashioned system that still works ok, a sort of genteel tradition in which research claims are understated rather than hyped, and there is a sense of obligation when submitting to the top journals to send in serious papers. But then there’s the whole parallel world of “Psychology Science”-type papers and the culture clash that arises when people naively think that papers published in that top journal must therefore be of top quality. Right now the field of psychology is doing a lot of introspection about publication and scientific evidence, and I think that’s a good thing.
Biology and medicine
Nature, Science, Cell . . . the tabloids. Big pressure, big money, big dramatic claims, headlines in the New York Times.
A parallel world of industry-funded studies, so much is published that there are journals that publish nothing but review after review after review of published studies, automated tools for parsing the literature . . .
I have no idea how to think about this world.
Just please, please, keep these people as far away from social science as possible. There are whole sections of the Arxiv that could be fruitfully redirected to /dev/null/