More on publishing in journals

I’m postponing today’s scheduled post (“Empirical implications of Empirical Implications of Theoretical Models”) to continue the lively discussion from yesterday, What if I were to stop publishing in journals?.

An example: my papers with Basbøll

Thomas Basbøll and I got into a long discussion on our blogs about business school professor Karl Weick and other cases of plagiarism copying text without attribution. We felt it useful to take our ideas to the next level and write them up as a manuscript, which ended up being logical to split into two papers. At that point I put some effort into getting these papers published, which I eventually did: To throw away data: Plagiarism as a statistical crime went into American Scientist and When do stories work? Evidence and illustration in the social sciences will appear in Sociological Methods and Research. The second paper, in particular, took some effort to place; I got some advice from colleagues in sociology as to where to send it (and I think it got rejected from a couple other places first, but now I can’t remember if I’m confusing that with some other paper).

Anyway, here’s the question: Why did I go to the (nontrivial) effort of publication? Why wasn’t it enough to post the papers on my website and blog them?

In this case I think it comes down to audience and impact. American Scientist has a circulation of 73,000. Not all its readers read every article, but I’ve gotta be reaching some people who don’t see the blog, indeed have never heard of it.

The circulation of Sociological Methods and Research has to be a lot smaller, so this one’s a tougher call. Somehow I feel the article is more “real” if it has appeared in a journal. Is this a reasonable attitude for me to hold? I’m not sure. There’s something that would bother me about having the article in limbo, not published anywhere. Ultimately I could stick it in a book, I guess, but I always worry that when I put original ideas in a book that they might not be noticed. (I think that happened a bit with our Red State Blue State book, which contained tons of original research that I fear was ignored because people just assumed the book was a popularization of existing material.)

So I don’t know, especially about the SMR paper. The paper will get some additional readers this way, but I don’t know if it was worth the effort (indeed, we are still in the copyediting phase; there was some glitch and we haven’t heard back from the copy editors who I think are in India). The ultimate win will be if scholars in this area ultimately encounter our work and start thinking differently about the ways in which stories fit into our social science understanding. For this, publication has the advantage, I think, that people will be more likely to take our arguments seriously once the paper appears “in print.”

Other examples

I could go on. I’ve written many papers that I love but have been a struggle to get published. For example, my three articles on the philosophy of Bayesian statistics. These papers express some ideas that have been bugging me for about 20 years. (I remember yammering on to someone about Bayesian model checking and falsification around 1995, and at that time I’d already been thinking about these issues for awhile, motivated in large part by disillusioning encounters I’d had in the early 1990s with Bayesians who’d just refused to check the fit of their models as a matter of principle.)

For a long time I’d wanted to write an article on the topic—I’m in the habit of considering “the article” as the basic unit of communication—but I couldn’t figure out how to get started.

Then one day in 2009 I was contacted by Harold Kincaid, the editor of the Oxford Handbook of the Philosophy of the Social Sciences, and asked to write an article on Bayesian approaches. This seemed like a great opportunity to put my ideas down and get them out there. In particular, writing for a journal is annoying because you have referees looking over your shoulder, whereas edited volumes tend to accept pretty much whatever you send them. (Not always; last year I was asked to write a chapter for an edited volume (on a completely different topic), and what I sent them wasn’t quite what they were looking for, and my collaborator and I didn’t feel like making major revisions. So that does happen sometimes.)

Anyway, I wrote this philosophy article and I really liked it, enough so that it seemed worth submitting something to a journal. I felt the article needed something more, though, as I hadn’t read anything in the philosophy literature after Lakatos (1970). So I approached Cosma Shalizi, a person I’d never met (I think) but who I knew from his writings was interested in the philosophy of statistics. Cosma added some useful things and we put together a paper, which we got several firm rejections after submission to several statistics journals. At that point we really weren’t sure what to do with the paper, but then I was contacted by Thom Baguley, who’d seen me moan about this on the blog and suggested we submit it to the British Journal of Mathematical and Statistical Psychology, a journal he edited. And then it eventually appeared with discussion. I do think all this got our article more attention and gave it some legitimacy, to the extent that maybe our ideas will have broader influence.

Two papers with Guido

Next I’ll discuss two articles that are currently in limbo. The articles concern econometrics and are in collaboration with Guido Imbens (in both cases because I wrote something that seemed to me to need something more, and I contacted Guido asking if he’d be interested in collaborating).

Why ask why? Forward causal inference and reverse causal questions arose from some things I’ve been thinking about for many years, concerning how to integrate “why” questions into statistical frameworks for causal inference. The conceptual breakthrough came when I realized that the resolution of a Why question did not have to take the form of an answer to the question. Guido and I passed this paper back and forth, fixing it up in various ways, then I posted it and it got a lot of attention. A publisher even contacted Guido and me to see if we’d like to write a book on the topic. So the paper got attention. But, for similar reasons as discussed in the example above, I wanted to publish it in a journal. We’re still in the midst of this process. Responding to reviewer and editor suggestions is definitely making this a better paper, mostly in that we’re learning ways in which our argument can be misinterpreted, and we’re getting the opportunity to better connect our ideas to the many literatures on causal inference and statistics.

The other paper, Evidence on the deleterious impact of sustained use of polynomial regression on causal inference, derives directly from a blog post. I think Guido’s forthcoming contribution on this paper will have a big impact. As it is, I have an example and some impressions. I’m hoping Guido will be able to add the theory that makes this all fit together. And, again, I don’t really feel this paper will have its full impact until after it is published in a journal.

A world without journals?

Now consider a world in which research journals have disappeared, having been replaced by some sort of beefed-up Arxiv or SSRN (or maybe Plos-One is a better model in that authors would then have to spend a few hundred bucks to defray the cost of organizing review reports).

Where then would these projects stand? I’d write these papers and post them and blog on them, just as before. I imagine that review would happen much more in a “post-publication” or “during-publication” sense, in that I’d post, then get comments, then revise, etc., but the location of the paper would stand in place, rather than moving from preprint to published article.

What about the larger benefits of exposure and indexing? In the super-SSRN world, all posted papers would get listed on Google scholar the way published articles do now, so indexing would be no problem.

But what about exposure? I’m on record as hoping that, in the future, organizations such as the Journal of the American Statistical Association or the American Journal of Sociology, will repurpose themselves, changing from publishers to recommendation systems. Instead of publishing X articles every quarter, or even Y articles every week, the volunteer editorial board of JASA (along with its volunteer referees) will choose X (or maybe 2X) articles every week to promote, out of all the articles appearing that period in this Arxiv/SSRN/Plos-One space. There could even be a mechanism for “submitting” to JASA so they’d consider promoting the paper.

I’ll conclude by discussing how things work now in some fields.

As some commenters noted, statistics is a somewhat decentralized field, which means that applied statisticians such as myself don’t have much difficulty publishing in journals, as we can choose among various outlets in statistics and applied areas. Even within statistics, there’s no clear ranking. The top journal is JASA, I think—but others would give the nod to the Annals of Statistics, or maybe Biometrika (although those latter two journals lean toward a no-applied-content rule which restricts their scope). Then there’s Biometrics, and the Royal Statistical Society, and also statistics journals in neighboring fields such as Psychometrika, Econometrica, and JMLR. Not to mention all the journals in probability theory. So statistics is not one of those fields where there’s a clear ranking of top journals.

On the other side, the transmission of statistics papers is still centered on journal publication. Various people (including me and, perhaps most notably, Radford Neal) post papers on their websites. Some things go on Arxiv but that’s a bit of a mixed bag with a lot of good stuff never getting in there.

And then there’s books. I don’t know so much about other fields, but within statistics there’s been a long history of influential books that fall somewhere between textbooks and research monographs. Perhaps the best description would be, graduate textbooks containing innovative restructuring of ideas. Books in this tradition include those of Snedecor, Cochran, Feller, Kish, Cox, McCullagh and Nelder, Tukey, Hastie and Tibshirani, Rasmussen and Williams, and, well, us. I’m sure I’ve missed a few, but you get the idea. For ideas to get disseminated, they just have to get into the influential books, somehow or another.

The other way our ideas get out there is through software. Think of all the Bayesian methods being spread through Bugs (or, now, Stan). Or the influence of lowess, or lasso. Theory helps too (there are lots of reasons for a method to be trusted), but my point here is that software implementation is a kind of publication, indeed, one of the best kinds.

Economics and political science
What about other fields? Economics seems much more well organized (for better and for worse) than statistics. They have a few journals that are almost universally agreed to be the best, and it seems that papers published in these places automatically get attention.

Parallel to this is a network of popular economics bloggers that has no real parallel in any other academic field. I mean, sure, if I happen to blog a new paper, people will read it, but I’m just one guy and I don’t try to cover the field or even all of Bayesian statistics. In contrast, economics has a bunch of blogs that are a lot more popular than ours and which regularly plug and argue about new published work.

Political science doesn’t have this density of blogging but it does have a recognized set of top journals.

The other thing that helps is that economists and political scientists typically publish less than statisticians, or at least it seems that way to me. In these social science fields, a paper gets workshopped for awhile before submission, then there can be a grueling review process. This is a pain in the ass but it does have the effect of reducing the rate at which papers get published.

Still, a lot of work in social science, especially in policy analysis, never gets published in journals. There’s lots of information in Gallup reports, Pew reports, various documents prepared by organizations doing studies in different countries, etc etc. This stuff gets emailed around but can be hard to find if you don’t know where to look.

I get the impression that developmental psychology (the subfield in which my sister works) has a somewhat old-fashioned system that still works ok, a sort of genteel tradition in which research claims are understated rather than hyped, and there is a sense of obligation when submitting to the top journals to send in serious papers. But then there’s the whole parallel world of “Psychology Science”-type papers and the culture clash that arises when people naively think that papers published in that top journal must therefore be of top quality. Right now the field of psychology is doing a lot of introspection about publication and scientific evidence, and I think that’s a good thing.

Biology and medicine
Nature, Science, Cell . . . the tabloids. Big pressure, big money, big dramatic claims, headlines in the New York Times.

A parallel world of industry-funded studies, so much is published that there are journals that publish nothing but review after review after review of published studies, automated tools for parsing the literature . . .

I have no idea how to think about this world.

Just please, please, keep these people as far away from social science as possible. There are whole sections of the Arxiv that could be fruitfully redirected to /dev/null/

22 thoughts on “More on publishing in journals

  1. “In this case I think it comes down to audience and impact.”

    Saying your impact is 73,000 because you published in some journal is about like saying, my exposure to the movers and shakers in the world is enormous because I paid $75 to get my picture in the Who’s Who of high school students.

    If your research and reviews/screenings/commenting on other work is good, then this provides an incredibly valuable service to others and your impact from your personal outlets will easy outstrip 73,000. If it isn’t that good, then how much impact should it have?

    Getting official papers published in official journals is a kind of safety net. It’s a way of making Academic’s life/career more predictable. It’s always possible to publish even when fields stagnate, so it provides a safe way to keep your job even though you contribute nothing to society. Getting a paper published is far more doable and predictable than a real advance, since to publish you need only convince some knucklehead, but a real advance requires wrestling truths from an unforgiving Universe.

    Publishing in a journal is a crutch. It’s part of the Academic safety net.

    95% of Academics have nagging doubts that they’re a fraud and an imposter. And for good reason; they almost certainly are. They know they’re coasting on what school they went to, or their contacts, or their ability to bamboozle taxpayers out of money, or manipulate the University (medieval guild) system. They know no one’s reading their work and those that do read it, see right through it.

    Most Academics will never get to experience the peculiar satisfaction of standing on their own two feet without some artificial system propping them up, of eating what they kill, of no longer being a fraud, or even for that matter, having science be that fun exciting purposeful thing they imagined it was when they were younger. Most will never know what it’s like to really control their research destiny during their most productive years. Or even what it’s like not to spend the bulk of their days playing status games or manipulating the system for money.

    What most Academics need is fewer crutches. Fever safety nets.

    • By “audience,” I mean that people read my article in American Scientist who would not find it on this blog.

      By “impact,” I mean that researchers and practitioners are more likely to use my results if they’ve been published in a journal.

      I’m not doing this to make my life more predictable, and I keep my job even if I don’t publish. That’s why I’m trying to explore here the reasons why I do go to the trouble of publishing in journals.

      • > more likely to use my results if they’ve been published in a journal.

        Certainly are less barriers if one can quote a published reference.
        (I have been informed numerous times that methods in my thesis cannot be used or referenced in methodology review papers as there is no journal publication.)

        It makes some sense, otherwise you are asking the reviewers/editor and readers trust _you_ as being able to discern the method developer knew what they were doing and could convience their peers.

      • Suppose you stop publishing in any location anywhere. Wouldn’t this affect your ability to get grants? And without grants, wouldn’t this affect the kind of work you do? Sure, you could teach you heart out… but I doubt after a while that you’d get grants to make Stan better, or to study voting patterns, or to improve methods in pharmicokinetics or whatever. So although you keep your source of income, you don’t keep all the benefits that you have with your publications.

        Now, taken to a much more extreme level, in Biology for example you go from running a lab with tens of people working for you doing hands-on experiments and generating data…. to a lonely corner office with nothing but undergraduate pre-med students complaining at you that your tests are too hard and are ruining their lives… it’s a very big loss in quality of life even if you keep your source of income, you don’t “keep your job”.

        So I think at the fundamental level, the big issue is not publication as way of getting tenure, but publication as way of convincing granting agencies to fund you.

        • Daniel: And this is multiplied by funds for employees and overheads paid to Universities.

          (In fact, I am aware of very successful grant _getters_ telling University vice-presidents what they need to do for them to keep them happy enough to not take there grant money elsewhere.)

        • Daniel:

          Sure, if I published nothing at all, I could find it difficult to get grants. That’s in point 6 of my earlier post:

          I should have some minimum number of publications that I can mention in progress reports for research grants and visiting positions.

          But that minimum number is something like one or two per year, nothing like my usual volume.

      • Your best argument against the idea that Academics have too many safety nets is:

        “and I keep my job even if I don’t publish.”

        There’s no arguing with that.

        It’s not all about you Andrew. I only address it to you because you are in a far better position than most.

        And maybe you’d do so if things weren’t so comfortable. Maybe lots of academics would if they weren’t encased in a protected position. Maybe they need a little more uncertainty, a little less reliance on pieces of papers and official approval or the stamp of correctness from some “peer”, a little more uneasiness might put a fire in their belly.

        Academia is a massive failure. The US spends over $350 billion on R&D every year and gets stunningly little for it. And at this point in the corruption cycle, Academia is significantly more likely to damage a Ph.D. candidate’s life than improve it. If you’re young and at all serious about your research then the last thing you should do is stay in academia. You’re more likely to create an environment conducive to work and generate the free time needed to work outside of Academia than you are inside.

        It needs to change. Tweaking journal submission policies or reviewing gimmicks doesn’t cut it. Better to tear the whole Journal/promotion process down and see what happens. It might be hard to predict exactly how that will play out, but I can tell you what won’t happen though: researchers won’t stop communicating with each other and they won’t stop recognizing good work when they see it. So I’m not sure what the big fear is.

        So Andrew, tear down this wall of Journals!

      • Also, forget about existing solutions and forget about trying to think of ways to improve arxive or whatever. The best thing to do would be to take all these points you’ve been making to someone like those guys that did Stack Exchange, or Github. Hand them a list of points saying “this is why I publish in journals”.

        Then let them figure out an open online system that would lure you away from your current habbits. The pro-Journal publishing points you’ve been making are perfect market/user research for those guys.

  2. As you speak of the your paper with Cosam Shalizi – and to give you quick proof read up front – the article is missing in the references.

  3. Andrew:

    Once all papers are in open source, machine readable databases I am sure all manner of individually customized recommender systems will spring up. I’m talking reaching orders of magnitude more -interested- readers than through a journal.

    For me the main reason to publish in a major journal is the belief it will archive your work, and still be there, after you kick over.

    Yet increasingly my money is on repositories.

    I think

  4. About that Chen at al. paper. (Re: Evidence on the deleterious impact of sustained use of polynomial regression on causal inference) I know you posted about it earlier, but I just thought of 2 new comments. 1) There are two locales with ~-11 and +10 degrees north of the river with incredibly high lifespans of about 88 and over 90 years. The next best result is about 81. What is known about those communities? Do they have some sort of nursing homes with disproportionately large numbers of very old people? 2) I think one of the checks of the polynomial discontinuity model should be this. They use cubic polynomials on both sides of the divide which gives them 4+4=8 fitting parameters. As a check they should have applied 7-degree polynomial fitting to the whole data set and compare the quality of fits. Alternatively, they might have used any other fitting technique (spline interpolation, maybe) with the same number of parameters and forced continuity to check for the H0. OK, this a Bayesian place so let’s call it a reasonableness check without reference to any kind of statistical theory.

  5. Reasons why it makes sense for an academic in the current system to publish:
    1. Career (not applicable to you as you mention)
    2. Grants
    3. Because there are incentives to publish a paper, not publishing a paper is a signal of low paper quality (why would you not publish a good paper).

    These don’t necessarily defend the system but give good reasons to publish given that you are in this system.

  6. Andrew,

    Your discussion on the publishing differences across academic fields is very interesting. I agree with you that the peer-review process in economics and political science is indeed a “pain in the ass.” But having no experience in trying to publish beyond these two fields, I didn’t realize that they may be more “grueling” than others. If they are, what do you think accounts for this difference?

    I’m also curious where sociology fits. My sense is that its process may be similar to economics and political science–the prolonged workshop approach–but I’m not sure. Do you have any insights?

  7. Pingback: Friday links: PhD models, great Tony Ives interview, shameless self-promotion, and more | Dynamic Ecology

  8. Ent: Don’t let the bitterness consume you. I do that to myself a lot, man. The retraction movement is a good one, but don’t lose sight of how many people do good work, how many editors are not spending most of their time being cronies, etc. There is joy (and efficacy) in just contributing. It can ruin the spirit to be too negative and it takes your eye off the ball.

    Bob: Chemistry, but unrevised publication is not the norm (my advisor, an editor, had never seen it). I really don’t think it is quite so hard as people think though. Just be candid, be descriptive, and don’t try to act all smart and all. More like writing up a college lab experiment than trying to look like Pasteur or Reimann (or worse, Edison).

    Andrew: You can treat Nature/Science/PRL the same way. Just reserve submissions there for things that are noteworthy: ‘Science doesn’t print non-superconducting cuprates’. Capisce?

    All: Many of these issues are far from new, to science as a field or to scientists as conflicted humans. Please read the very short pamphlet by Katzoff (public domain) on Clarity in Technical Reporting. In addition, some of the chapters by E. Bright Wilson in An Introduction to Scientific Research are very appropriate here.

    P.s. FWIW, I’m a big advocate of publishing everything. HATE the 20 paper limit. (and really, the number of first author papers is what matters…that’s who did the work!) If it’s not published, it’s lost work. Work done at taxpayer expense. You don’t even really understand your own work until you summarize and synthesize. And it does no one else any good. Nothing wrong with simple, constructive work (“stamp collecting”). And “grinding your pigments” will make you a better artist.

      • One of the posters here said that people should be limited to no more than 20 papers in their life. Cutting down on chaff in the literature or something.

        I could write more than 20 good technical reports in my life. And a good tech report is a good paper. So I could write more than 20 papers.

        And when you add in people who collaborate (e.g. crystallographers) or who are supervisors, than obviously one could have a lot of papers as a co-author (not the main (i.e. first) author).

Leave a Reply

Your email address will not be published.