Why edit a journal? More generally, how to contribute to scientific discussion?

The other day I wrote:

Journal editing is a volunteer job, and people sign up for it because they want to publish exciting new work, or maybe because they enjoy the power trip, or maybe out of a sense of duty—but, in any case, they typically aren’t in it for the controversy.

Jon Baron, editor of the journal Judgment and Decision Making, saw this and wrote:

In my case, the reasons are “all three”! But it isn’t a matter of “exciting new work” so much as “solid work with warranted conclusions, even if boring”. This is a very old-fashioned experimental psychologist’s approach. Boring is good. And the “power” is not a trivial consideration; many things that academics do have the purpose of influencing their fields, and editing, for me, beats teaching, blogging, writing trade books, giving talks, or even . . . (although it does not beat writing a textbook).

I’ve been asked many times to edit journals but I’ve always said no because I’ve felt that, personally, I can make better contributions to the field as a loner. Editing a journal would require too much social skill for me. We each should contribute where we can.

Also recall this story:

I remember, close to 20 years ago, an economist friend of mine was despairing of the inefficiencies of the traditional system of review, and he decided to do something about it: He created his own system of journals. They were all online (a relatively new thing at the time), with an innovative transactional system of reviewing (as I recall, every time you submitted an article you were implicitly agreeing to review three articles by others) and a multi-tier acceptance system, so that very few papers got rejected; instead they were just binned into four quality levels. And all the papers were open-access or something like that.

The system was pretty cool, but for some reason it didn’t catch on—I guess that, like many such systems, it relied a lot on continuing volunteer efforts of its founder, and perhaps he just got tired of running an online publishing empire, and the whole thing kinda fell apart. The journals lost all their innovative aspects and became just one more set of social science publishing outlets. My friend ended up selling his group of journals to a traditional for-profit company, they were no longer free, etc. It was like the whole thing never happened.

A noble experiment, but not self-sustaining. Which was too bad, given that he’d put so much effort into building a self-sustaining structure.

Perhaps one lesson from my friend’s unfortunate experience is that it’s not enough to build a structure; you also need to build a community.

Another lesson is that maybe it can help to lean on some existing institution. This guy built up his whole online publishing company from scratch, which was kinda cool, but then when he no longer felt like running it, it dissolved, and then he ended up with a pile of money, which he probably didn’t need and he might never get around to spending, while losing the scientific influence, which is more interesting and important. Maybe it would’ve been better for him to have teamed up with an economics society, or with some university, governmental body, or public-interest organization.

Good intentions are not enough, and even good intentions + a lot of effort aren’t enough. You have to work with existing institutions, or create your own. This blog works in part because it piggybacks off the existing institution of blogging. Nowadays there isn’t much blogging anymore, but the circa 2005-era blogosphere was helpful in giving us a sense of how to set up our community. We built upon the strengths of the blogosphere and avoided some of the pitfalls.

Similarly this is the challenge of reforming scientific communication: to do something better while making use of existing institutions and channels whereby researchers donate their labor.

26 thoughts on “Why edit a journal? More generally, how to contribute to scientific discussion?

  1. Quote from the blogpost:

    “You have to work with existing institutions, or create your own. This blog works in part because it piggybacks off the existing institution of blogging.(…) Similarly this is the challenge of reforming scientific communication: to do something better while making use of existing institutions and channels whereby researchers donate their labor.”

    This reminded me of a quote by Bruce Lee: “Adapt what is useful, reject what is useless, and add what is specifically your own”.

    I actually think this quote by Bruce Lee (in some way at least) fits with how i view science should work. It may also be in line with another quote of the blogpost: “I’ve been asked many times to edit journals but I’ve always said no because I’ve felt that, personally, I can make better contributions to the field as a loner. Editing a journal would require too much social skill for me. We each should contribute where we can.”

  2. I edit a small quarterly that’s owned by a scholarly society and published by one of the big commercial academic publishers. I do it mostly because it’s a way to serve authors and readers. The older I get the more important I think this service is (and the less important I think my own research is). Some articles don’t need much help from reviewers or editors, but many articles we publish are a *lot* better after getting feedback from reviewers and advice from editors. The work we do is especially helpful for authors from developing-world institutions who are still figuring out publishing norms in our field and need advice on how to write the best article possible based on the data. A lot of that work involves improving the English language usage so that readers can clearly understand the discoveries in the article. Like Jon Baron’s experience, a lot of those discoveries seem to me to be conventional (or boring, depending on your POV), but that’s ok. Unlike Baron, I don’t get much of a power surge from the experience. Most of the articles we publish could be published elsewhere. I do occasionally get to reject a manuscript that I think really shouldn’t be published *anywhere* because it includes mistaken or misleading conclusions not supported by data. I think of that as a service to the community as well. It’s also fun: I get insight into the creative thinking of researchers from all over the world who I’ll never get to meet but whose work I get to see in its embryonic form.

    • An editor may think that a paper is not publishable *anywhere*, but the ultimate decision to publish really should lie with the authors because (1) they are free individuals (2) they often know better than the editors (and the reviewers who advise the editors).

      • Thanks for the reply. I’m late to check in on this thread again.

        “(1) they are free individuals”

        Yes, for sure, authors should decide where to publish their research. Often authors and editors disagree about publishing in any particular journal, but there are lots of journals out there. That seems ok to me. I had a spirited disagreement of that kind with an author a couple of days ago, in which I actually cited this blog as the source of good arguments against presenting tables full of hundreds of p values (and using differences in p values as a stand-in for differences in effects).

        “(2) they often know better than the editors”
        Well, sure, we all have favorite stories about idiot reviewers and ham-fisted editors. One of my favorites came from a grant proposal by a friend, who proposed to analyze human genetic variation in some very-hard-to-sequence parts of the human genome by using DNA samples from hydatiform moles. This ‘mole’ is a tumor that develops in the uterus of a woman whose fertilized egg has lost its maternal genome copy, and replaces it with two copies of the paternal genome (from the sperm). Particular forms of this tumor (where the two genome copies come from the same single sperm cell) will have no genetic variation at all, making parts of their genomes much more easily sequenced. My friend explained all this in his grant proposal. In response, a reviewer asked in the written comments why my friend was proposing to study genetic variation in small mammals (ba-dum).

        But on balance my experience is that most reviewers and editors give helpful (if sometimes unwelcome) advice to authors, and that most authors benefit from that advice, even when it comes with a rejection. That’s usually been my experience with reviews of my own work, even (especially) reviews that revealed my errors in understanding or analyzing data.

    • I was on the editorial board of the Computational Linguistics journal, and it took the same approach. The result was endless revise-and-resubmit decisions where I felt more like I was managing a group of grad students than trying to decide what to publish. I always came back with either a sharp publish or reject recommendation, but my colleagues inevitably chose revise-and-resubmit. This went on so long at such a cost in delayed publication time, that the Association for Computational Linguistics sponsored a second journal, Transactions in Computational Linguistics, to try to provide an output where it didn’t take two years and several resubmissions to publish a paper.

      I had a long discussion with Michael Collins (a Columbia/Google CS researcher) when he was co-founding the journal on how long reviewers should get. I recommended two weeks to a month max. That’s based on my own experience of getting two or three months to review and then forgetting about until I got a late notice, then finishing it in the few hours or so I’ll put into a serious review (not quite as speedy as Andrew, nor as slow as a grad student). I cited journals like Bioinformatics, which managed a one month turnaround. I was probably not the only one lobbying, but they went with a month, and it’s been fantastic. I love TACL and have already published two papers in it (out of three I’ve published in the last ten years).

      • Thanks Bob. I’m not sure what “the same approach” is?

        I agree that many months of multiple review-and-revise-and-resubmit is awful. But I think publish-or-reject misses out on opportunities to help authors improve their analysis and presentation of data. My journal sees a lot of articles based on studies with interesting ideas (or at least based on work with interesting study systems), but that have many flaws in the presentation and analysis and in the English language usage. I’m not in computing science or linguistics, where maybe authors are more careful and precise in their writing and presentation of first submission, and revising would just be polishing the knobs. At my journal, revising once gives many authors a much-needed chance to benefit from reviewers and editors who see improvements to make.

        The process could be faster: our typical times from first submission to DOI are 6-8 months; in the majority of cases the longest stage in that process is the authors’ revision; we try to speed up the other steps (recruiting reviewers, writing decision letters, copy editing the accepted manuscript), but authors are slow, man!

  3. In the old days, researchers communicated by private letters, meetings and articles in academic journals. Things have moved on a lot from when the printing press was the main way of publicly disseminating information, but we still have traditional journal publishing! This seems to be because it is a system that is heavily supported by universities and of course publishers.

    I am not sure how long this system can sustain itself, but I feel that when it does go it will go dramatically with no halfway houses (of the kind that have been proposed on this blog in the last couple of days). It seems natural to suggest that it will be social media that takes over in some way, with people publishing what they want (for free!) and having their work openly criticised in social media forums (like this valuable blog).

  4. “In the old days, researchers communicated by private letters, …”

    I was around then. When I retired and had to clean out my office before giving it up, I asked the archivist at the Archives for American Mathematics what kinds of things they might like to have. One of the things she said was correspondence about work between women mathematicians (since we were few and far between when I got my Ph.D.). I still haven’t gotten the things I saved to the Archives, but do have them all collected in a box — and among them is letters (handwritten on the small stationary we used for short notes) from another woman mathematician that resulted in a joint paper.

  5. Question I always wanted to ask.

    Dear Andrew, dear readers of this blog, what, in your opinion, is the best way to contribute to Science if you doing it as a part time job?

    Lets say, I spend 4 days/week at my day job and have 1 days to do Science. What should I do? How to maximise my usefulness?

    • “Dear Andrew, dear readers of this blog, what, in your opinion, is the best way to contribute to Science if you doing it as a part time job?”

      Interesting question!

      I have been trying to contribute to (improving) science for about 7 years without being part of academia.

      I have tried to do this by:

      # Making clear to my former professors and university i felt i was not educated well (so they knew this)
      # By participating in discussions at several places (including this blog)
      # Participating in collaborative replication efforts (but only those i think are useful and “true” collaborations)
      # Trying to come up with ideas how to improve matters, and sharing them on several forums and blogs
      # Writing a few pre-prints, and posting them

      I think in this all, i have tried to follow my instincts, have the “right” intentions, and use my brain. I also think i may have a certain personality, and history, that may perhaps not be typical of the avarage person at universities. This may have all have worked together (with a lot of other things) to perhaps create a unique, and fitting, way for me to have tried to contribute to (improving) science.

      I like to think that in doing things this way, i may have contributed the exact things i could have best contributed. Whether this is the case or not, i don’t and may never know. But perhaps that doesn’t matter.

      So in light of your question, i think the best way for someone to contribute to science (doing it as a part time job) might be to try and be mindfull of what, how, and why they want to contribute something to science. And perhaps once in a while think about what “their thing” might be that they (want to) bring to the table.

      I think is also in line with what is written in this blog, and what i quoted above in the 1st comment:

      1)“Adapt what is useful, reject what is useless, and add what is specifically your own”.

      2)”We each should contribute where we can”, to which i would like to add “Perhaps we each should contribute what we want” and “Perhaps we each should contribute what may be the exact thing we could best contribute”.

      • > [@Anonymous:] # Making clear to my former professors and university i felt i was not educated well (so they knew this)

        I’m guessing they already knew. I knew when I was a professor that a lot of students weren’t getting the material in the way I intended. The hard part’s knowing what to do about it, particularly in the face of a heterogeneous student body, which is inevitable in any classroom setting with more than a handful of students.

        I found by the time I got to grad school that a lot of the teaching and mentoring was done by other grad students with different backgrounds or who just got to some material before I did. I realized why when I was a professor and had eight students. It’s very hard to keep up with a band of good students exploring science if you’re not a particularly effective manager.

        • Quote from above: “I knew when I was a professor that a lot of students weren’t getting the material in the way I intended. The hard part’s knowing what to do about it, particularly in the face of a heterogeneous student body, which is inevitable in any classroom setting with more than a handful of students.”

          Next to the professors, i tried and include the (head) deans (if that’s the correct term) of the masters program and university in my attempt at communicating certaing things (including trying to make clear i felt i was not educated well enough)

          I can understand how teachers/professors may not have all the time in the world to teach the things (in the way) they would like to teach.

          I can understand that teachers/professors may be dependent on other people that tell them what they should teach, and how.

          I can however NOT understand how crucial, basic things, may not have been taught to me. Things like:

          # (logical) reasoning
          # that there might be assumptions that go with statistical analysis that you can (should?) be aware of and/or test
          # that there might be “questionable research practices”
          # that there might have been really important and useful papers that should be known to students (e.g. certain papers by Meehl)
          # etc.

          As i have said before on this blog, i sincerely feel 60%-70% of my “education” was pretty useless.

          I have thought from time to time that if i was a teacher/professor i would start each lecture by saying something like:

          “Being critical could be an important part of science, and being a scientist. I hope you all will think about this, and perhaps adopt it as a view and practice. I am (partly) obliged to teach you certain things, and/or point you to certain sources and papers, by the unversity and/or program committee. However, what i teach you can be seen as just a small part of the information and views that are out there.

          I, as a scientist and teacher, think it could be important to say to you that i hope you will be mindful of this, and that you can find additional information yourself. Here are some sources and/or papers that are not part of the regular curriculum but might be important, and/or interesting, and/or useful to you (insert list of sources and papers here)”

          Anyway, things are hard. Being a teacher/professor is hard. I just know what i did, what i thought, and what i think i would do.

        • # (logical) reasoning
          # that there might be assumptions that go with statistical analysis that you can (should?) be aware of and/or test
          # that there might be “questionable research practices”
          # that there might have been really important and useful papers that should be known to students (e.g. certain papers by Meehl)
          # etc.

          Well, there are a lot of reasons why you professor does not teach this. First, there is a professional blindness, i.e. you professor may don’t even know that you don’t know these things, because professor thinks they are obvious! Second, professors who were hired to teach, say, epidemiology, would probably want to teach epidemiology, not philosophy of science.

        • A lot of what Anon is complaining about in statistics teaching is what I often call “That’s The Way We’ve Always Done It”, or TTWWADI.

          I was fortunate in coming to statistics as a mathematician, so in teaching stats, I did, indeed put a lot of focus on things like model assumptions that needed to be taken into account. My background also gave me the ethic of explaining *why* a conclusion was justified.

          I incorporated critical thinking into teaching statistics largely by asking students to justify their assertions — and often asked the class what they thought of another student’s reasoning, to try to instill critical thinking and asking questions as a habit, as something one “of course” does, rather than relying on rote learning.

          As you might guess, it was hard to find textbooks compatible with this teaching style, so I usually had to write supplements (with lots of questions) for the students.

      • > # Making clear to my former professors and university i felt i was not educated well (so they knew this)

        Sometimes I feel I don’t belong to science because I don’t have such a strong impostor syndrome like everybody else =(

        • “Sometimes I feel I don’t belong to science because I don’t have such a strong impostor syndrome like everybody else =(”

          I never understand this “imposter syndrome”. As a result of your comment, i looked it up at Wikipedia. I still don’t get it.

          It seems to me to possibly be some sort of amalgam of certain thoughts, and fears.

          I think many of these thoughts and fears i encounter in the Wikipedia page seem common to me, understandable, possibly valid, and even possibly “healthy” in a way.

        • Quote from above: “I never understand this “imposter syndrome”. As a result of your comment, i looked it up at Wikipedia. I still don’t get it.”

          I just had the following thoughts about this “imposter syndrome”. I wonder if “feeling like an imposter” is directly related to a specific type of activity and/or skills (or whatever the appropriate words are).

          For instance, are there many tennis players with this “imposter syndrome”? Or are there many house-painters with this “imposter syndrome”? Are there many blacksmiths with this “imposter syndrome”? Are there many veterinarians with this “imposter syndrome? Etc.

          If not, i think this “imposter syndrome” might be largely related to activities and/or skills that are very hard to “objectively” evaluate. For this possible reason, i am therefore also not surprised that (some/many current day) academics might feel this “imposter sydrome”.

          If this makes (any) sense, i view this as a possible being in line with my 1st reaction above that these thoughts and fears may possibly be understandable, possibly valid, and “healthy” in a way.

        • “If not, i think this “imposter syndrome” might be largely related to activities and/or skills that are very hard to “objectively” evaluate. For this possible reason, i am therefore also not surprised that (some/many current day) academics might feel this “imposter sydrome””

          Agreed.

    • “what… is the best way to contribute to Science if you doing it as a part time job? …Lets say, I spend 4 days/week at my day job and have 1 days to do Science.”

      I sometimes feel that that 4/1 split of Not-Science / Science pretty well describes my job as a university professor…

  6. I was just talking to Andrew about a related topic today, which is how researchers learn about papers. For the last 30 years or so, I pretty much exclusively find out about things by word of mouth from other researchers and from tracing through bibliographies or just straight-up search for a topic. I guess for that to work, you need a good network of people and you need to be able to sort through a bunch of random literature to find what you need.

    What I don’t need is a journal showing up in my mailbox with a table of contents. The closest I get now is occasionally doing surveys of methodology in recent JMLR and TACL papers—I don’t want to read the papers so much as see what the field’s publishing.

    Andrew says “we should publish this” often enough to make me believe he believes in traditional journals. I think we should all just go the Radford Neal approach and write case studies and technical reports and let the literature sort them out. If people want to volunteer to help others write better papers, more power to them. But I don’t see that it needs to be institutionalized other than to help academic hiring committees decide whom to hire and to promote.

    The message from computer science should be sobering. To get around the ridiculous amount of time traditional journals take to publish things, computer scientists started publishing primarily in conference proceedings. Journals seem almost as old fashioned as books, but have not completely gone out of style. But then the field started getting competitive. Then the conferences started getting competitive. Now we’re in the ridiculuous state where every job talk and CV is festooned with not only referneces but acceptance rates and best paper awards, because that’s what hiring committees respond to. I know several CS professors who insist all their charges write papers for all the major conferences. They’re down to something like 10% acceptance rates, so that’s the only way to stand a chance of getting enough papers to get a job. And because this is so critical for jobs, the conferences now have multi-stage reviews like journals where the author can respond to reviews and then the committee can re-evaluate. And all this is done at a crazy volume in a compressed time frame. Then journals like TACL showed up, the primary purpose of which is to take those conference papers and make them “real” journal papers, because it turns out some deans and provosts care about these details.

    I almost always say “no” whenever anyone asks me to review something for a journal and almost always say “yes” when a researcher asks me directly for feedback on a paper. I universally refuse if it’s not free (no author charge) open access. That’s why I’m not on a lot of the recent papers with Andrew—I was there and invited, but I continue to refuse to publish in paywalled or pay-to-publish journals. Journals are really cheap to publish and the whole publishing industry’s a huge scam where we generate content, give it to publishers, then collectively pay them billions to get the content back that we donated. I’m not even convinced the value is positively signed. And yes, you read that right—billions. You can read about it in their trade association journal. Elsevier alone makes a billion dollars profit on their two and half billion dollars in revenue. That’s a profit margin (if not a profit) that’d make Apple jealous.

    I’ve found pretty much anyone who wanted to read a paper of mine because they were interested in the topic or just helping me out gave me better and more constructive feedback than the journal editors, who weren’t ever really my target audience (they’re too diverse—I’d get comments from natural language conferences like needing to justify Bayesian inference against the alternative of plugging in approximate maximum likelihood estimates). The other reason I say “no” is that I don’t feel like participating in sorting and ranking and deciding who’s “best”, which is largely what all this make-work is about. Not the work of giving people feedback, which I quite enjoy, but all the work involved in deciding what to publish. Let’s just publish everyting on arXiv and letter references sort things out.

    The ony reason I wrote the JSS paper for Stan is that the Wikipedia required a citation about Stan (not using Stan) before they’d let us have a Stan page.

    Just to be clear, I’m not complaining about “publish or perish.” What good is research if you don’t communicate it?

  7. Amen, brother. I still publish in journals, because I am still relatively early career. But I have been increasingly wondering what the added value is. Peer review is helpful for editing and quality control (especially for those of us not so well connected as to have many potential reviewers in your network) but there surely is a better way to organize it that doesn’t cost a billion dollars and lock research behind pay walls.

    • Quote from above: “Peer review is helpful for editing and quality control (especially for those of us not so well connected as to have many potential reviewers in your network) but there surely is a better way to organize it that doesn’t cost a billion dollars and lock research behind pay walls.”

      I have commented here https://statmodeling.stat.columbia.edu/2018/07/25/journals-refereeing-toward-new-equilibrium/#comment-809001
      about an idea i had where reviewers can earn authorship in case of a truly helpful review.

      I was reminded of that after reading your comment that mentions how not so well connected people may have trouble finding reviewers and/or co-authors.

      The idea i wrote about may also be beneficial concerning those issues.

      • “I was reminded of that after reading your comment that mentions how not so well connected people may have trouble finding reviewers and/or co-authors.”

        Slight correction: your comment was about reviewers only, not co-authors. I added the co-author part because it might be similar, but should have written it differently as to not conflate your comment with my thoughts. I hope i made that clear now.

  8. Andrew wrote:

    “Why edit a journal? More generally, how to contribute to scientific discussion?” See below for a response from the Editor-in-Chief of https://www.bou.org.uk/ibis/ Comments are highly appreciated, see https://osf.io/j69ue/ for backgrounds (see also https://osf.io/ajsvw/ , in Arab).

    “From: Ibis; To: Klaas van Dijk; Sent: Thursday, January 03, 2019 6:37 PM; Subject: Decision on Ibis IBIS-2018-VIEW-011

    Dear Mr. van Dijk:

    RE: IBIS IBIS-2018-VIEW-011 entitled “Publisher Taylor & Francis is facilitating research misconduct by its refusal to retract a fraudulent study on the Basra Reed Warbler”

    Thank you for submitting your manuscript to IBIS for publication. This viewpoint article is a detailed opinion piece raising an interesting issue of research misconduct and I appreciate how very important this is for author and the wider scientific community. However, the reason I am rejecting this article is twofold: Firstly, for a successful viewpoint article Ibis is normally looking for an article that may raise awareness of a broad issue. If the ms addressed the general theme of how do we tackle research misconduct in ornithology by citing examples from a range of studies (historical and recent) then I think this may possibly be more of interest to a wider range of readers.

    Secondly, by focusing on a specific case here, then I would be concerned Ibis could become embroiled in a legal argument with another publisher. In my opinion this specific issue needs to be tackled via a legal process with publisher at this stage and not debated here in Ibis.

    Regards / Dr. Dominic McCafferty / Editor-in-Chief, IBIS”.

Leave a Reply to Anonymous Cancel reply

Your email address will not be published. Required fields are marked *