“For better or for worse, academics are fascinated by academic rankings . . .”

I was asked to comment on a forthcoming article, “Statistical Modeling of Citation Exchange Among Statistics Journals,” by Christiano Varin, Manuela Cattelan and David Firth.

Here’s what I wrote:

For better or for worse, academics are fascinated by academic rankings, perhaps because most of us reached our present positions through a series of tournaments, starting with course grades and standardized tests and moving through struggles for the limited resource of publication space in top journals, peer-reviewed grant funding, and finally, the unpredictable process of citation and reputation. As statisticians we are acutely aware of the failings of each step of the process and we find ourselves torn between the desire to scrap the whole system, Arxiv-style, or to reform it as suggested in the present paper. In this article, Varin, Catelan, and Firth argue that quantitative assessment of scientific and scholarly publication is here to stay, so we might as well try to reduce the bias and variance of such assessments as much as possible.

As the above paragraph indicates, I have mixed feelings about this sort of effort and as a result I feel too paralyzed to offer any serious comments on the modeling. Instead I will offer some generic, but I hope still useful, graphics advice: Table 2 is essentially unreadable to me and is a (negative) demonstration of the principle that, just as we should not publish include any sentences that we do not want to be read, we also should avoid publishing numbers that will not be of any use to a reader. Does anyone care, for example, that AoS has exactly 1663 citations? This sort of table cries out to be replaced by a graph (which it should be possible to construct taking up no more space than the original table; see Gelman, Pasarica, and Dodhia, 2002). Figure 1 violates a fundamental principle of graphics in that it wastes one of its axes, in that it follows what Wainer (2001) has called the Alabama first ordering. Figure 2 has most of its words upside down, which is a result of an unfortunate choice to present a vertical display as horizontal, thus requiring me to rotate my computer 90 degrees to read it. Table 4 represents one of the more important outputs of the research being discussed, but it too is hard to read, requiring me to try to track different acronyms across the page. It would be so natural to display these results as a plot with one line per journal.

I will stop at this point and conclude by recognizing that these comments are trivial compared to the importance of the subject, but as noted above I was too torn by this topic offer anything more.

And here are X’s reactions.

15 thoughts on ““For better or for worse, academics are fascinated by academic rankings . . .”

  1. Could you elaborate a bit on the recommendation around Table 4 as a plot with one line per journal. I’m assuming the comment meant to make each journal series in a chart, and plot their rank as the series values? Wouldn’t this be subject to the vagaries of the sort order of the measurements – making apparent upward or downward trends appear where no such trend really exists?

    Perhaps a box plot or similar visualization showing rank and variability in rank depending on the measurement? While you wouldn’t be able to see the individual rankings by scoring mechanism, the sorting by median rank perhaps is useful to demonstrate that the measures provide generally less disagreement about the top and bottom ranked journals and more variability in the middle.

    • As an IT professional who spends a lot of time trying to measure software development outcomes, I feel confident saying that at least the IT profession abhors being measured at all (much less ranked or compared to others). :)

  2. The paper mentions the UK RAE (research assessment exercise) which takes a small number of publications and uses these with some other information to rank each person and then the department, and this relates to government funding. This means where a paper appears is very important for the department. This means that it is not just for your own ego that you publish in “top” places, but for your department.

    There has been a lot of discussion in the UK about such “league tables” (http://www.bristol.ac.uk/media-library/sites/cmm/migrated/documents/league-tables-technical-review.pdf).

    This contributes to why in the UK academics are concerned with journal rankings.

    ps. I’m no longer there, so my knowledge is a little out of date. If anyone who knows more than me says something has changed, please correct me!

  3. Do the strong feelings (often negative) evoked by this issue arise because academics are fundamentally opposed to any sort of quantitative assessment of scientific and scholarly publication or because the particular metrics that are used are perceived to be unfair & fatally flawed?

    Do we object to the ratings themselves or due to the thoughtless application of these ratings for taking funding & tenure decisions?

    • Sort of relevant: http://statmodeling.stat.columbia.edu/2015/04/15/message-just-sent-class/

      What are you measuring and why? What’s the endgame? How well does what we measure correspond to ‘good’ science’?

      We currently have a mathematical model of scientific discovery with a rate constant of X papers/year. Is this a good model? Is is useful for judging individual scientists? How variable are or should individual scientists be?

      Plus, I am not a number! I am a free man! ;-)

    • I recall reading on Robin Hanson’s blog Overcoming Bias the thesis that foragers far prefer overt social equality, with differences in ability only acknowledged covertly. Foragers who try to integrate into modern society have a very hard time coping with the modern workplace’s emphasis on judging, comparing, and ranking work performance. Apparently humans have to be trained up to tolerate this; see modern schooling.

  4. Rahul and Corey,
    Given the popularity of all of ESPN’s ranking shows, it seems like we humans do like rankings. I think we statisticians are more cautious about them because we know the margin of uncertainty is so large we really shouldn’t be confident that #5 is better than #15 and that most of the ratings take into account different things than we think are most important (unless our team/lab/self is #1, and then ESPN/ImpactFactors are doing it correctly). Maybe it is not like being ranked ourselves (or being ranked lower than we would be if a different ranking method was used).

Leave a Reply to Corey Cancel reply

Your email address will not be published. Required fields are marked *