“Columbia Loses Its No. 2 Spot in the U.S. News Rankings”

Here’s the latest news, from Anemona Hartocollis at the New York Times:

Without fanfare, U.S. News & World Report announced that it had “unranked” Columbia University, which had been in a three-way tie for the No. 2 spot in the 2022 edition of Best Colleges, after being unable to verify the underlying data submitted by the university.

The decision was posted on the U.S. News website a week after Columbia said it was withdrawing from the upcoming 2023 rankings.

The Ivy League university said then that it would not participate in the next rankings because it was investigating accusations by one of its own mathematics professors that the No. 2 ranking was based on inaccurate and misleading data.

So far, so good. Indeed, math professor Michael Thaddeus pointed out many suspicious things about the Columbia data—see here (Arts & Sciences), here (Engineering), and here (more on Engineering)—and the university has yet to seriously question any of his specific claims (the best they could come up with so far was reported as, “The 100 percent figure was rounded up, officials said, and they believed they were allowed some leeway,” which isn’t very encouraging), so at this point it would be pretty hard for me to not believe that Columbia’s ranking was not, based on inaccurate and misleading data.

So, yeah, Columbia steps back and U.S. News delists us. Fair enough. It’s like what the IOC does if you’ve been doping or the NCAA does if you break one of its regulations. The IOC and NCAA have notorious problems; still, if you’re in the game, you’re supposed to follow the rules.

From the news article:

In its blog post on Thursday, U.S. News said that after learning of the criticism in March, it had asked Columbia to substantiate the data it had reported, including information about the number of instructional full-time and part-time faculty, the number of full-time faculty with the highest degree in their field, the student-faculty ratio, undergraduate class size and education expenditures.

“To date, Columbia has been unable to provide satisfactory responses to the information U.S. News requested,” the post said.

That sounds about right.

But this bothers me . . .

One thing, though. Here’s something from a recent statement issued by Columbia:

“A thorough review cannot be rushed,” the university wrote. “While we are disappointed in U.S. News & World Report’s decision, we consider this a matter of integrity and will take no shortcuts in getting it right.”

The “matter of integrity” thing seems fine—I guess maybe some people will get fired or, more likely, be encouraged to take early retirement or seek jobs elsewhere—but what’s this “While we are disappointed in U.S. News & World Report’s decision” bit?

Why is Columbia “disappointed in U.S. News & World Report’s decision”? What would they want U.S. News to do? What, in Columbia’s view, would be an appropriate action by U.S. News? I’m honestly not sure. Presumably it would be inappropriate for them to keep Columbia at the #2 ranking, as this ranking is based on numbers which have now been revealed to be inaccurate and misleading. And, even more so, it would be inappropriate for Columbia to be moved up to the #1 ranking. So, if U.S. News wasn’t going to delist Columbia, what should they have done? Move the college’s ranking down to #8? #18? #38? Should U.S. News impute the missing values in Columbia’s data? Maybe hire some (non-Columbia) statistician to help on that?

I get that Columbia is disappointed that it turns out they have some employees who were supposed to putting together accurate numbers on enrollment, etc., and didn’t do that. That’s annoying! But to be disappointed in U.S. News’s decision—that doesn’t make sense at all. What else could U.S. News have possibly done? Keeping Columbia at #2 after all learning about all these data problems would be like continuing to label North Korea as a country with “moderate electoral integrity.”

So, anyway, I liked that NYT article, but I wished they’d pushed back and pointed out how ridiculous that “we are disappointed” statement was.

P.S. I very much appreciate the academic freedom by which I can write posts like this, and Thaddeus can report his findings, without fear of retaliation by the university. It’s not like Google, where if you question their numbers, you might get canned. Columbia is a great place, and the existence of real problems here should not lead us to think that everything is bad.

P.P.S. From a note that Thaddeus wrote to the administration a few days ago:

[Two deans at the university] informed me on Thursday a few minutes before your public announcement, that Columbia will not submit to the U.S. News National Universities ranking this year. This is a welcome development, if only as Columbia’s first acknowledgement of the gravity of the U.S. News matter. Nevertheless, it is disappointing that Columbia apparently intends to return to the National Universities ranking next year. It is also disappointing that Columbia is continuing to submit surveys for other U.S. News rankings, including that of the online master’s program in engineering, which has been seriously misrepresented in Columbia’s U.S. News submissions, as I told you in April. . . .

Conspicuously absent is any commitment to look at the submitted data to which these practices led — not even those data submitted as recently as last summer . . . those questions are about the accuracy of the submitted data, and they can’t be adequately addressed without reviewing the submitted data. Just reviewing the process is not enough. . . . The seeming inadequacy of Columbia’s internal investigation reflects the severe conflict of interest whenever an institution tries to investigate itself. . . .

OK, I don’t want to overstate the importance of this. It’s just college rankings. Nobody’s being tortured, there’s no hawking of unproven diet supplements, no misappropriation of funds, not even scientific fraud or misrepresentation of the research record. It’s just some paperwork. No joke, I hate paperwork too and I can see how some people could get sloppy and then try to cover up what they’ve done. So, yeah, let’s keep it in perspective. Still, if you’re gonna play the game, you should play by the rules, and when you get caught, you take your lumps. And they should be getting mad (or “disappointed”) at whoever approved those numbers, not at U.S. News for pulling the lever on this particular merry-go-round.

24 thoughts on ““Columbia Loses Its No. 2 Spot in the U.S. News Rankings”

  1. This is so bizarre. Does anyone think that applications for Columbia will dramatically decline because it is no longer being listed in US News? We had two daughters go through the college applications process in the 2000s and US News played no part at all in the schools they selected. From my perspective, it’s just something that allows administrators to put on fund raising letters and has zero impact on the type of education a student will receive. Schools should take a page out of the Nancy Reagan playbook and ‘just say no’ to US News!!

    • Alan:

      First, yes, I think it can make a difference on the margin. The U.S. News ranking is only one of many factors, but it’s a factor. Second, I’ve been told that in some foreign countries people take these rankings very seriously, so it could have a big impact on potential international students.

      Also, it’s one of the stories that people can tell about the university. Berkeley’s the place with the left-wing radicals and John Yoo; Harvard and Yale have plagiarists in the law school; UMass Amherst has the people who discovered the Excel error in that econ paper; the MIT Media Lab has celebrities like Jeffrey Epstein; Ohio State has those pariahs in the medical school, the voodoo psychology guy, and a football team that lost in Ann Arbor; Cornell’s got the ESP guy and the Food and Brands Lab guy; . . . and, here at Columbia, we have Dr. Oz and the U.S. News scandal.

      • Alan:

        Just to follow up:

        1. I agree with you 100% that schools should just say no to U.S. News. There’s nothing stopping U.S. News from doing ratings on their own, but universities don’t have to supply U.S. News with convenient datasets and they don’t have to advertise their U.S. News rankings on their websites.

        2. I very much wish that you were correct that these rankings play no role in students’ decisions. Unfortunately, I don’t know that your kids’ experience in college applications is typical. That’s anecdotal evidence on your part; on my part, I have anecdotal evidence that students from other countries have told me that the U.S. News ratings are taken very seriously in some places. I think they were talking about where they were applying to grad school, but I’m guessing it’s an issue for undergrad admissions as well.

        3. Another piece of evidence of the potential importance of the U.S. News rating is that universities often will prominently advertise them. I agree, this could all just be irrelevant stuff they’re publicizing, but I think we should consider the possibility that the rankings make a difference.

        4. Again, I don’t want the rankings to make a difference. I just suspect they do. I have not done a quantitative or empirical study to estimate the effects of Columbia going from #18 to #2. Another, separate question, is what does it do to Columbia’s reputation to have been in the news for cheating on its U.S. News numbers. Again, I have no idea what the effect will be. I’d guess the effect of changes in numerical ratings would be much larger than the effect of the cheating—there are just so many academic cheating scandals out there, and this is just one more.

    • I’m going to guess that you’re an academic, or at least are highly educated (post-college), and so provided your daughters with a lot of information, perhaps casually and implicitly, about various universities. Most people have stunningly little information about what college is actually like, and they pay a lot of attention to factors “we” know are silly, which include US News rankings. (They also pay attention to marketing, sports teams, …) The removal of Columbia from the list, which I think is a wonderful decision from US News and which gives me some faith that there is justice out there, is a big deal.

  2. What should we make of the “thorough review?” It is to their credit to be thorough, but it has been months since the issue was raised and they haven’t been able to complete the review? If it takes that long, then the idea that we can verify virtually any published paper in any field (given that they are often on the order of 100 pages with very complex data – that is not publicly available) must be impossible. If they were serious about the review, then it should have been completed by now. I would not let them off the hook on that statement. As they say, “it’s not rocket science” (at least I think it is not).

    • I would not be surprised if the “thorough review” (which probably won’t be thorough if they don’t “look at the submitted data”) took longer than they normally take to collate and submit their yearly data. It seems similar to a journal taking much longer to retract suspect published research than it took to peer review and accept the initial manuscript.

      • Eric:

        Yeah, it’s already been 4 months since Thaddeus’s first report, and they seem to have done nothing so far but delay and issue statements. How hard could it be for them to just collect and report the numbers??

        • I quite agree with Andrew’s perplexity about Columbia being “disappointed.” U.S. News says that it wrote to Columbia raising some serious questions, and Columbia never wrote back. What does Columbia think U.S. News should do in this situation?

          I also agree with Andrew that a rankings scandal is small potatoes compared to a lot of other things. It’s crucially important for us at Columbia, however, because it reveals that our administration’s strategy is fundamentally at odds with reality. For example, they want to make more money by admitting more students and enlarging class sizes, but they also want us to be considered very prestigious because our class sizes are remarkably small. They can’t have it both ways.

          Finally, I too find the four-month delay hard to understand or excuse. These figures aren’t so incredibly hard to compile, nor are the relevant data sets very large. It should have been possible to review the process of data compilation and submission — as well as the submitted data themselves — within a few weeks.

  3. I wonder how common this is in other universities. In my view a correct response from us news would be to ask a random sample of universities in their ranking the same thing they are asking Columbia. Whenever I look at their numbers and compare to the reality of my institution, I am always skeptical of those rankings.

  4. I think Columbia’s response is as close as you’re ever going to see to being appropriate for the situation. Some weasel words here or there are inevitable, it is to their credit (yeah, I know it’s sad our society has come to this) that they are publicly acknowledging the potential fraud and at least promise to put it right.

    The question of course is whether they follow through and to what extent. But let’s not be too nitpicky about what’s basically a worthwhile response.

    • Name:

      See my comment to Eric above. I don’t think it’s “nitpicky” to say that they could just compile the damn numbers instead of issuing carefully-worded press releases. There must be someone on campus who’s paid to gather these statistics, no?

      • There’s enough fuzziness and room for interpretation in some of those numbers that I’d imagine it’s non-trivial to compile them in a way that’s sure to satisfy every potential critic, now that the world’s attention is focused on Columbia’s data. The only thing they could do to make the situation worse is to release revised numbers and have someone point out a different set of issues with them.

        I’d also think they aren’t about to release any numbers that might result in next year’s ranking tumbling to 100th because they failed to sufficiently game the rankings system. Pretty much a can’t win for them at this point IMO.

        • Name:

          Sure, I’m not saying it’s trivial to put together the numbers. It takes some work. But there must be someone on campus who’s paid to do this sort of thing as part of their job, no? That said, I hadn’t thought about your concern of the ranking falling to 100th if they don’t push the envelope. It’s like the NBA if you’re playing with 5 fouls and there’s no margin for error. At this point I might suggest that Columbia just give up on the U.S. News rankings entirely and then they could just eliminate whoever’s job it was to count those particular beans. Problem solved.

        • Andrew said: “At this point I might suggest that Columbia just give up on the U.S. News rankings entirely and then they could just eliminate whoever’s job it was to count those particular beans. Problem solved.”

          From your mouth to God’s ears.

  5. I think the most charitable reading of Columbia’s statement is something along the lines of ‘we are disappointed this situation developed to the point where USNWR was forced to take this decision’ rather than an expression that USNWR should have made some other decision.

    • No, I think the obvious reading is that Columbia is disappointed that it can no long brag about being No. 2. And I am sure that is true.

      The deans may not see these data issues and integrity issues as being that important.

  6. This calls to mind Goodhart’s Law (there are many more subtle ways of manipulating these rankings) and A. Usher’s observation that university rankings reinforce prestige economies (because the way most people give them a sanity check is to make sure famous research universities in the US and UK and close to national capitals are on top, so any successful ranking will support those existing ideas about which universities are the best, which those incumbents just add to their quiver of arguments why they are superior)

  7. “Why is Columbia “disappointed in U.S. News & World Report’s decision”?

    My take is they want the process to be fair, but are disappointed with the drop in (lack of) rankings. Isn’t it obvious that Columbia would be less disappointed and more content if it were still highly ranked?

  8. Andrew says, “I can see how some people could get sloppy and then try to cover up what they’ve done”. I would’ve chosen a phrasing other than “get sloppy”. I’d be very surprised if this was an accident rather than an instance of what Andrew calls “The Lance Armstrong principle” (i.e., If you push people to promise more than they can deliver, they’re motivated to cheat.)

  9. Thaddeus is mostly correct ultimately bollinger is to blame and should be held to account.

    Where I disagree with thaddeus is his conflating the school of general studies with the undergraduate colleges namely Columbia college and seas. General studies is meant purely for non traditional students and their numbers should not be included in any of the stats. Just because columbia is the only ivy that has such a school it should not be punished in the rankings for it.

    The other stuff he mentioned is correct and it was a fraud and the university leadership must pay for it.

  10. Another thing worth pointing out is that the rankings are inherently liable to gaming or manipulation simply because the magazine is doing an apples-to-oranges comparison. Universities are all set up very differently according to their own histories and requirements, making it an iffy proposition to compare “undergraduate” colleges of “national universities.”

    In the past, Columbia did not even include SEAS in what they sent to US News & World Report. I remember the other Ivy League colleges complaining that it was not an accurate picture. By excuding SEAS (which at least then was far easier to get into), they could maintain a reported low admission rate. But other universities like Harvard or Princeton did not break down their engineering students into a separate college.

    Eventually I guess they added SEAS to the figures they reported. But if they add SEAS, what about the other schools that aren’t graduate schools? General Studies is not a graduate school and yet they get Columbia diplomas. What about Barnard? They get Columbia diplomas and have access to all the classes and campus, why wouldn’t they be included?

    So you have this issue of a very heterogenous group of universities and trying to compare them as if they were all the same. This gives the universities justification for “gaming” it in a way they think would make things more fair to them or more advantageous (ethically it can be argued it’s not clear cut).

    Another thing is that some universities like Columbia own huge “research” hospitals, with huge medical staff. As pointed out by Thaddeus, are these really instructors or researchers at all or are they simply doctors who only tangentially do anything related to research/instruction if at all?

    The amount of money spent per student can also be misleading simply because the same dollar does not buy the same thing everywhere in America. You would expect expenses to be much higher in NYC than in Urbana for example, so a comparison is going to be misleading.

    This difference in expenses, especially capital expenses, is also reflected in Columbia’s endowment compared to peer institutions. The cost of building and cost of land is radically higher in NYC, even in Morningside Heights. So Columbia’s endowment is never going to approach that of peer institutions even if alumni gave the same or if the endowment was managed equally well. Columbia’s expenses are simply far higher and that’s going to eat into the endowment.

Leave a Reply

Your email address will not be published. Required fields are marked *