Last month we discussed Columbia math professor Michael Thaddeus’s deep dive into Columbia rise in the U.S. News ranking of universities. The short story was that Columbia was—is—presenting some iffy numbers: it seems that the “percentage of faculty who are full time” is not really “96.5%,” the “student-faculty ratio” is not really “6 to 1,” our “spending on instruction” is not really higher than the corresponding figures for Harvard, Yale, and Princeton combined, etc.
It seems like a clear case of the Armstrong principle: If you push people to promise more than they can deliver, they’re motivated to cheat.
Columbia’s public relations office followed up with a disappointing response. No correction, no apology. As Thaddeus puts it, “Neither Columbia nor U.S. News has, so far, made any substantive response to my assertions.”
In the meantime, Thaddeus seems to have become addicted to this data-detective thing, and his next step was to look into some of the numbers for the engineering school. His new article is called “The U.S. News Ranking of Columbia’s Online Engineering Programs,” and here’s how it starts:
In an earlier article, I [Thaddeus] documented a pattern of inaccurate or misleading statements made by Columbia University to U.S. News and World Report in support of its National Universities ranking. The purpose of this note is to show that such misrepresentation by Columbia is not confined to the portrayal of its undergraduate programs.
The Columbia Video Network (CVN) is a distance education initiative of Columbia Engineering. . . . Total enrollments in CVN have grown dramatically in recent years, from 151 eight years ago to 356 today. . . .
For the last five years in a row, CVN has been ranked #1 nationally by U.S. News in its Best Online Master’s in Engineering Programs ranking. Columbia advertises this ranking prominently on social media and on its web pages.
These pages also emphasize that the degrees granted by CVN are “indistinguishable” from Columbia degrees granted to in-person students, even though the experience of studying online is, of course, profoundly different from studying in person. The point appears to be that prospective employers will not know that a CVN degree was obtained online.
OK, fine. But I know what you really want to hear. What did Columbia do to rig the numbers to get this #1 ranking of which they are so proud?
Thaddeus tells the story:
Many components of the U.S. News ranking are indices measuring aspects of administrative support . . . Other major components include the selectivity of the program and its reputation with high-ranking officials at other schools, both of which are factors that U.S. News has been de-emphasizing in its flagship ranking.
One component of the ranking has more to do with academic quality as it is commonly understood, and that is class size. Although class size counts for only 3.75% of the overall ranking, it has clearly been a key factor in providing Columbia with the competitive edge which has made it #1 year after year.
This is because Columbia far surpasses all of its rivals in this category. Columbia reported that fully 88.7% of its classes enroll under 10 students. This is an astonishing figure, miles ahead of the competition. No other school ranked in the top ten even breaks 60% in this category. The nearest runners-up are Illinois, reporting 59%, and Purdue, with 37.7%.
How, then, does the Columbia Video Network manage to keep its class sizes so incredibly small?
The answer is obvious from a glance at the listings of CVN courses in Columbia’s Directory of Classes. Online class sections, typically very small, are listed alongside in-person sections of the same course, taught by the same instructor and typically much larger. In the example shown below, an in-person Robotics class with 101 students was accompanied by an online section with 5 students. . . .
Engineering faculty with whom I discussed the matter told me that, despite the separate sections listed in the Directory of Classes, online and in-person students are in the same section for all practical purposes. Online students appear on the same course roster as in-person students. They are given exactly the same lectures (which are recorded during the in-person section). They do the same assignments and take the same exams. They are taught by the same instructor and graded by the same teaching assistants. . . . Even the CVN course directory gives the impression that in-person and online students are enrolled in the same classes. . . . CVN’s reporting to U.S. News, likewise, gives the impression that its students enroll in the same sections as other students. In response to the prompt “Classes include campus-based students,” it explicitly states “Yes.”
The full text of the U.S. News survey for online engineering programs is not publicly available, but its survey for another online program clearly states that “programs in which distance education students are integrated with face-to-face students must include face-to-face students in reporting.” It reiterates the point by adding, “when determining the size of your…program’s class sections include all students at your institution registered in those classes.” . . .
Since most lectures for CVN students are delivered “asynchronously” — meaning that they are not viewed live, but rather as prerecorded videos — these students have no opportunity to ask questions or otherwise participate during lectures. Nor do they receive the extra personal attention from instructors that students in small classes would, since they are taught as part of a much larger in-person section. That is to say, CVN students reap none of the purported benefits of small class sizes. Their class sizes are small in name only.
The punchline:
The class size figures on which Columbia’s #1 ranking rests are therefore meaningless. There is nothing to prevent Columbia, in fact, from further subdividing CVN sections into smaller sections. Without making any changes to instructional arrangements, or altering the educational experience in any way, it could thus make CVN “class sizes” as small as it wished.
Thaddeus continues:
According to a post on Reddit, one of the typical questions in the admissions interview for the Columbia Engineering M.A. is “What does integrity mean to you?”
It would be worth asking the same question of whoever decided to place CVN students in very small pro forma sections, report these small class sizes to U.S. News, and then use the resulting #1 ranking as the centerpiece of CVN’s marketing strategy.
He loops back by reporting that the administrator responsible for this program, the Vice Provost for Teaching and Learning, has been promoted to Senior Vice Provost, and one of his achievements was listed by the provost as “propelling online engineering programs at Columbia to be ranked first in the nation for the past five years.”
I guess the first step of the Columbia administration should be to thank Thaddeus, apologize to the world, compute the correct course enrollment numbers, and ask U.S. News to recompute the rankings. The next step should be to figure out who’s been fudging these numbers and remove them from any administrative role at the university. And the next step after that should be to open up this process.
As has been much discussed, the U.S. News rankings are horrible, and if Columbia chose not to participate in them at all, that would be fine with me. But if you’re gonna play the game, don’t cheat. And when you get caught with your hand in the cookie jar, admit it and do the restitution.
P.S. As I wrote before, I have not looked into these data myself, so I accept that Thaddeus could be completely confused here, maybe Columbia’s claims are all correct, etc. I doubt it—Thaddeus just seemes to have too much convincing detail in his reports, also if he were all wrong, I’d assume it would be easy for Columbia to have rebutted him on the merits, which it doesn’t seem that they’ve done—but it’s possible. I’m open to being convinced that Columbia did nothing wrong here, but they’d have to make the case.
P.P.S. If Columbia’s screwing with the numbers, I guess lots of other universities are doing it too, in which case the U.S. News rankings are something like what rich executives’ tax returns would look like if the IRS didn’t do regular audits. To decide not to cheat could really be bad for your ranking! It’s the Armstrong Principle all over again.
“What does integrity mean to you,” indeed. Really bad news if this is intended to be a template for Columbia University going forward.
Very circumstantial, but the stats grad program at Columbia appears to have leapt ahead in the rankings recently, as well. I wasn’t able to find rankings on the official site for past years, but another site the school was tied for #16 in 2018. This year, it’s tied for #5. Although…Andrew, didn’t you move from Princeton to Columbia then? That may very well explain it! :)
A bonus track for you statistics mavens.
Columbia’s reported “Class sizes” (in its online engineering programs) were:
2–9 students: 88.7%
10–19 students: 8.8%
20–29 students: 2.5%
Yet its reported “Average class size” was 14.
Right off the bat, this is a sign that something fishy is going on!
All these figures are from the “Academics” tab at
https://www.usnews.com/education/online-education/columbia-university-OENG0031/engineering
This is not an isolated event, either, as similar contradictions have occurred in past years.
Maybe the catalog is large enough that the one class with 1,800 students doesn’t show up here because its percentage rounds down to 0.0%.
If you take the upper bound of each interval and multiply it by the proportion, the sum is 10.38. For fun, I took Jeff’s 1,800 student course example and gave it a 1% proportion, and took off 1% from 2-9 students. The sum was 12.17. Something fish is definitely going on
Michael:
This might not necessarily be a contradiction, if “average class size” is averaged over student enrollments rather than classes.
Consider the following example. A university has 10 small classes with 10 students each and 1 large class with 100 students, so 11 classes in total. Averaging by class, the average class size is (10*10 + 1*100)/11 = 18.2. But you can also average by student enrollments. Here there are 200 class enrollments, 100 in classes with 10 students and 100 in a class with 100 students, so if we average this way, the average class size is 50.
Weighting by enrollments in this way might sound weird, but depending on how you are calculating, it could make sense. For example, if the university really did have one class with 1000 students as Jeff suggests, this could be buried in the averaging-by-class approach but not in the averaging-by-enrollments approach.
I don’t know if that’s how Columbia and U.S. News do it; it just seems like it could be a possibility. More likely, I guess, is that Columbia was computing average class size from a different dataset than they were using to compute the proportion of classes with fewer than 10 students.
Regarding the more general issue of how Columbia got that #1 ranking: you write, “Although class size counts for only 3.75% of the overall ranking, it has clearly been a key factor in providing Columbia with the competitive edge which has made it #1 year after year.” But if it’s really only 3.75% of the formula, isn’t it likely that the #1 ranking is mostly coming from other things such as measures of reputation? I agree completely that it’s bad for Columbia to be cheating here; I just wonder whether the cheating was even necessary?
Dear Andrew and company,
Thanks to you all for your stimulating comments. Here are my responses.
@Jeff: Nice idea, but the same web page also says that the maximum class size in the program is 20. See the “Academics” tab here:
https://www.usnews.com/education/online-education/columbia-university-OENG0031/engineering
So I think it’s impossible to come up with 14 as a weighted average here.
@Andrew: Rather shockingly, the text of the U.S. News survey calls for “average” figures in many places, but it never specifies what kind of average is sought (median, mean, weighted…?). Maybe there is some written meta-guidance somewhere about how to compute “average” figures for U.S. News, but if so, I wasn’t able to find it. It almost makes you wonder whether affording institutions the freedom to choose whichever average is most favorable to them is a feature, not a bug.
U.S. News does not make public the text or instructions for the survey, but several institutions post their filled-out forms on their web pages, e.g.:
https://www.csudh.edu/Assets/csudh-sites/uepa/docs/student-data/us-news-and-world-report-2021.pdf
https://docplayer.net/225486020-U-s-news-world-report-best-online-nursing-master-s-programs-statistical-survey-2021.html
Andrew’s idea of weighting by enrollment size in computing average class size is a good one, but (especially when the 20-student maximum is imposed) I still think it’s impossible to get anything rounding to 14 as a weighted average in this way. The highest I can get is ((88.7 x 9) + (8.8 x 19) + (2.5 x 20)) / 100 = 10.155, so I think Andrew’s alternative suggestion, that this average came from a different data set, has to be the right one. Assuming that it wasn’t just pulled out of thin air.
A closely related “average” measure of class size, which I prefer, is the size of the class with the median enrollment. I addressed this in footnote 2 of my previous article, where I found that this average figure, for undergraduate courses at Columbia, is about 40 and growing.
https://www.math.columbia.edu/~thaddeus/ranking/investigation.html#555f
Regards to all,
Mike
PS — @Andrew’s comment:
> But if it’s really only 3.75% of the formula, isn’t it likely that the #1 ranking is mostly coming from other things such as measures of reputation?
Yes, but:
(1) Since Columbia is so far ahead of its rivals in this category, thanks to its use of pro forma sections, this category alone may make a substantial contribution to the margin of victory.
(2) The reputation score may largely be an echo chamber that primarily reflects past years’ rankings. For what else do high officials at other institutions have to go on?
(3) This is one of the few categories that may be checked against information available from another source. In literally every case where such checking is possible, Columbia’s numbers have proved to be inaccurate or misleading. So it’s a good bet that they are fudging the other numbers as well.
This is unlikely to be situated in the engineering program, but I entertain a vague hope that most of this kind of manipulation will be traceable to a single topics course with a title like “Gerrymandering and its Effects on Ranked Choice Voting.” It’d at least suggest someone had a sense of humor.
This is amazing and appalling. I wonder if students in the Columbia Video Network program could sue for their tuition back, since the program supplied false information about course sizes and (therefore) rank. If I knew anyone who had been a student, I’d forward this post to them!
Honestly, I’m surprised we aren’t seeing comments from students who used this information to decide to enroll only to immediately see that the claims were misleading at best. Of course, maybe they are and I just don’t see the complaints. Or the students fear devaluing their degree.
Jacob:
Yeah, I kinda wonder about that too. I guess it’s scandal fatigue. This story didn’t even get much play in the Columbia student newspaper!
So many scandals in the world that it’s hard to keep track of each one. I still wonder what it would feel like to be a student at the University of California and take a class from that Why We Sleep guy.
I wouldn’t be surprised if past students aren’t aware of a lot of this. I don’t know how many people read it, but you could post a link to this post at https://www.reddit.com/r/columbia/ . (But get ready for some angry administrators, Andrew…) Or some helpful commenter could post there…
I suspect taking a class with the Why We Sleep guy would be fun! Sleep seems like it’s good to do and also like we don’t do enough of it, just like the “drink more water” stuff we get all the time. I would assume students taking a class with such a guy would have no idea of the problems behind the research and feel like they learned a lot about the importance of sleep.
On the other hand, if I was told all the classes in my program have <20 students and all my classes actually have 40 students, I'd think something was wrong with the advertising.
> feel like to be a student
Curious point.
When I had Rehab medicine students at U of Toronto do quality assessments of published studies in their field (1986) they chose papers by their professors. When none of them scored more than 3 out of 10 they were very amused (I never got too teach there again).
When I was at Duke teaching social science and psychology students and pointed out the problems with statistical methods their professors or papers there were assigned to read were likely using – they were very uncomfortable. Inline with their fear of devaluing their degree and what they were learning. I guess it’s like living in an authoritarian country – it is so much easier and pleasant to believe the propaganda and push back at anyone who devalues it. I also heard that the social science and psychology were planning to start teaching the stats course themselves.
“Neither Columbia nor U.S. News has, so far, made any substantive response to my assertions.”
This seems to becoming a thing worthy of its own name like The Armstrong Principle. Digging in one’s heels doesn’t quite do it justice. For example, despite the 1619 project’s major and numerous flaws, neither Nikole Hannah-Jones or the New York Times have, so far, made any substantive responses. ‘tegrity indeed.
Mark:
I’ve written about this general topic: A ladder of responses to criticism, from the most responsible to the most destructive. I agree that it’s really annoying when people refuse to engage!