The most stunning thing to me about the whole Columbia-cooking-the-books-on-their-U.S.-News-ranking story is that this information was all out there, and it took so long for someone (math prof Michael Thaddeus) to notice.
The second most stunning thing is that whoever’s in charge of this hasn’t apologized and fixed things yet. Yeah, I know that’s how organizations work—they play defense and then they play defense some more—still, this one seems so clear that I’d’ve hoped that they’d move forward.
Instead we get this:
The 100 percent figure was rounded up, officials said, and they believed they were allowed some leeway . . .
I love that! As we used to say of movies, it’s so bad it’s good. “The 100 percent figure was rounded up” . . . that’s pure genius!
It’s in response to this:
By poring through the 958 full-time faculty members of Columbia College listed on its website (the only public list he could find), Dr. Thaddeus came up with 69 people (he has since corrected it to 66) whose highest degree, if any, was a bachelor’s or master’s degree (not including a master of fine arts) or a degree that was not in the field that they were teaching.
OK, the whole “terminal degree” thing is pretty much unimportant—I’m much more concerned about the apparent misreporting of class size, student-faculty ratio, and the implausible claim that Columbia spends more on instruction than Harvard, Yale, and Princeton put together—but it’s just so funny . . . “The 100 percent figure was rounded up”! I’m reminded of that classic statement from Harvard that the replication rate in psychology “is statistically indistinguishable from 100%.”
Also amusing that they cover their butt by saying they were allowed some leeway. Just in case that rounding-up-to-100% thing doesn’t work out.
Pro tip: if something’s not 100%, don’t claim it is, as it only takes one counterexample to destroy you. (OK, in this case, 99.5% rounds up to 100%, so with 958 faculty members, it will just take 958*.005 = 5 cases to shoot you down.)
In all seriousness
I think Columbia is a wonderful institution. I love it here, and my colleagues throughout the university do cutting-edge research and quality teaching, with student involvement at all levels. I’m proud of so many things that are done here. (OK, not everything.) My high regard for Columbia is why it makes me particularly sad to see this sort of thing happen. I get it, everybody makes mistakes, there’s lots of pressure to keep that ranking high, people around the world see these rankings, etc. I respect that. But what they did here seems like a few steps too far, and this kind of defensive reply is distressing in itself and also shows a disturbing lack of respect to Michael Thaddeus, who put in all this effort on his own to check out the numbers. I wish the university spokespeople would start by thanking Thaddeus and admitting they did some things wrong, and then they can go fix it. Denial is not a good start, and it disappoints me, especially in contrast to all the amazing aspects of Columbia in research, teaching, and service.
P.S. I have not looked at these detailed data myself, so I accept that Thaddeus could have messed up in his calculations, maybe Columbia’s claims are all correct, etc. I doubt it—he just seemed to have too much convincing detail in that report, also if he were all wrong, I’d assume it would be easy for Columbia to have rebutted him on the merits rather than making various vague claims—but it’s possible. Best move at this point would be for the Columbia admin to at the very least admit that they might have really screwed up, and then they can share all their numbers and redo from scratch without using their questionable past claims as a starting point.
Could have been worse… they could have rounded *down* to 100%.
Wins thread. No later comment could top this.
Here’s the thing, if they fix it, their rankings are going to PLUMMET since the rankings all differ in the third decimal place so there will be maybe a hundred institutions that have similarly cooked the books between their REAL score and the one they currently report.
I guess it will leave a mark even if they fixed it only to the point where it takes them back down to #18. The striking thing to me is not that an institution games a metric that’s used to assess it, but how dreadfully seriously people take these rankings to begin with.
Given that the rankings are affirmatively harmful to society, I hope that they are submitting fraudulent numbers.
What is the real number then? What did they round up from.
Rahul:
Perhaps it was 51% and they rounded to the nearest integer?
I spent a decade working in the equivalent office (“institutional effectiveness”) in a flagship public state university. I was directly responsible for all or part of all of these reports (IPEDS, CDS, US News and many many more; you have no idea how many more there are, not to mention the never ending stream of internal data needs). I have three general observations:
1. Columbia’s OPIR only lists three employees, two of which have titles that suggest a managerial role. That is very, very small for an institution the size and prestige of Columbia. I have to suspect that they are relying quite heavily on a much larger number of isolated admins or ‘research analysts’ in individual schools & colleges to gather data for them. I’d be curious how they manage that and coordinate definitions and training. In my experience, it would be fairly unusual for managers & directors in these offices to be involved in directly gathering any raw data ((or even have the technical skills required; SQL and comfort with relational dbs, mainly).
2. I found Thaddeus’s original blog post fairly persuasive that something fishy was going on with at least some of the figures. I can’t really tell how much though, because I’d need much more detailed information on how Columbia’s internal data systems work to say for sure.
3. My priors, however, are somewhat different than this crowd, probably. I spent a decade being accused, periodically, by faculty members (usually pretty condescendingly) that our office’s data was “obviously wrong”. It never was. (We made mistakes, like anyone else. Oddly, we almost never got complaints about things we actually messed up.) Every single time our data was questioned by a faculty member, when I sat down with them in front of my computer and walked through it with them, they admitted I was right. In my experience, there is a strong tendency for faculty (particularly STEM faculty) to blithely assume that the output of these sorts of offices is trivial bean counting, and in a sense they are right. But faculty also typically have no direct experience with the raw data systems that underlie the institution. HR systems, for instance, are designed to cut checks and comply with IRS rules. Those dbs are not often set up with IPEDS reporting or definitions in mind. So you frequently are in situations where the definition specified by a federal agency relies on information that isn’t tracked at all or is tracked only incompletely. The analyst is left making approximate judgement calls that can look “fishy” unless you know where the data is coming from. Very commonly, there are subtleties in the data that faculty just don’t think about. I would routinely have to explain to faculty, for instance, that there was no way to link a specific dollar amount paid to them to a specific individual course they taught. It’s really no different than data in any area that you are new to. It was not uncommon for dept chairs to be 100% certain my lists of faculty & classes IN THEIR DEPT were wrong, when they weren’t! I came to feel there was a distinct pattern, where since data in “institutional effectiveness” offices wasn’t an *academic research* area, faculty didn’t approach it with the same humility they would if they were wading into data from a different academic discipline. i.e. institutional effectiveness employees weren’t considered “colleagues” and so were not granted the respect that you might of a fellow faculty member discussing their area of expertise.
All that said, I’ll reiterate that I am certainly convinced that *something* is off with at least some of the figures he discussed. Maybe even all of it is bogus, I think he made a plausible case for that. I guess I’ve just had too many personal experiences with literally each of these data elements (class size, student-faculty ratio, “instructional” spending, that’s a real squirrelly one) where a faculty member claimed my data was wrong and I was able to demonstrate that it wasn’t, that my priors on faculty understanding these things are….well….different.
I mean…maybe I was just at a place with unusually dumb faculty? I dunno; it could be my experiences were atypical. But I thought it might be interesting to share, since it’s a different experience probably than most of the folks here.
Justme:
I know what you mean, and one thing that bothered me about the Columbia admin response is that they did not say something like, “Let’s look at the detailed numbers and you’ll see that we know what we’re doing.” Their response seemed much more defensive, and not what you’d expect to hear from someone who has confidence in their data. Also, Thaddeus’s comparisons to other institutions seemed convincing. But, yeah, I’m open to the possibility that Columbia’s numbers are correct. It’s just on them to show it, at this point.
Justme, your point #3 is excellent, especially “HR systems, for instance, are designed to cut checks and comply with IRS rules. Those dbs are not often set up with IPEDS reporting or definitions in mind. So you frequently are in situations where the definition specified by a federal agency relies on information that isn’t tracked at all or is tracked only incompletely. The analyst is left making approximate judgement calls”.
As a former co-worker (in a different industry) used to put it: the man wants a number. We have to give him a number, the best we can.
Here’s my story about that.
This should be a lesson: always report percentages to at least 2 decimal places.
Reporting the percentage to 0 decimal places would have been already nice in this case. If they rounded 93% up to 100%, as it seems, maybe they would have done the same from 93.11% to 100.00%.
But see Ehrenberg on the virtue of rounding to two effective digits: https://sites.stat.washington.edu/pds/stat423/Documents/Ehrenberg.numeracy.pdf
Like justme, I have worked in an institutional research office. It’s not just wonky statistics but also wonky policy that results from the rankings calculations. See this 2009 [article](https://www.insidehighered.com/news/2009/06/03/manipulating-er-influencing-us-news) in Inside Higher Ed, for example,
>The easiest moves, she said, revolved around class size: Clemson has significantly increased the proportion of its classes with fewer than 20 students, one key U.S. News indicator of a strong student experience. While Clemson has always had comparatively small class sizes for a public land-grant university, it has focused, Watt said, on trying to bump sections with 20 and 25 students down to 18 or 19, but letting a class with 55 rise to 70. “Two or three students here and there, what a difference it can make,” she said. “It’s manipulation around the edges.”
Also, schools can pack small classes into the fall term, since that’s when the census is taken. And since the stats for student outcomes are based on first-time, full-time students (not transfers), schools can “optimize” who is in each official cohort. An extreme example is the president who allegedly wanted to screen out students before the cohort was defined, reportedly referring to them as [“bunnnies to be drowned.”](https://www.insidehighered.com/news/2016/01/20/furor-mount-st-marys-over-presidents-alleged-plan-cull-students).
Other dumb statistics are simply baked into the system. For example, the practice of “super-scoring” standardized tests by adding up the best sub-scores across all test sittings for a student. If you can pony up to take the SAT five times, you get SAT = max(SATM) + max(SATV) for admissions and reporting purposes. US News also considers the fraction of students who have standardized test scores (SAT or ACT), but simply counts the percentage of each as if there was no intersection.
In my experience in working with US News to try to correct data we accidentally misreported (which was detrimental to our ranking), they aren’t interested. Of course they can’t recalculate the ranks because that would shred the veil of its Platonicness. But they wouldn’t even correct the number on the website. At that point I stopped thinking of them as a news organization.
This is definitely a problem.
That’s why here at University of Administration, we’ve beefed up our administrative staff, adding an entire new office with several new positions to provide oversight to the new office and positions we added to provide ranking analysis and remediation strategy, which oversees the office we added before that to design effective attraction and retention strategy, which oversees the Office of Education and Knowledge, to ensure that actual education policies don’t a deleterious impact on Administrative salary growth.
“or a degree that was not in the field that they were teaching.”
So someone with a PhD in Mathematics might be teaching in a Statistics department? I would count such a person as having a terminal degree.
Roger:
Yes, I think that would be ok as the fields are close enough together. I’m a professor in the political science department and my Ph.D. is in statistics.
does anyone else see a parallel between between the power of the college rankings system and the power of prestigious research journal rankings? maybe it’s just me, in which case, I’m happy to hijack this discussion:
departments now defer to journal rankings for decisions about admissions, hiring, and promotions.
when a talented grad student produces a piece of work, the pressure is on to shoehorn the problem into a paper which will get accepted in Nature, etc. (and then making it open access will cost another $12K). researchers and research suffer alike. maybe accepting these rankings makes it easier to dodge discrimination lawsuits – but it’s not leading to better research or more diversity. just more boring dinner party conversations about the journal editors and reviewers.