One more reason to remove letters of recommendation when evaluating candidates for jobs or scholarships.

This is just one more sexual harassment story, newsworthy only in the man-bites-dog sense. But it reminded me of something that gets discussed from time to time, which is that we should stop using letters of recommendation to evaluate candidates for jobs or scholarships.

Here’s a list of hoops that people recommend you jump through. How horrible!

Here are the usual arguments against requiring letters of recommendation: They discriminate against shy students, they favor students who are bullshitters as well as the students of professors who are willing to hype, they’re used by admissions and hiring committees as a replacement for evaluating the candidates’ actual records, they perpetuate old boys’ networks, they promote flat-out lying (it’s an arms race: if your advisor honestly assesses your strengths and weaknesses, the resulting letter for you can’t compete with the letter from some other prof who is willing to hype), and they’re a waste of time and effort, both for the letter writer and the person who has to track down three people to write the damn letters.

I agree with all the above criticisms. (For elaborations, see for example here, here, and here from Fabio Rojas).

In addition, this latest harassment scandal (also this one) made me particularly aware of another problem with letters of recommendation, which is that they’re an uncontrolled source of power, which creates three problems. First, some people will use this power unethically. Second, students can be in confusion, not knowing if this power will be used against them. And, third, the information in the signal is distorted. We sit in these committees reading these letters of recommendations . . . it’s a joke, it’s a scandal, and it’s part of that horrible power game.

I agree with the commenter who wrote:

I think a letter of reference has two requirements: it should show that the applicant isn’t lying about having been a student of [the recommender], and it should show that [the recommender] would not be positively embarrassed to have helped persuade the hirer to take the applicant on.

That’s fine with me, it’s like what companies do, when someone in the Human Resources office calls your recommender to check that you really existed, that you really worked or studied at the institution in question, and that you showed up on time, did your job, and didn’t break any vases when you were there.

Beyond that, potential hires might want to know what you learned in school, so it would make sense for them to see your grades. Nowadays, we pretty much just give A’s to our Ph.D. students, but that could change. I think it would be fairer all around to give some B’s, C’s, and D’s, and let this convey some information. We could grade the Ph.D. thesis too. They do that in some countries.

This is a basic principle of psychometrics: It’s better to get many pieces of information from different sources. For evaluating recent graduates, better to be evaluated based on a portfolio of grades, than from these letters. For evaluating people who’ve been out of school for awhile, better to be evaluated based on a portfolio of work products than from letters that are possibly rigged and also largely based on out-of-data information.

And, sure, informal recommendations are still gonna happen. If your advisor thinks you’re great, he or she can contact people and tell them. Or if your advisor hates you, he or she can try to blackball you. No way of stopping that. But I think this would all be a lot harder to do, if letters are removed from the formal system of evaluation. I’ve been on a lot of hiring and admissions committees, and people are often using letters of recommendation as definitive evidence. I don’t think this would happen so much if letters no longer had their formal role.

P.S. Here’s another example:

A former assistant reported to a Harvard human resources office in late 2008 that Dr. ** was sending her unwelcome and sexually suggestive nighttime text messages. . . . He refused to write recommendations for economics graduate programs she was applying to, according to her complaint, and all rejected her.

P.P.S. I was corresponding with an economist colleague about the idea of eliminating formal letters of recommendation, and we had the following conversation:

Economist:

I’m not sure what the alternative is, a phone call? A rating system?

AG:

1. Grades
2. Manuscripts

Of course you can’t stop informal recommendations. But I think it would be a step forward to remove the formal recommendations. Now it’s my impression that these recommendations are the first thing that many people read. And I’m getting sick of various stat profs who each year describe their favorite students as the second coming of Jesus H. Cauchy.

Economist:

Won’t work for Econ. Fewer are published at first job market (though that is changing). Also, they want to know what role advisor played in the manuscript.

AG:

Knowing that there are no official recommendations, we can make grades more informative. As for manuscripts, they don’t have to be published to be sent to the committee.

Economist:

But good grades do not imply good researcher, unless you have an informative researcher quality grading system

In the Econ job market people don’t read papers until much later in the process, and they want a signal of the paper’s quality from someone who has their reputation at stake. Maybe they should just ask for that.

Maybe so.

32 thoughts on “One more reason to remove letters of recommendation when evaluating candidates for jobs or scholarships.

  1. That’s fine with me, it’s like what companies do, when someone in the Human Resources office calls your recommender to check that you really existed, that you really worked or studied at the institution in question, and that you showed up on time, did your job, and didn’t break any vases when you were there.

    Uhhh. This isn’t what 99%+ of US companies do. They will confirm your dates of employment and that is it. They will not provide any information about your tardiness or vase-breaking tendencies.

    I agree with your views. Why not start with Columbia? The Masters (!) in Statistics program currently requires 3 letters of recommendation. I doubt that anyone even reads them!

  2. When hiring new economics PhDs for private sector consulting, we would get about 500 packages a year. These consisted of a cover letter, three letters of recommendation, a job market paper (JMP), a cv, and a transcript. I don’t believe I ever looked at a transcript. I’d then glance at the cover letter to see if it was any different than the modal one explaining how much they wanted to go into consulting. I then read the abstract of the job market paper; if it was interesting enough I’d then read the JMP and circle back to the recommendations last. If it wasn’t interesting I’d read the recommendations and see if the recommenders could convince me to put the time in to read the paper.

    Reading recommendations requires a number of filters, as you correctly point out. First, you have to separate hype from honesty. With experience, that actually isn’t too hard to do. It is especially easy over time when the same professors generate letters for lots of students over time. And occasionally, recommenders can easily sabotage a candidacy from codewords that clearly signify low quality students.
    To me, the most annoying part of the recommendation letters is the recommenders’ long recap of the PhD thesis of the applicant. That part is a total waste of the recommender’s effort. I don’t care what *they* think about the thesis or how well *they* can describe it and put it in context. I want to know that the applicant substantially wrote it. The JMP should speak for itself. If it can’t, I don’t want the applicant. (It has become common in recent years to get jointly written JMPs. It is critical in the recommendation to let me know what part this particular applicant had in that process.)

    But all of this is just a filter to winnow down 500 applications to 40 or so interviews at the AEA convention. And that is just a filter to winnow down the 40 interviews to 15 or so flyouts where the actual hiring is done. The recommendations by that point are long forgotten.

    Bottom line: write an interesting abstract and a good JMP to go with it and the recommendations are almost completely irrelevant, except to confirm that you really wrote it. Recommendations can get interviews for those who failed this JMP test, but that’s pretty much a crapshoot. It’s about you and your work, not the recommendations.

  3. …what ? — you think academic “grades” are ‘objective’ student evaluation criteria … but consider letters-of-recommendation from those very same “graders” as dangerously subjective ?

      • My experience is that graduate school grades beyond prelim courses are pretty much A’s unless they student didn’t show up — and sometimes even then are automatic A’s (In fact, I remember getting an A in a course I’d dropped at the beginning of the semester, but the drop didn’t get recorded.)

        • There were very few times where I thought the grading at Berkeley could have been biased, and it always involved the grad students who were the teaching assistants and not the professors.

          The teaching assistants are evaluated somewhat on how their group of students does, so the teaching assistants have an incentive to do whatever they can to help them. In some cases the teaching assistants would go over problems very similar to what was about to appear on the upcoming midterm, which gave their students a definite advantage. Another thing that can happen is the teaching assistants are the ones grading the papers, and for problems which aren’t multiple choice could theoretically grade their students easier than other students. And of course sometimes you have teaching assistants favoring or disfavoring specific students.

        • Any departmental policy is going to say that there’s no expectation of favors for grades, but what’s to stop a bad actor from using the threat/offer of bad/good grades to extract favors?

        • … the larger point is that grading student performance always involves some subjectivity/bias, beyond the fairly basic college level courses.
          The student is not the only significant variable in the overall instructional/grading system. Grades reflect a system output.

          Grades are useful but noisy measurements, especially when attempting fine distinctions among highly qualified job candidates.

        • Exactly!

          I hear sometimes people talking “Ill better would not go again my supervisor, as he/she is responsible for writing my recommendation letter, and thus able to send negative ripples through my entire life”. I never heard people speaking that way about their grades.

        • Yah, except if schools who might hire you looked at your grades rather than your rec letters and we stopped giving A’s to everyone in grad school this would change. Quite often courses in a particular subfield are always taught by the same people and if grades are examined rather than recs not taking or doing badly in *your* area would become a deal breaker.

          Arguably, it would make it worse since doing badly in even one of the 4-5 classes in your subfield would probably cost you a job so all it takes is for one of the 3 profs who teach courses in that area to want to screw you (assuming they don’t rotate class assignments much). On the other hand you only need to get along with one of them to have a good relationship with an advisor.

  4. When I left UVA one argument the administrators said to try and get me to stay was “what are you going to do about letters of recommendation.”

    I’ve never been a fan of letters because at Berkeley we had an undergrad in the lab who constantly messed up experiments, and when a company called to ask about him the grad student who worked with him picked up the phone and told the company how bad the undergrad was, but I’m sure the undergrad got a generic positive letter of recommendation for med school. Sure, lab work and med school are 2 different things and I obviously haven’t seen the letter, but the letter probably wasn’t an accurate description of the undergrad’s lab abilities.

    Perhaps naively I thought that if my work was of high enough quality I wouldn’t need letters of recommendation, and that has been true when it has come to forming scientific collaborations, but unfortunately administrations still want to see that generic positive letter of recommendation.

  5. Nathan Kuncel and his colleagues published a meta-analytic review of the evidence on letters of recommendation for college and graduate school admissions. The literature is a bit thin but the overall evidence seems to be that letters are modestly predictive of grades in college and more weakly predictive of performance in grad school (e.g., grades, degree attainment).
    https://onlinelibrary.wiley.com/doi/full/10.1111/ijsa.12060

  6. I doubt it’s feasible to eliminate rec letters because HR departments will continue to demand them out of due diligence. But one person can do a lot to change how they get used, either early in the process if they perform early screens, or late in the process when people are convening to discuss candidates. Focus on factual claims, like what they’ve accomplished, their contribution to joint work, teaching and presentation quality, and maybe direct peer comparisons (Jen is better than Joe but not as strong as Jack). When someone starts seeming influenced by which adjectives are used to describe the candidate, call them on it. Not perfect, but I’ve found it helps.

  7. Sometimes you can get lucky even under a bad system, or maybe because of a bad system.

    My doctoral advisor and I had a falling out, and I realized he had two other students finishing that year, and one the next year, whom he thought much more highly of.

    I went into a tailspin, and eventually left grad school, going into industry where I had a nice career. I finished the degree some years later, with a different committee entirely. But as my recent years as an adjunct taught me, I dodged a bullet. I was much better suited to the non-academic life. So, things worked out for the best.

    By the way, nobody has ever asked me what my grad school GPA was, and I’ve never asked it of anyone I’ve interviewed for a job. I found letters of recommendation mostly useless, but if you talked directly to the person writing the recommendation you might get some useful information (e.g. what their actual role was on a group project).

  8. One thing that hasn’t been mentioned (unless I missed it) is that when asking for letters of recommendation, the potential hirer can ask for the writer to focus on specific things — e.g., the part the applicant played in the research; their teaching experience and teaching effectiveness; their best work (and why the writer thinks is best; …

  9. I imagine ranking unsupervised data in a high-dimensional space is intrinsically hard for any committee. Of course, supervised data, or at least a letter from the supervisor, can be as noninformative as unsupervised ones when the measurement error is too large — e.g., “the second coming of Jesus H. Cauchy” clearly has a half-Cauchy error.

  10. It makes no sense for referees to ask to see the student’s grades or a resume, that just results in double-counting those materials, since the recipient of the reference letter already has them. I understand that referees have a selfish interest in not looking stupid by recommending a poor candidate – but then that calls into question what value the recommendation letter adds in the first place – the referee should be able to make an assessment *without* leaning on grade info, otherwise the assessment offers little incremental value. I have always given recommendation letters based purely on performance/work that I have overseen myself, and when a resume or transcript is nonetheless thrust on me by the applicant I avoid looking at it in order to not be biased.

  11. When I moved to Spain from the US one of the things that surprised me was that letters of recommendation are never used in the university for admissions or hiring at any level. Moreover, the idea that they are used somewhere in such processes is met with incredulity. Their inherent potential to facilitate nepotism and favoritism is seen as immediately obvious and hopelessly compromising. People in Spain find it hard to imagine that an advisor would do anything other than exaggerate the virtues of his or her students. The process is so poorly understood that it is common for a student applying to graduate school in the US and England to ask me for letters simply because I am a native English speaker, even though I had the student in class once, two or three years ago, and can’t say anything that will be of use on the other end (I explain this to them, of course, but it is hard to understand for whom the idea of such letters is not clear).

    The counterpoint is that in a country like Spain all such decisions are made by supposedly “objective” schemes that require quite rigid assignment of points based on different “merits”. So there might be 50 points for research, 25 for teaching, 25 for grants. The 50 are based on a paper count weighted by impact factor tercile (none of this normalized for area or subarea), and other areas are evaluated by similarly “objective” criteria (thousands of euros). This has pernicious consequences. Beyond the obvious disincentivization of deep prolonged research, it leads departments, desperate to hire the best fit, or the already known well liked internal candidate, to manipulate these supposedly objective schemes in what appears to outsiders to be horribly corrupt ways.

    Letters of recommendation are a way of trying to avoid these perversities of European style bureaucracy. I think they fail in that regard, but the problems they intend to remedy remain and need to be addressed.

  12. I needed 2 letters of recommendation for even getting in to a research masters education. So not even a job but just a 2 year masters program. I was only able to get 1 letter, but still applied (and got in).

    Anyway, the feeling of the absurdity of, what is view as the apparent “subjectivity”, of letters of recommendation in an “objective” thing that science should be, has never left me after that moment. Letters of recommendation are a joke in my reasoning and opinion. Later on i found out about many more similarly absurd (to me at least) things, like authors sometimes having the opportunity to recommend peer-reviewers.

    I thought about it on a more global level, and specifically how to get the “best” people in science. I reason in (let’s say) psychological science, it is difficult to assess the quality of the work, and the quality of the people who are doing it. It’s not like tennis, where i can beat you and it’s clear who is “best” (at least on that day). It’s not like building a bridge, which later collapses due to it having been built badly, so it becomes clear that some people messed up.

    It seems to me that this possibly entails a related, but separate, problem: not only could the quality of research, and researchers, be difficult to assess, it could be highly dependent on the people doing the assessment. How is a “bad and not very bright” researcher that does the assessment even going to recognize a “good and bright” researcher?

    Combine this with possible problematic issues at universities like researchers possibly having to get the most funds, write the most papers, etc. (e.g. see Binswanger (2014) https://link.springer.com/chapter/10.1007/978-3-319-00026-8_3), and i reason Professors, PhD students, other staff, etc. may not really be selecting for the “best” scientists. Perhaps they can’t even do that if they wanted to anymore, because they may not be smart enough to even recognize what they “best” people could be, and why.

    If this makes any sense, a letter of recommendation is just a BS way to get the people you want, where you want them. If they would stop using letters of recommendation, they will use something else for that purpose i reason. That’s no reason to possibly stop using letter of recommendation, or be critical of them, but perhaps something to keep in mind.

  13. it was 2002. I was graduating from provincial Russian university and applying for masters degree in Helsinki.

    Helsinki requested a recommendation letter from my home university. They got so shocked someone from abroad contacted them, they called Federal Security Service to interview me.

  14. Hi Andrew and Others,

    Was chatting with a friend about this on the way home and they suggested that Professors could be assigned a limited number of letters of recommendation, and that these would be tracked by a centralized system. This centralized system could then also attempt to standardize the most important content (i.e. length of relationship, nature of relationship, total hours, positive/negative scale towards student accomplishments under supervision, and perhaps a SHORT text box for open comments). What are your thoughts on this?

  15. I agree that hiring committees should spend more time reading a candidate’s papers and less time counting beans. That said, I think the letter of recommendation provides a way for an adviser to convince a hiring committee to take a closer look at someone who would otherwise not appear to score well by standard measures. The adviser can explain special circumstances (e.g., for someone coming from a non-traditional background) and highlight unusual contributions. Another case is where the institution is trying to hire outside their own areas of expertise. Committees at such places have trouble evaluating the candidate’s papers; the letters of recommendation can help situate with work and explain its significance. (Of course a better practice is for the hiring institution to recruit some external expertise for the hiring committee, but this is not always possible or legally permitted.)

    In my own case, I entered grad school from outside the standard path, and I suspect that my letters were the key to getting admitted.

    • +1

      Takes me back to serving for a few years on an NSF committee to rank applicants for NSF graduate fellowships. Comments such as “Could do the hard problems in Herstein” in letters of recommendation were important information in helping to distinguish between applicants who scored in the 900’s on the math advanced GRE subject exam.

Leave a Reply to Mikhail Shubin Cancel reply

Your email address will not be published. Required fields are marked *