One more reason I hate letters of recommendation

Recently I reviewed a bunch of good reasons to remove letters of recommendation when evaluating candidates for jobs or scholarships.

Today I was at a meeting and thought of one more issue. Letters of recommendation are not merely a noisy communication channel; they’re also a biased channel. The problem is that letter writers are strategic: they’re writing their letters to manipulate you, the reader. Yes, I know we all try to adjust for these biases, but that just introduces one more complicating factor.

Really, what’s the point? Letters of recommendations have so many other problems, and this is one more.

13 thoughts on “One more reason I hate letters of recommendation

  1. Not to mention that letter requestors are unlikely to go request letters from people who dislike the work they’ve done.

    A letter of recommendation comes down to answering this question: can you find someone who doesn’t hate you?

    Turns out it’s not that hard for the vast majority of people.

    • If your applying for an academic position out of grad school and your advisor isn’t one of those people that’s a big red flag.

      Also, at least in my area of mathematical logic, there are rarely more than 2 or 3 people who are even capable of understanding your work at any particular school greatly limiting who you can ask for letters if you want an academic job.

      On the other hand for jobs in industry I tend to agree.

  2. To evaluate the Type M error and Type S error of a recommendation letter is far more difficult than those of a p-value. As humans are ultra-high dimensional objects, a typical Bonferroni type post-correction will downgrade the value of every single recommendation letter to literally 0.

  3. I think it depends a lot on the size of the discipline. In small disciplines where you personally know all the letter writers and expect it to be a repeated game it seems to work pretty well.

    Moreover, for many kinds of academic work in grad school it’s very hard to get the necessary information otherwise. There are a range of different degrees to which the applicant may have contributed to their thesis from simply taking an idea their advisor suggested and doing the grunt work to coming up with all the great ideas and directing it with only minimal guidance needed.

    • It works great if all the people you know personally have good judgment (usually not the case) and there are never any outsiders (unless keeping it that way is the whole point).

  4. Also the problem with using grades instead is that in many fields you really don’t care how the applicant performed in classes. For instance, take mathematics. The skills involved in doing homework in a class are only very loosely related to those we care about in professors. Being able to do sustained work on a problem over a long period of time and come up with creative answers is very different from solving the little puzzles that one gives in class to ensure understanding.

    I know tons of people who are excellent mathematicians who would never have done well in graded classes in graduate school. Based on what I’ve seen of my friend’s experiences in areas like physics the gap there is even larger, especially in experimental subjects. You simply can’t evaluate the skills you care about in someone who you’ll hire to build instruments and run experiments in your lab based on their course grades.

    I see the problems you mention but I just don’t see an alternative in many disciplines where the extent of the individual’s ability you care about isn’t well captured
    in grades or their thesis. Also the repeated interactions and personal knowledge minimizes many of those problems.

    • I should also clarify that in some fields (like mine) it’s quite common for very promising candidates to graduate without a single published paper (critical skills needed are often only learned while doing their thesis research…and paper review often drags on for multiple years).

    • “I know tons of people who are excellent mathematicians who would never have done well in graded classes in graduate school.”

      I know many mathematicians, some very good, some not so good. I know very few if any who weren’t excellent students. Many of the better ones were better than excellent students – of the perfect score on every exam sort – too.

      Getting good grades is not the best indicator of potential to do mathematical research, but poor academic performance is rarely seen in those who later succeed in mathematical research. The few contrary stories are mostly exaggerated to make the point that some people with real talent react badly to the constraints placed on them by traditional school work. Most future mathematical researchers exhibit broad intellectual curiousity and strong academic performance throughout their development.

      • In math (my field), course grades are not equivalent to exam grades. First, many of the required courses have very challenging homework problems, which might count more toward the grade than exam scores — and are more indicative of research potential. Second, courses beyond the required ones are often lecture only, with no homework or exams, and everyone gets an A (or maybe grade is based on attendance). (Or not: I once got an A in a course I had dropped before classes started, but the drop somehow didn’t show on the professor’s final grade roster.)

Leave a Reply to Dan F. Cancel reply

Your email address will not be published. Required fields are marked *