This is Jessica. I was talking to another professor in my field recently about a talk one of us was preparing. At one point, the idea of mentioning, in a critical light, some well known recent work in the field came up, since this work had omitted to consider an important aspect of evaluation which would help make one of the points in the talk. I thought it seemed reasonable to make the comment, but my friend (who is more senior than me), ‘We can’t do that anymore. We used to be able to do that’. I immediately knew what they meant: that you can’t publicly express criticism of work done by other people these days, at least not in HCI or visualization.
What I really mean by “you can’t publicly express criticism” is not that you physically can’t or even that some people won’t appreciate it. Instead it’s more that if you do express criticism or skepticism about a published piece in a public forum outside of certain established channels, you will be subject to scrutiny and moral judgment, for being non-inclusive or “gate-keeping” or demonstrating “survivor bias.” The overall sentiment being that expressing skepticism of the quality of some piece of research out of the “proper” channels of reviewing and post-conference-presentation QA makes you somehow threatening to the field. It’s like people assume that critique cannot be helpful unless its somehow balanced with positives or provided in the context of some anonymous format or at a time when authors have prepared themselves to hear comments and will therefore not be surprised if someone says something critical. Andrew has of course commented numerous times on similar things in prior posts.
I write these views as someone who dislikes conflict and publicly bringing up issues in other people’s work. If I’m critiquing something, my style tends to involve going into detail to make it seem more nuanced and shading the critique with acknowledgement of good things to make it seem less harsh. Or if there are common issues I might write a critical paper pointing to the problems in the context of making a bigger argument so that it feels less directed at any particular authors. But I don’t think all this hedging should be so necessary. Criticism in science should be acceptable regardless of how it comes up, and you can’t imply it should go away without seeming to contradict the whole point of doing research. This has always seemed like a matter of principle to me, even back when I was getting critiqued myself as a PhD student and not liking it. So I still get surprised sometime when I realize that my attitude is unusual, at least in the areas I work in.
One thing I really dislike is the idea that its not possible to be both an inclusive field and a field that embraces criticism. Like the only way to have the former is to suppress the latter. It’s unfortunate I guess that some fields that embrace criticism are not very diverse (say, finance or parts of econ), and that other fields that prioritize novelty and diversity in methods over critiquing what exists tend to be better on diversity, like HCI or visualization which do pretty well in terms of attracting women and other groups.
In a different conversation with the same friend above, they mentioned how once in giving an invited seminar talk at another university, another professor we know at that university made some critical comments and my friend got into a back and forth with them about the research. My friend didn’t think much of it, but as their visit went on, got the impression that some of the PhD students and other junior scholars who had attended saw the critique and exchange between my friend and the other faculty member as embarrassing (to my friend) and inappropriate. This was surprising to my friend, who felt it was totally normal and fine to that the audience member had given blunt remarks after the talk. I had a similar experience during an online workshop a few months back, where a senior well known faculty member in the audience had multiple critical comments and questions for the keynote speaker, which I thought was a great discussion. But others seemed to view as an extreme event that bordered on inappropriate.
Related to all this, I sometimes get the sense that many people see it as predetermined that open criticism will have more negative consequences than positive, because it will a) undermine the apparent success of the field and/or b) discourage junior scholars, especially those that bring diversity. On the latter, I’m not how much evidence people opposed to criticism have in mind versus they can simply imagine a situation where some junior person gets discouraged. But a different way to think about it could be, It’s the responsibility of the broader field, not just the critic, if we have junior researchers fleeing in light of harsh critique. I.e., where are the support structures if all it takes is one scathing blog post? There’s sort of an “every man for himself” attitude that overlooks how much power mentors can have in supporting students who get critiqued. Similarly there’s a tendency to downplay how one person’s research getting critiqued is often less about that particular person being incompetent than it is about various ways in which methods get used or claims are made in a field that are conventional but flawed. If we viewed critique more from the standpoint of ‘we’re all in this together’ maybe it would be less threatening.
A few months ago I wrote a post on my other blog that tries to imagine what it would look like to be more open-minded about critique, e.g., by taking for granted that we are all capable of making mistakes and updating our beliefs. I would like to think it is possible to have healthy open critique. But sometimes when I sense how uncomfortable people are with even talking about critique, I wonder if I’m being naive. For all the progress I’ve seen in my field in some respects (including more diversity in background/demographics, and better application of statistical methods) I haven’t really seen attitudes on critique budge.
“Criticism in science should be acceptable regardless of how it comes up, and you can’t imply it should go away without seeming to contradict the whole point of doing research. This has always seemed like a matter of principle to me, even back when I was getting critiqued myself as a PhD student and not liking it.”
Talking about “not liking it” seems an understatement:
“I remember thinking, this is the worst thing that could possibly happen to me. I would rather lose a friendship or be dumped by someone I really cared about than have someone attack my work. It felt a bit like the world was coming to take away the one thing I cared about.”
“While I recall feeling much less personally bothered by this critique, I do recall a somewhat depressing realization that no matter how careful I was, it wouldn’t necessarily ward off people taking shots at my work.”
“However, the bigger reason for me feeling uncomfortable during the months surrounding these events was the more obvious one — having people in your field publicly discuss you, is pretty awful, and perhaps even worse the second time around.”
True, but who isn’t going to feel like that when critiqued on one of their first papers? The point is that this kind of mentality, that the world is over for you when you get critiqued, is not helpful.
Also, why try to use someone’s honesty against them? I write about critique because I genuinely hope that other people (including students who are just starting their careers) don’t have to live in fear of getting critiqued, because its ultimately not helpful to them or the research. If you care about doing good work, chances are you can handle hearing criticism.
I’m a stats phd student. In my limited experience I have observed the following relationships that move from “open to criticism” to “less criticism”:
Italy -> US
Males -> Gender balanced
Physics -> Statistics
So if Jessica, an American Female Statistician, felt quite bad the first time she was harshly criticised, it means today is a good day for my confirmation bias!
I think this problem is important for the quality of research. When I was teaching, a high school (female) student once told me, “when I go to college, I want somebody that every morning tells me I’m good and I’m doing well.” On the contrary, when people do not criticize what I say, half of my brain suspects they are trying to steal my soul or something.
Jessica:
I’m ok with criticism (giving or receiving) as long as it’s specific. What bugs me is when people criticize without giving a good reason, as with the psychology professor who labeled certain unnamed researchers as terrorists, or the law professor who labeled certain unnamed researchers as Stasi, or the economics professor who called me disingenuous. So annoying when people make the inflammatory statement and don’t back it up.
Annoyingly, some of the people who do that horrible thing—making strong criticism while offering no actual reasons for their critiques—are the same people who are bothered when they or their friends have work that is criticized on specific substantive or methodological grounds.
It’s almost as if they prefer empty criticism to actual criticism. Why is that? I don’t know, but maybe they view criticism as essentially political rather than scientific. That is, when someone gives a legitimate criticism of their work, they treat it the criticism as a move in a zero-sum game rather than as a positive-sum attempt to move the ball forward.
I wrote something about this a few years ago: My hypothesis is that one reason that scientists respond so negatively to criticism is that they (we) have been trained to treat scientific critique as something to work around, not to take seriously. I think about all the conversations that I’ve had with colleagues over the past few decades on the topic of responding to journal referee reports: almost all of these have come from the presumption that our article is 100% correct, the negative reviewers are some mixture of clueless and nasty, and our job is to make whatever minimal effort required to get the damn paper published. OK, not completely: we do recognize that negative reports represent some failure of communication on our part. But it’s rare that our reaction to a negative referee report is, Damn, we got it wrong, we’re so lucky the reviewer pointed this out!
My point is that, starting in grad school, scientists are socialized to treat criticisms (in job interviews, referee reports, conference talks, etc.) as obstacles to overcome—how best to handle difficult questions, negative reports, etc.—, rather than as indications that we might be making major errors. So perhaps no surprise that many scientists continue with this attitude as they progress in their fields.
Yes, I’m aware of your past comments on how we are trained to deflect criticism. I agree that’s part of it. What really bugs me though and led me to write this post is less scientists defending their own work, which I expect to see, and more other scientists negatively whenever someone criticizes outside of etablished channels, as though its their duty to defend the canon or the junior scholars from having to endure anything but very polite comments after a talk.
I’m not talking about empty criticism so much as comments on specific methodological flaws or bad premises that are brought up in this post, but I agree, there are people who will make broad statements themselves about others research being naive or whatever but then jump up to defend in other circumstances where the criticism is much more careful.
Jessica:
Oh yes, I agree. This is sometimes called “white knighting.” I discussed one such example a few years ago. The white knight in question (a senior faculty member, whom I’d never met, who taught at a different university) sent me some very angry emails.
It seemed that, from his point of view, I was breaking the rules by publicly criticizing published work. The whole thing seemed so ridiculous to me, but to him it was all about good guys and bad guys. So frustrating.
A related point is that scientists, even those who are prominent in their fields, are generally obscure in the general sense, and in the rare instances when they (we) get media attention, it will be respectful. So then when they see reporting of actual criticism, suggesting that media-celebrated scientific work might actually be wrong, it seems like the world upside down.
That is a great term for it.
I like that term too. The bad news is I think it gets used by the same people who disparage “social justice warriors.”
The world might be a better place if it became conventional at the end of a presentation for the presenter to indicate the limits for the subsequent questions:
Level 1: Questions and clarifications
Level 2: Suggestions
Level 3: Mild criticism
Level 4: Harsh criticism
Level 5: Soft projectiles
Puts the presenter in charge of interactions, avoids cultural misunderstandings, explains why there’s no harsh criticism, etc. Of course, all but Level 5 should be welcome in private, and Level 5 should be reserved for suitably-sized teams.
Andrew, I think you’re onto something here, that we’re conditioned to view criticism as obstacles rather than attempts to improve our work. That makes sense given that science and academia essentially have become engines by which jobs are earned and sinecures procured.
But…what contributes to this feeling is how poor, misguided, or irrelevant so much of the criticism in what Jessica calls “certain established channels” happens to be. Yes, I have gotten useful peer reviews before. It’s happened. But far more often than not a peer review misses the mark completely–I just had a referee call one of my manuscripts “shoddy” because (quoting) “the data are not normally distributed and so does not satisfy the assumptions of the author’s analytical method of linear regression.” And I could go on as could you as could basically everyone who has posted here. So, your colleagues may feel like negative reviewers are clueless and nasty because so often…they are clueless and nasty. Especially that Reviewer #2.
Academic criticism in general is pretty pathetic, imo.
I want to contrast it with the sabermetric community, where criticism I don’t even think is seen as such (at least not among certain sabermetricians when I used to follow it). Someone like Tom Tango or Phil Birnbaum would post his thoughts on his blog, transparently outlining his methods and reasoning, and then other sabermetricians would ask questions and offer suggestions. Discourse was out in the open, free and available for all to see and even to contribute to–I, as an enthusiastic observer and hobbyist, could ask questions and have them answered, even.
So maybe if academia writ large were more like that, and less an impact factor career mill, scientists would be more receptive to criticism and less likely to see it as obstacle.
Kanarua:
Yes, good point. Much of the criticism we see (in referee reports and elsewhere) is pretty bad, which indeed goes some way toward explaining why researchers tend to see criticism as a nuisance or even to consider it to be illegitimate (as with the leading academics who refer to criticism as “terrorism,” “Stasi,” etc.).
I also agree with your positive assessments of blogs. Blogs aren’t perfect but when blogs have comment sections and interact with other blogs, we really can see progress in these discussions. I don’t like twitter so much because the tendency there seems to be to give a thumbs-up or thumbs-down without reasons or links to full arguments.
Can someone explain to me why stating that an analysis is “shoddy” because “the data are not normally distributed and so does not satisfy the assumptions of the author’s analytical method of linear regression.” misses the point completely?
I don’t know about the data but for example fitting a linear model to exponentially distributed data seems a bad idea, doesn’t it?
Stat:
For one thing, the normality assumption of linear regression corresponds to the errors, not the original data. The bigger picture is that normality of the errors is typically the least important assumption of linear regression; see discussion here, with more in Chapter 11 of Regression and Other Stories.
For your example: for all-positive data we generally prefer to fit that linear regression on the log scale, but the reason is not the distribution of the errors but rather because a multiplicative model will typically make more sense than an additive model.
@Andrew
Thanks for responding and for the link.
I love this blog and I feel it provides me with good intuition in many hard to grasp and difficult areas: Like not to over-generalize and to be careful with causal interpretations. But then there are really basic concepts that I do not understand fully. Will be working on this.
As an economist, I have to say I find this so mindblowing as to be from another planet. You shouldn’t criticize people’s work, especially those from underrepresented groups? Surely the inference will be that the work of these underrepresented researchers is less well-established since, even if we know problems, everyone is afraid to criticize it?
Academia is about the production of knowledge, which requires rigorous debate. It is not a social club. I know folks knock the economics style of (very) harsh seminars and feedback, but at least what we hear in person actually represents what people think about the work when they discuss it privately! Like anyone, I always prefer seminars to “go well”, but if people weren’t willing to give me honest feedback, why would I even bother flying over to give the talk in the first place?
The mentality Jessica describes here is so pernicious as to destroy public trust in academic research (and correctly!) – thanks greatly for pointing out how much of a problem it is.
As a Physicist-turned-Economist, I knew that Econ had the image of being an aggressive field. However, I did not fully grasp the extent of it until I made the unfortunate choice of volunteering to listen to my wife’s job talk. She’s a Chemist and I don’t know much chemistry so I thought I was mostly tossing softball questions when fifteen minutes in, she explodes “What the fuck is your problem?”
Many Economists arrogantly claim that all fields must be more like Economics and they are wrong, most of the times. But if there is one thing about Economics that other fields should emulate, it is the (over)abundance of criticisms.
Excellent post.
About “…got the impression that some of the PhD students and other junior scholars who had attended saw the critique and exchange between my friend and the other faculty member as embarrassing (to my friend) and inappropriate. This was surprising to my friend, who felt it was totally normal and fine …” — Yes, I’ve seen this as well, that the more junior people seem most uncomfortable with the existence of critique. This has certainly gotten worse in the 16 years that I’ve been a faculty member, with negative consequences for the liveliness of seminars and, I think, for the vitality of science. I think it connects to trends that are opposed to judgement and assessment more broadly, but I’m not sure. The one constructive thing I can add is that if lots of people regularly ask questions, it seems to help — it seems more natural, to everyone, to ask questions. Different seminar series can develop markedly different cultures, even at the same institution.
There is some implication here that visceral and fearful reactions to criticism is irrational; more disturbing is the possibility that it is rational. What if criticism feels personal because it is? For the non-tenured, maybe there is no such thing as constructive criticism. It feels like critics might destroy your career because they might destroy your career. There are so many more highly qualified candidates than jobs in academia that if someone is publicly known to have made a mistake, there’s no reason to not throw them away and replace them with someone else.
In that light, when faced with criticism that feels true to you but is arguable, what is your best course? To double down and not admit a mistake may be your best move!
To those praising the culture of criticism in economics, I have to say that it doesn’t seem productive. It doesn’t feel methodologically driven, it feels like ideologically driven schoolyard cliques. Whatever your general working philosophy is, you’re guaranteed to receive disproportionately harsh criticism from the other schools, and the opportunity for harsh criticisms are always infinite because it’s a social science and naturally rests on assumptions of subjective and changing soundness. How insulated your career is from said criticism seems to depend on how big the names of your friends are and how mathematically difficult your arguments are, not how sound they are as they apply to real economies. And the criticism more frequently crosses the line into naked insults and chest thumping in economics than in seemingly any other field. I can’t help but feel that the existence of ejmr is in some way indicative of the field’s Id.
I don’t disagree with your argument that criticism is hard not to take as personal and that it can seem in the authors best interest to double down. I don’t think that it always is in their best interest, but in some situations it probably does save some face. Related to this, in the linked Medium post above I describe how I don’t its fair to go after people who were critiqued because their immediate reaction is to defend or not admit weakness. If they publish a defense though, then it seems more fair to assume that they really mean what they say.
But I also think the taking it personally part isn’t well aligned with how many critics feel, hence the part of my post that you quoted. Many of the people I think of as critical are much more frustrated because they want things to be better in general than out to get specific researchers. So interpreting it as a personal attack is not really that helpful because it isn’t always meant that way.
Even if the critic doesn’t mean for their criticism to be a personal attack, it can still be a personal attack. A well-meaning (and valid) criticism can still tank someone’s career. In general, I’m not sure how much intention matters.
A
If a valid criticism does “tank someone’s career,” maybe that career should be tanked. Do the students at Columbia University or the taxpayers of Ohio or whatever deserve to have their valuable money paid to someone who is doing low-quality work? I object to the idea of privileging a researcher’s career as more important than progress in science and engineering. I’m not saying the critic should have a goal of harming someone’s career; the critic should be pointing out issues with the research, with the goal of helping the original researcher, others working in that field, and society in general (not necessarily in that order). There’s no reason for the researcher’s career to be a concern of the critic, one way or another.
My objection here is that the present academic system keeps every non tenured career in increasing precarity for increasingly long. Young researchers feel like their whole world can come crashing down at any moment because, insofar as their academic career is their whole world, it can.
Most of the subjects of this blog are those frustratingly above criticism, Epstein and Gladwell and whatever. But if either of you, Jessica and Andrew, wanted to ruin a random grad students career for the fun of it, you could probably write a few antirecommendation letters and have a major effect in less than a day unless they were well connected to someone equal to your stature.
Criticism is always going to feel bad, and some personalities get defensive no matter what. But young academics are in a position where they’re always extremely replaceable and the opinion of people of stature is the sole determinant of success. So for them, public criticism is a veritable crisis.
I don’t think it should be, or that you shouldn’t criticize anyone who makes a mistake. Rather, I think the system is built such that hearing criticism calmly is difficult and accepting criticism can be against one’s best interest.
And on the flipside, many tenured faculty members seemingly don’t know how to behave like adults, and will snarl capricious criticisms and insults behind each others backs. Nearly ever person I know in academic has a story of a manchild faculty member throwing tantrums in ways I have never seen in industry, and every department always seems to house at least one intense feud. So maybe “proper channels” and process have to be established just to prevent constant hissy fits and warfare.
Andrew said:
“If a valid criticism does “tank someone’s career,” maybe that career should be tanked. ”
Yes. Absolutely.
But what’s really surprising to me about this discussion is why people aren’t prepared for criticism. For goodness sake that’s half the job of research: answering to the potential shortcomings of the work! People are supposed to understand the assumptions they make, recognize the limitations they imply, be able to discuss those limitations and suggest research to address them in the future. That used to be what comps were all about didn’t it? If you couldn’t defend your research in your comps, then you didn’t pass to candidacy.
I guess that’s all gone by the wayside in the rush to get grant money. After all, if faculty don’t have PhD candidates, they don’t have anyone to do research, so they can’t advance their own careers. So if the candidate quality is low, the obvious solution is to lower the bar.
Somebody said:
“My objection here is that the present academic system keeps every non tenured career in increasing precarity for increasingly long.”
Only because so many unqualified people have been rubber-stamped up the ladder that the number of applicants for tenure jobs is way out of balance with the number of jobs. It would be much better for everyone, including the people that **don’t** succeed, to be pushed out of the system earlier, before they commit so much of their lives to it.
That, in part, used to be accomplished early with the undergrad “knock-out” classes in every discipline. But I guess everyone from faculty to department chairs to deans figured out that knocking people out early meant they couldn’t be milked for more dollars (dept TA dollars are handed out by the number of majors in part, right?). That, combined with the whining of students, lead to knockout courses being watered down and eliminated and a similar mentality propagating up through the system; grade inflation, and ultimately way to many job candidates at the top level.
Complete nonsense. The ratio of applicants to PhD programs to available positions has never been higher. The ratio of PhD holders applying to academic jobs to the number of positions has never been higher.
https://www.science.org/content/article/stellar-opportunity
Tenured faculty don’t have to “lower the bar” to get enough researchers working for them. The bar for PhD recipients could be considerably higher and there would still be a surplus of researchers. The problem is that nobody agrees on what “raising the bar” would even mean.
Somebody:
Yeah, we get ridiculous numbers of highly qualified applicants to our PhD programs. Then again, often we admit students who then decide not to attend, as students will sometimes apply to dozens of programs. The whole system is a mess.
Somebody said:
“Tenured faculty don’t have to “lower the bar” to get enough researchers working for them. The bar for PhD recipients could be considerably higher and there would still be a surplus of researchers.”
Do you have trouble reasoning?
If the bar were “considerably higher” the number of applicants would drop like a rock!!! The reason there are so many applicants is that they know the bar to candidacy is low, meaning they have a great chance of succeeding to candidacy regardless of their ability. Any quack proposal will get them through the mill.
Andrew: if there are so many qualified applicants, why are we having this discussion? And why do we have so many discussions about poor work? Why is work so poor that a scientist from a traditional hard science discipline can easily recognize how bad it is without any guidance whatsoever?
I’d suggest it’s because of exactly what I pointed out. Being an R-wizard or stats geek isn’t enough. PhDs need to be able to present their work, anticipate criticisms and defend it, and they should be able to think on their feed enough to make a good showing when unanticipated criticisms emerge. If those abilities were part of “the bar”, I’d suggest the number of qualified applicants would be substantially fewer.
Chipmunk:
You write:
I agree. Just to be clear, though, our PhD admissions at Columbia (in statistics or political science) does not select on “being an R-wizard or stats geek.” I think it’s mostly about grades, test scores, letters of recommendation, and occasionally about other things such as projects that students did while in their undergraduate or M.A. programs or at work. Anyway, the requirements are very stringent: we almost only admit students with near-perfect grades, test scores, and recommendations. It might actually be better for us to relax on those standards and admit some R wizards and stats geeks.
Chipmunk:
As I said before
I agree that candidate quality is not good, but it is not because the snowflakes got rid of weeder classes or whatever. The old bars you propose, like good grades in rigorous coursework and standardized tests, have not been lowered, they have been raised and raised and raised. The kids have broken them.
I think ability to propose a philosophically grounded, promising research program and respond to criticism by improving one’s work are more important than being extremely clever. But those were never standards in the glorious past. I’m just guessing here, but I think it didn’t have to be, the kids were probably more self-selected for curiosity since academia was not as glamorized. But now that everyone wants to be an academic for some reason, it’d be a great idea to make that a standard and filter on that. But how do you do it?
Your comment makes sense for undergraduate programs, but it does not propagate upwards to the PhD level because the incentives are fundamentally different. The PhD candidates do not pay the university. PhD class sizes remain extremely small. While the number of PhDs awarded is increasing, it is not ballooning like undergraduate degrees. The number of undergraduate degrees awarded increased from 1.2 million in 2000 to 2 million in 2020.
https://nces.ed.gov/programs/digest/d21/tables/dt21_322.10.asp
The number of doctorates awarded increased from 41000 to 55000 over the same period.
https://ncses.nsf.gov/pubs/nsf23300/data-tables
With drastically more candidates every year and not more admittance slots, the deciders for PhD program admits are not then throwing darts at a chalkboard; the apparent standards are going up. A competitive graduate student applicant today is so far beyond the level of undergraduate weeder classes that their influence is totally irrelevant. Students are taking algebraic geometry as undergrads and being rejected from graduate programs, they were never going to be weeded out by single variable calculus, and at this point most likely aced it in high school anyways. Where standards are measurable, like GRE scores, they are increasing (outside of verbal/literature subscores).
I am not saying that actual candidate quality is increasing, because I don’t think it is. I am saying that along the metrics you are proposing, like rigorous classes and test scores, candidate quality appears to be increasing. Rigorous classes and test scores serve as good minimum standards, but at some point saturate and decorrelate from ability as a researcher. If you increase them any more, you’re just going to select for candidates who don’t make mistakes (and, therefore, who don’t take risks). We’ve probably already passed that point, which is why recommendation letters are now the dominant factor in graduate student admissions. Which then leads to recommendation inflation and cronyism.
With such a massive increase in the candidate-position ratio, whatever hoops you throw up, too many kids are going to jump through. The salient question for me is why do so many kids want so badly to be professors? You take a low pay for a so long, at the absolute mercy of unaccountable professors who may or may not be decent people, often getting entangled in their proxy wars against each other. Many people eventually get fed up and have successful careers as engineers in industry making much more money working for less capricious bosses, but still live with a lingering sense of having failed at their original mission. They could have happily been great engineers the whole time–what is the allure of the misery of academia?
> Where standards are measurable, like GRE scores, they are increasing (outside of verbal/literature subscores).
Do you have a reference for this? It seems plausible, but a cursory web search doesn’t turn up what I want.
I think there is a durable belief that being a Tenured Professor at a University is a very sweet lifetime gig. And it really is compared to the great majority of its alternatives. Traditional universities are attractive places to work with lots of amenities, you’re surrounded by young people, on the whole there’s more tolerance of personal quirk and eccentricity than many places and there’s a very clear hierarchy with the Tenured Professor ensconced well toward the top.
Of course nothing is “lifetime” nowadays (beside maybe being a Supreme), only a tiny proportion of academics will ever become Tenured Professors, many higher education enterprises are more akin to degree factories than “traditional universities” and the relatively few of those sweet gigs remaining tend to be filled with people who hang around into their 70’s and even 80’s. So that dream, to the extent there’s reality remaining, is getting to be something like becoming an athlete with an eye toward a gig as a multimillionaire famous professional.
Higher education trades on that dream to staff itself with overworked, underpaid, not particularly appreciated workers on the degree assembly line. There are plenty of wicked smart, awesomely hard working, well educated applicants for even the jobs that are basically school teachers trying to impart the education we once thought could be taught in primary and secondary schools. It has worked out well, there haven’t been all that many more successful growth industries over my adult lifetime than higher ed (in its most inclusive sense).
“There is some implication here that visceral and fearful reactions to criticism is irrational; more disturbing is the possibility that it is rational.”
Such fears are rational to the extent that the profession has created an environment in which criticism is unexpected and therefore unwelcome. If that’s the case then faculty are doing a **TERRIBLE** job preparing their students to do **science** – although they may be preparing them well for some other pursuit. Perhaps, through a combination of the Scientist as Hero meme and pure pursuit of self interest, researchers have come to see questioning each other’s work as morally unacceptable – in the first place because it undermines the heroic pursuit of justice; and in the second because it undermines the researcher’s pursuit of their own self interest.
good post…sad that it needs to be written, but it does.
what does everyone think of the thing where you have to say “this was incredibly fascinating” (etc) before asking any question at conferences….
> what does everyone think of the thing where you have to say “this was incredibly fascinating” (etc) before asking any question at conferences….
Very annoying indeed! We only have 5 minutes for questions (usually less since the speaker goes over). Why waste precious seconds on meaningless boilerplate?
I think “somebody” above makes good points. Ideally, criticism would just be ideas but practically it’s interwoven with people.
Consider also the hypothesis (not sure what the current evidence base is but here’s a relevant paper [that I have not read], HT to Marginal Revolution: https://web.stanford.edu/~pdupas/Gender&SeminarDynamics.pdf) that there are differential amounts/quality of criticisms based on presenter identity.
Obviously though as you and others have mentioned here, there are strong benefits to open criticism so it would be great to have a solution that keeps the open criticism but also allows for inclusivity. You hint at a solution when you mention support systems and somebody’s remarks about the effect on careers also may suggest a solution. So I think there’s hope but no easy answers!
Things are also complicated by the fact that this issue within academia intersects with the general public debates on free speech, censorship, equity, etc.
The difficulty with this is that there is a fine line… and not even in an objectively fixed place, as all this depends on perception, previous experiences and the like. Criticism can be helpful and is surely needed in science, but in can also be (more or less consciously) personal, unfair, and/or politically motivated, an instrument of power, and it is often hard if not impossible to cleanly tell apart the helpful and unhelpful parts. One can decompose any communicated “message” into Factual Information, Appeal, Self Revelation of the communicator, and message concerning the Relationship between sender and receiver (https://en.wikipedia.org/wiki/Four-sides_model), and will often find some less nice aspects on top of the Factual Information even if the latter is accurate and potentially helpful. In fact the model also comes with the idea that the four sides are relevant both on the sender’s and receiver’s side and may play out and be perceived differently (and potentially even differently in the position of an external observer, who will bring in their own background when deciphering the message).
Criticism is necessary in science and can be very helpful, but the critics have to be criticised as well, and should try to be self-critical (not only regarding their own work but also regarding their way of criticising) and aware of the issues that come with criticism. This is not really meant as disagreement with the original post, but I think it helps to explain and understand what is going on.
I remember feeling this way as a seminar student. I’m not sure I got this type of criticism myself (I tended to get crickets, which is actually worse). But I remember incidents in which a number of students were mortified by the tone taken by some faculty in questioning a visiting speaker. Do we really want to join a profession where this is the norm? Many of us did not. In fact few of the students I can think of who were in that seminar went on to jobs in similar departments. One respecialized, one moved over to education, a bunch of us left the faculty track altogether. I did a stint at the National Science Foundation as a AAAS fellow and in that capacity I got to witness a lot of panel discussions of grant proposals, organized by subfield. That was illuminating. In these panels I saw that the numeric score ranges for the proposals were similar across fields, no evidence of score inflation, but the panels closest to my field of origin were easily the nastiest in tone. It gave me a new perspective on why I’d been so unhappy. You can have a productive conversation that engages enough with the problems to move forward without being rude or indulging in personal attacks — and science is much more fun that way, because empty praise is very boring. It just takes social skills that are in short supply in some places.
I had several PhD advisors. One of them very germanic — totally honest, never sugarcoated things, but always courteous, never dismissive. The other stereotypical American, would always disguise his true opinion, with the goal of making students feel good. Needless to say, I learned so much more from them Germanic style. Parallels to seminar culture are obvious.
This discussion seems to leave out several items. First, context. An audience member at a talk may make critical comments, but need not make a scathing excoriation or be dramatically offensive. Those types of comments are generally unnecessary in this context. However, in a different context, such as a review, or a presentation that is a proposal, or is meant to provide a justification for a proposal or concept it is often the job of the reviewers to identify errors and critical issues. In this context, frank criticism is essential, although it need not be scathing nor imply negative innuendo to regarding the individual presenting. Second, even in oral presentation or talk scenarios, there are ways to raise questions or reveal doubts without denigrating either the effort being presented or making denigrating the information or person presenting. I know some great scientists who can raise questions doubts about information being presented using artfully phrased questions. I have also seen others who simply denigrate an idea, or worse, the speaker. It takes knowledge and sensitivity to be critical but not derogatory. In my experience, most derogatory comments are made people who are extremely egotistical, and whose intent is mostly just to show off, garner their own personal attention, and be noticed. Their main interest is in promoting their own visibility, not in contributing to the ideas presented. Criticism is an important part of gaining or refining knowledge and skills. Most of us would do well to hone our skills at articulating issues or questions about ideas presented without disparaging the ideas and work and without ridiculing or casting aspersion on the presenter.
Yes, this accords with my experience also.
Tom Lehrer pointed out that, “When properly viewed, everything is lewd.” And so it is with regard to most aspects of life in this modern era. No remark, no matter how innocent and well intended, will be taken as undue criticism. My personal favorite has to do with bread baking. I wrote an article for my local co-op regarding my recipe/technique and mentioned that males find kneading especially satisfying precisely because, unlike females, males are rarely doing the nitty-gritty in the kitchen. Charges of political incorrectness were hurled for which there is no defense.
This is a fascinating discussion for me. Like many of you, I’ve had a lot of first-hand experiences, some of them contradictory. I’ve also sat in on seminars, panels and committees in a range of fields and over a long period of time. There’s so much to say.
1. What’s different? I think a lot of the “anti-criticism” tide is the result of a variety of extreme subjectivism that has established itself through much of academia and political discourse. It has many strands, but the relevant one here is the view that affect is “real” in the same sense that any other aspect of reality is. Making someone feel bad is an assault on their well-being in exactly the same way that physically assaulting them is. Obviously, I wouldn’t defend the opposite, subjective-experience-isn’t-real, pole, but surely there is an intermediate position that makes more sense. But according to the subjective-is-objective view, if criticism makes the target uncomfortable, anxious or unhappy, it’s an attack and must be discouraged. (A lot of what’s problematic about social justice activism has nothing to do with social justice, but rather its incorporation of extreme, counterproductive subjectivism.)
2. Criticism and hierarchy. In a lot of fields, including my own of economics, criticism in public venues is the privilege of hierarchs. There is a reasonable aspect to this, to the extent that high position and influence is the result of knowledge and insight; this is often true! But some of it is performative, a way the higher-ups express dominance over the less exalted or signal their inclusion in a rarified agonistic circle. That’s messed up and has little to do with the noble goal of critical scrutiny advance by Jessica, Andrew and others. A personal story. I used to teach at a large research university whose economics department (at the time) had one especially well-known and influential professor. He gave a talk (this will reveal his identity to those who know about such things) in which he claimed that essentially all of the measured pay discrimination against women and black people was not discrimination at all because they are less productive than men and whites. He had an econometric model for this, of course. I sat there in the back, waiting for someone to challenge him. There was lots of polite murmuring, nothing critical. Finally, as the time allotted was about to expire, I raised my hand and asked whether his assumption that wages always reflect marginal productivity might itself be undermined by the existence of discrimination. He gave me a withering stare and said something dismissive that I can’t remember. Later I asked a friend, a female economist, who had been at this talk why no one with more standing than me challenged this guy. She replied that, if you’re junior and care about your career or ability to get grants, you don’t disagree with someone like that in public.
3. Economics. This is held up as the model child whose critical behavior in seminars etc. should emulated by all the others. I would qualify this by my observation of (2) above, but there is something to it. One aspect, not the only one, is that economics has been far less influenced by recent subjectivist fads. On the contrary, the field sees itself as having systematically rooted out subjectivity through its embrace of revealed preference and clear delineation between what is observable and what is not. And there’s something to that! My own sensitivity to what I see as wayward subjectivism could be a product of my disciplinary background. On the other hand, the dominant economic worldview (welfarism) rests on its own set of highly subjective and largely invisible/unreflected assumptions, so it would be more accurate to say that the field is resistant to alternative subjectivisms. That’s obviously a prelude to a different discussion.
Enough for now.
My experience of economics is that creating a culture of open criticism without fixing the underlying mismatched labor market or the unaccountability of professors has created a gang system. New researchers seek the protection of a “made guy,” who is their corridor to access the top 5 journals. Until they get made, they cannot pick a fight with a made guy, but the other made guys may pick a fight with them. They’ll be okay, as long as they are under the protection of someone made.
As an aside, the way professors in economics will openly talk shit about other professors with their students and underlings is unrivaled. It was honestly extremely funny to experience, especially when they used their favorite insult “noneconomist.” But I can’t imagine it’a a comfortable work environment, when professors are trying to convince the incoming class to hate other professors.
Love the mafia analogy. So on point.
We should be skeptical about emulating anything that goes on in economics. What’s the word for this phenomenon, where if a group of misfits do a thing…then you probably shouldn’t do that thing?
I largely agree with you, and many of the other commentors from the Econ field. As a graduate student I was really put-off by the demeanor of some of the older and more reputable faculty members during visitor seminars. I will say though that certain questions seemed to be *off-limits* because they could be levelled at just about any empirical economics paper. Echoing somebody’s thoughts above it seemed to me like one could always come up with a methodological critique if a paper challenged one’s ideology.
This is not limited to econ. The best recent example is ivermectin for river blindness vs for covid.
There is actually better (although still very crappy) evidence it helps with covid, but one yields a nobel prize and the other disparagement. There is essentially no objective evidence involved, only people indirectly measuring collective priors on the various plausible explanations.
The problem is that applying the appropriate amount of skepticism to the enture field makes you realize there is no way forward without giving up some sacred cows.
Eg, a lot of researchers would not trade the idea that amyloid beta causes alzheimer’s for a dementia cure. Or how about if we needed to drop the idea “smoking causes cancer” to get a cure for cancer? If you are honest with yourself, you realize most people would not be willing to make this trade. There are a lot of “unquestionable” premises like those that have accumulated.
Anoneuoid –
> Eg, a lot of researchers would not trade the idea that amyloid beta causes alzheimer’s for a dementia cure.
Is there no limit to your mind-probing capabilities?
Thanks for the perspective. #1 does a great job of capturing what makes me uncomfortable about the attitude that because an author will be emotionally distraught, critiques must be limited. Feelings are real, and I am not trying to argue that they aren’t important to consider or even that we can critique in ways that would avoid any bad feelings. It’s the seemingly predetermined conclusion that since bad feelings can arise from open critique, then we must avoid it that bugs me. I can only speak for myself, but having been critiqued as a student, as much as I hated the experience, I knew deep down that a world in which my feelings were so precious that the critic is banished was not a world I would be comfortable with. If there really was something wrong with the work I did, I wanted to know it.
I was thinking about this again tonight, and I realized I misremembered the talk I attended in (2). The speaker had constructed production functions that enabled him to calculate the marginal productivity of different job categories, and then he assigned those productivities to the individuals holding them on the grounds that each person ends up in the job that best employs their human capital. From this he inferred productivity differences between whites and blacks and men and women that more than offset their measured pay differentials. The question I asked was whether the sorting of individuals into jobs could itself be influenced by discrimination. That was the criticism he regarded as utterly out of place. Sorry to get this wrong the first time. It was a long time ago….
Peter:
Yes, this seems often to be an issue with economists. They are fine with aggressive criticism and debate within the confines of their assumptions, but they can get all defensive if you criticism their overall framework.
To be fair, I guess that political scientists are similar. In a political science talk, it would be fine to question the assumptions of “Econ 101” but it would seem really weird to ask if the speaker has considered whether dictatorship might be a better form of government than democracy. In contrast, in an econ talk it might be considered charmingly counterintuitive to make an argument in favor of dictatorship but it would be considered obnoxious to argue against the concept of consumer sovereignty.
There should be two different kinds of universities. One that doesn’t hurt anyone’s feelings and one that might. We can then all pick and choose.
Jbayes:
Universities are big places. Columbia has both me and Dr. Oz! OK, they don’t have Dr. Oz anymore, but you get the idea…
Some thrive in contentious environments and others in safe environments, so there’s a benefit to having both. But they don’t seem compatible, so a separation works.
Plus I’m curious to see how well each does.
Jbayes:
It’s a fun idea. Maybe they’d have to split each university into two parts, the way they split the University of Louvain (Leuven) into two parts for the two languages.
Over a couple of years, I noticed that many people are ready to take criticism when giving a talk. I feel that people are mentally prepared to take some punches because they are already in a boxing ring. However, on social media like Twitter, many people consider criticism of their work as a “call-out in public”. It can be considered that way for some people who engage with others in and outside their work professionally and casually using social media. I agree with the general feeling that the field of science should be open to criticism and lively discussion about accepted work, but my point is that we should pay extra attention to the words we use, ***especially on social media***, because people don’t want to be embarrassed by their colleagues in front of their family and friends.
Thanks for a good post.
Nearly all of the worthwhile research and work is being done by crazed men to publish on their websites, because of nonsense such as this. Unsurprisingly, opening up university to everyone utterly destroyed the value of university to those who would benefit from it.
Academic standards vary.
The physics department at my uni (I have a physics PhD) has a colloquium which has consistently good talks. The CS department used to have a colloquium which had an occasional good speaker (Geoff Hinton just before he became a superstar) but frequently had people who just didn’t know what they were talking about.
There is a physics prof who I call ‘the meanest physicist in the world’ who through a series of unfortunate events I was not on speaking terms with that nevertheless we would go to the CS colloquium and tear bad speakers apart together like those crabs that devour starfish. I am sure both of us were appalled that people who were giving awful talks were getting away without any criticism. In the end they cancelled the whole colloquium and now the CS department doesn’t even try to put in talks of general interest.
You might call us sadistic or cruel and you might be right but we were not going to the CS colloquium just to make trouble, I had become a software developer and he had pretty much stopped doing physics to supervise a software project. I was still reeling from quitting physics myself where the APS had announced that the odds of a physics PhD getting a permanent job in my cohort were 2% while it was pretty clear that people were getting away with murder in CS because the demand for undergraduate degrees was so high.
I have admittedly not read everything in this very very long comment thread, but one point I think I haven’t seen raised: an important way for senior people to pass along the culture that polite, respectful, but stringent criticism is a good thing is to model this behavior in **more private** settings – one-on-one meetings and presentations in lab meetings – i.e. in spaces that can be both safe and critical. In a recent meeting someone said (and I agreed) that “we want students to feel safe at all times, but not always comfortable”.
Agreed. I’ve never really been good at holding back criticism when collaborating with others on projects (often Phd students), and I find that the more senior I get, the harder it is to not be blunt. But I’m pretty sure the students that work with me come to realize that I’m 100% behind them as long as they’re committed to doing good work. So the most valuable thing I can give them is my honest opinion at all times.
What are comprehensive exams are like in the social sciences? Anyone? Come to think of it I’ve never heard anyone talk about it on this blog. Are there comprehensive exams in the social sciences? Is there any barrier at all to PhD candidacy?
At my undergrad school – a tech and engineering school (with almost no social sciences!) – comps were feared by PhD students in all disciplines and while it wasn’t an every day occurrence people failed comps and left the program. Comps had both oral and written components. The oral exams included several sessions of several hours over a few days.
Some of my undergrad friends wound up failing comps at other schools. At the PhD program I wound up in, the attitude was that PhD students had already proven their mettle and they’d be better off spending their time doing the research that would lead to publications than preparing for comps. My comps were embarrassingly easy. This is the exact attitude that leads to problems later on and undermines the quality of PhDs.
Comps and thesis proposals are important. Without them students never get challenged on anything.
Chipmunk:
It depends a lot on the department. Every Ph.D. department I know of has a requirement of a thesis proposal and a thesis. Some departments also have exams after the first or second year that serve as barriers to progress until students are deemed to have mastered the fundamentals. At some schools the failure rate is high; at other schools the failure rate is low and students get a chance to re-take the exam if necessary. Either system can make sense: there’s the idea of admitting more students than there is room for in the program and using the exams to select the ones who can proceed, and there’s the idea of admitting students who will be able to do it and using the exams to check that they have no major gaps in their knowledge.