I came across this book from 2018, Interrogating Ethnography: Why Evidence Matters, by law professor Steven Lubet. It’s a crisp (137 pages) and fascinating discussion of the role of evidence in qualitative social science, and I think it should be of interest to many of you, as it parallels so many discussions we’ve had over the years regarding the role of evidence in quantitative research.
Sometimes I’ve had negative reactions to writings by law professors on social science, but in this case Lubet’s expertise is relevant, as so many legal cases turn on evidence.
Lubet discusses several examples, focusing on sociologist Alice Goffman’s controversial 2015 book On the Run. As we discussed a few years ago, it’s a problem of trust. Goffman offers no documentation for her extraordinary claims and thus must rely on her readers and colleagues to trust her statements and treat them as fact. In this case, trust is brittle, and once the trust is gone, not much remains.
One reason Lubet’s book is interesting is that he gets into the details and presents things very carefully. Just for example, from page 131:
It is unfortunate that ethnographers have so seldom essayed revisits to others’ research sites. Despite the obvious difficulties, there are cases in which the impediments can be readily overcome. It would not take long for an ethnographer to interview personnel at the hospitals in West Philadelphia where Alice Goffman claims to have seen police cordons at the entrances. Moreover, there are only six hospitals in Philadelphia with maternity services, so it would be possible, even now, to fact-check Goffman’s story of having observed the arrests of three new fathers on the same ward in a single evening.
I’m guessing that this maternity ward falls into the same category as Marc Hauser’s monkey tapes, Brian Wansink’s bottomless soup bowl and his 80-pound rock, Diederik Stapel’s survey forms, Mary Rosh’s survey forms, Michael Bellesiles’s probate inventories, Matthew Walker’s National Geographic video, the Surgisphere dataset, and Dan Ariely’s paper shredder. But all things are possible.
The other thing notable about Lubet’s book is its even tone. Some of the stories in the book are funny, others are kinda shocking, and Lubet manages to convey all this without himself ever expressing amusement or outrage. There’s nothing wrong with expressing amusement or outrage—I do it all the time!—; it’s just impressive to me how he wrote this entire book with a straight face. I recommend it.
I think you mean ‘Interrogating Ethnography’ rather than ‘Ethnicity’, but sounds like an interesting book.
Fixed; thanks.
Looking at Goffman’s wikipedia page, it’s really interesting – and I fear emblematic – to note how politicized scientific criticism is presented: “Conservative law professor…”, “On the left…” etc. Fits into what I perceive as a more general, ‘culture war’-fueled trend to implicitly define bias of evidence not with respect some parameter of the true data generating process, but rather with respect the central tendency of some ideological opinion space. Depressing. Had to smile at the Andrew Gelman blog, good to see how much of an impact this blog can have.
I went to read the Wikipedia page after reading this comment and found that the conservative professor was … Amy Wax. In my opinion, omitting the description would be pretty misleading.
I was coming back to this comment to say the same thing. To be honest only referring to her as “conservative” is probably under-informing the reader with respect to her point of view.
I think there a couple of questions to ask ourselves here:
Are political/ideological labels like ‘conservative’ , ‘on the left’ etc. useful signals for transferring information about the quality of the argument? I’m not so sure. And as pointed out by Michael, the term ‘conservative’ even seems rather uninformative here in particular. As a non-US academic, I’m was not familiar with Amy Wax, and I can’t say that ‘conservative’ would have allowed me learn much about her viewpoints. To me, it would seem like the only people would be able to take a meaningful signal from this word, are either those that are already familiar with Amy Wax (in which case, it doesn’t really add anything) or those that take a general term like ‘conservative’ as a heuristic cue to disqualify an argument.
Let’s assume for the sake of the argument that political/ideological labels do have useful informational content for the quality of an argument. After all, the original blog post did make a point about the trustworthiness of the source, which implies that it can be useful to rely on such heuristics. The question for me then would be: ok, are willing to value this informational content above the potential negative impact of framing scientific debates as political debates? What message is that sending? Is this the trade-off we are willing to make? Again, I’m not so sure.
On a medium like Wikipedia, which is widely used by academics and the general public alike, what standards should be applied to judge which criticism/controversies are included on a page, and how are these standards communicated? After all, there is a selection process going on here. I can imagine two criteria here a) scientific relevance and b) some kind historic relevance (e.g. bad criticism might still be included because they had some kind of societal impact). And if it’s both, I’m skeptical about the ability of most people to see the differences when these criteria are not clearly delineated, as is now the case (to the uninformed, the wiki page more or less places Andrew’s criticism in the same category as those of Amy Wax).
Amy Wax publicly endorses the idea that the country will be better off with more whites and fewer nonwhites.
OK, here’s my question. It says that Lubet first wrote on Goffman for the online magazine, The New Rambler. I was curious so I searched online . . . The New Rambler still exists; it’s here. But it seems that they’ve been slowing down. They started off with a rush in March, 2015, publishing multiple articles per month. By 2017 they were down to approximately one a month, then they had a two-year hiatus. After their relaunch in 2019, they were publishing multiple articles per month for awhile. But in 2023 they only had 7 articles, and in 2024 they only had 2.
I don’t know their business model. Maybe originally they had some funding and could pay for articles but then they ran out of money?
What happened? If you go to their website, there’s no announcement that they’ve slowed or stopped publication.
I mean, as far as I can tell, it’s really just a website maintained by some academic enthusiasts, so I’d guess all it requires is some folks willing to invest some of their time and some hosting costs. In that case, it wouldn’t seem surprising that output varies considerably over time.
Almost a year ago, I asked on Bluesky for some qualitative research fraud references. I found the NYT Magazine article of Goffman was a nice synopsis: https://www.nytimes.com/2016/01/17/magazine/the-trials-of-alice-goffman.html
Some other examples show the potential value in investigating ethnographies:
Finnish autoethnography that resulted in a university research integrity investigation: https://tenk.fi/sites/default/files/2020-06/TENK_annual_report_2019.pdf
Dutch political anthropologist Max: https://www.dutchnews.nl/2013/09/professor_faked_61_pieces_of_r/
and there’s always the American historian Bellesiles:
https://en.wikipedia.org/wiki/Arming_America
Maybe some fields are easier to muck up?
Jay:
There’s also this story from the sociology department at Columbia.
The abovementioned NY Times article is outstanding. I read it in the context of my critique of the completely debased academic discipline of sociology.
https://www.nytimes.com/2016/01/17/magazine/the-trials-of-alice-goffman.html?smid=nytcore-android-share
The crazy Alice Goffman story illustrates a special case of the decline and intellectual bankruptcy of sociology. Goffman was accused of doing descriptive ethnology work on marginalized populations without accounting for her own privilege, i.e. ignoring ‘positionality’. So of course she was guilty of being exploitative and stereotyping and overstating the sins of the group she studied.
The author of NYT article makes the point that if you prohibit Goffman from studying poor communities of color, you should also prohibit sociologists who grew up in poor communities from studying elite institutions like hedge funds.
But sociologists will tell you that Goffman was ‘punching down’. This is not a partisan or ‘culture wars’ argument. There is agreement across the political spectrum that sociology can no longer be taken seriously. For example, Professor Hazlit, who is a liberal, states that the premise in sociology ‘is that men/white people are the perpetrators of all that is wrong in the world. This is an unquestioned assumption”. See his quote (and many others) on the page below:
https://www.ihatesociology.com/voice-of-the-public
Separately, the only book I recall reading on the social sciences by a law scholar is Melissa Jacoby’s ‘Unjust Debts’ which is well-written and for a general audience. I disagree with much of it, and chapter 2 on the structural racism of the bankruptcy code is wrong according to commonsense and the experts I have spoken with. I like articles by law professors. They are well-reasoned, pay attention to facts and often well written. Also, unlike the hermetically sealed ultra suspicious cult of sociology, law professors usually respond to emails.
Who accused Goffman of punishing low? Naturally, she didn’t “punch” the people she studied, quite the contrary.
No one who like Hazlit thinks that “It is only in recent years that it has become political” (from the references quote) can be taken seriously.
It has been highly politicized for more than half a century. Also, the quote about men/white people is not about sociology per se but “courses in social justice.”
Sociology is at least an empirical enough discipline not to have accepted any papers from the “Sokal squared” hoax.
Thanks for the correction on Hazlitt. I should have caught that.
To document the extent to which white men are under-represented in sociology, I suggest someone go to the websites of the top 200 sociology departments. Look at the profiles of the assistant professors and graduate students, and calculate the percentage of white men. Then see how many of the white men do not focus on LGBTQ activism.
Why is this? I have an explanation. Historically, Jewish men (like me), often red-diaper baby types from the East (unlike me) were over-represented in the discipline. I could guess at why this was also.
In any case (if my reasoning makes sense to anyone other than me), in my parent’s generation and in the USSR antisemitism was more open and the reasons were clear. Many entities in Milwaukee – the police and fire departments, corporations, corporate law firms, many German restaurants, resorts, etc – just did not welcome Jews. I have learned from speaking to countless Soviet era people, mostly in Brooklyn, is the the USSR just set quotas and counseled Jews to other academic departments and jobs when the quota was filled. This is how Columbia and Yale used to be.
In today’s discrimination, they deny it even exists and if you point it out they won’t give you a reason or even speak with you.
One of the most famous cases of alleged “fraud” in ethnography involves Margret Mead’s book Coming of Age in Samoa, which argued that adolescent psychosexual development was culturally variable, contrasting restrictive Western sexual mores imposed on young women w/ the supposedly much more relaxed Samoan mores.
I did a short twitter thread on it here:
https://x.com/ed_hagen/status/1637253768231407616
Basically, later ethnographic research seemed to suggest that Mead was duped by her young female informants, yet that claim was itself called into question. See the Wikipedia article linked in tweet #3 for details (the Mead-Freeman controversy).
Ed:
Vaguely relevant is this discussion from 2005.
John Maynard Smith, a major figure on the “nature” side, was also a communist early on. Bob Trivers, another major figure on the “nature” side, joined the Black Panthers:
https://www.thecrimson.com/article/1979/7/6/sociobiology-pioneer-joins-black-panthers-probert/
Ed:
This reminds me of a conversation I had with someone a couple years ago. We were talking about the left-wing slant of academia, and we decided that what had happened wasn’t so much that academia had moved to the left, but rather that the variance in political academics had narrowed. The distribution of political views of academics used to range from the right to the far left, whereas how it ranges from the center to the left. Roughly speaking, if we consider a scale where -5 = far left and +5 = far right, we could say that most academics fell in the range from -5 to +3, whereas now they’re mostly between -3 and +1. That’s not exactly right, and that’s just the center of the distribution–there are exceptions in the extremes–but that’s roughly where I see things. There are fewer academic conservatives, also fewer of the far left in academia.
Andrew, I’ve been hoping to stumble across a good survey of the political orientations of academics, in the US and elsewhere, broken down by discipline and by time period. One of the readers of this blog might know if it exists.
Ed: I’m a few years out of academics. But if we take Andrew’s -5 to +5 scale to reflect the distribution circa 1985 I think today you’d find the humanities extremely left skewed (-2 to -7) and the sciences moderately left skewed (+1 to -5) with rare outliers from that range. I agree with Andrew that there are overall fewer conservatives in academics. But I’d suggest in addition that the national left has shifted significantly futher left especially since 2000 and that shift is also reflected in academia.
I have seen a poll that indicates Journalism faculty are something like 90% Democrat, but I can’t lay my cursor on it at the moment.
Anon:
There used to be many prominent academics who were communists or left-wing revolutionaries in their politics. Now, not so much. So I completely disagree with your claim that the left 10%, say, of the distribution of academic political attitudes is further left today than it was a few decades ago. But I also agree with Ed that it would be good to do survey data on this. I agree that faculty are overwhelmingly on the left half of the distribution but I think they’re more concentrated in the left and center-left, not so much in the far left.
Daniel Klein and Charlotta Stern published a number of articles on the political opinions of social scientists about 20 years ago.
Neil Gross has published extensively on this topic in articles and in Why Are Professors Liberal and Why Do Conservatives Care?, and Professors and Their Politics (co-edited with Solon Simmons). More info on surveys of academics political views here: https://en.wikipedia.org/wiki/Political_views_of_American_academics
Andrew said: “There used to be many prominent academics who were communists or left-wing revolutionaries in their politics. ”
It’s hard to disentangle “prominence” and “abundance”. I’m aware that there have been communists in universities throughout the 20th century. But by the time I came of age in the 1980s, I percieved them as very small fringe group with no connection to power and taken seriously by no one. I think that’s a fair assessment for that time. That’s not true today. Like the 1930s, there is a much stronger leftist mileu and like the 1930s it has a strong connection with universities. I don’t think it’s an accident that Biden idolized Roosevelt. Among the left there is alot of nostalgia for Roosevelt right down to the “Green New Deal”. AFAIK, that’s what we’ve seen in the last few decades: the echo of the New Deal.
Modern leftists don’t call themselves “communists” and would emphatically reject that label. They certainly would not refer to themselves as “extreme left” on a survey. They think of themselves as the epitome of democratic – the ultimate centrists. They view it as unfortunate that there are so many “far right” criminals that refuse to conform to their beliefs and thus must be silenced and convicted. While communists and socialists of the past behaved in exactly the same way, they didn’t percieve themselves as “democratic”. :)
LOL
https://en.wikipedia.org/wiki/Weather_Underground?wprov=sfti1#Background_and_formation
“Goffman offers no documentation for her extraordinary claims and thus must rely on her readers and colleagues to trust her statements and treat them as fact. ”
This applies literally to every ethnographic study ever. The idea about fact-checking ethnography is ridiculous. To wit, I would like to have seen people try to fact check Goffman’s study. I am sure the guys she studied would happily have talked to people coming around asking questions. Or not. And I am sure the hospital personnel would have been equally keen to talk to “fact checkers.” Or not.
@Jay Patel: The NYT Magazine is not about research fraud. It’s about the attacks on her based on woke identity politics. This sums it up: “what frustrated her critics was the fact that she was a well-off, expensively educated white woman who wrote about the lives of poor black men without expending a lot of time or energy on what the field refers to as ‘‘positionality’’ — in this case, on an accounting of her own privilege.”
A lot of the “criticisms” against her had nothing to to with her research, it was ad hominem attacks based on her demographics. Plus, that she’s Erving Goffman’s daughter made more than a few people lose their shit, because privilege!!
As an aside, Bellesiles’ study is not an ethnography, he’s an historian. The study is about gun culture in the 19th century.
I’m not sure if the maternity ward claim is amenable to easy replication. Exactly what’s the claim in detail? If it’s that a few cops had a “custom” of running warrant checks on hospital visitor lists, that’s not inherently implausible. But that’s specific to those cops, perhaps doing it as part of a crackdown right then, with a friendly administrator. But all those factors need to be in place, which is not every day. It’s also something that asking hospital administrators directly may not get a truthful answer. Asking police who work in that area might be workable – “ever run a warrant check on a hospital visitor?”. But again, this gets back to the problem that proving it happened somewhere, won’t prove any particularly account happened. A quick search found people talking about incidents, e.g.
old.reddit.com/r/Psychiatry/comments/11n4kgw/when_your_patient_has_a_warrant_out_for_their/
“At a different facility, we had a patient who had a warrant for failing to show up to probation/parole check-ins. (It was strongly suspected that he got himself into the hospital as an excuse for missing some of them.) I don’t know who actually called, but it was known ahead of time on discharge day that police would be waiting outside the unit to arrest the patient the second he was discharged. We were warned to avoid the main entrance for a time in case the patient tried to fight or run.”
old.reddit.com/r/legaladvice/comments/6ceyil/il_possibly_need_to_go_to_the_hospital_but_have_a/
“My fiance has been in a similar situation where he went to the hospital and had a warrant, and he was unable to leave. The hospital released him to the police, and he eventually did time.”
GB, thanks for that pointer. Here are some results on political orientation from the UCLA HERI Faculty Surveys: https://heri.ucla.edu/publications-fac/
1989-90 (earliest report on that page):
Far left: 4.9%
Liberal: 35.6%
Moderate: 39.8%
Conservative: 19.2%
Far right: 0.5%
2016-17 (latest report):
Far left: 11.5%
Liberal: 48.3%
Middle of the road: 28.1%
Conservative: 11.7%
Far right: 0.4%
Ed:
Interesting. Still, I’m skeptical about the change in “far left.” I see from the documentation that this is from self-description on a 5-point scale. I’m guessing that far-left in the 1960s-1970s included lots of communists and revolutionaries, whereas far-left now is milder. I’m not sure about 1989-90. I guess the point is that absolute and relative scales can have different meanings.
Andrew, yes, fair enough. This is your wheelhouse, not mine.
In terms of identifying as actual Marxists rather than vaguely “far left”, I can’t find the 2006 Gross & Simmons survey, but this citing it claims 3% identified as Marxists across all disciplines. Within the social sciences, however, 17.5% did and 24% identify as “radical”. 25.5% of sociologists identified as Marxists.
I’m skeptical about the usefulness of measuring political orientation among academics, at least in the context of evaluating scientific evidence and criticism.
Issues of measurement validity and comparability are one problem.
Besides what was already mentioned, I (as a non-US, European academic) would add that the meaning of ideological labels varies considerably across regions and periods. E.g., ‘liberal’, ‘conservative’, ‘democrat’ and ‘republican’ generally have a very different connotation on our side of the big pond. Meanings also have – and are – changing over time, and that may be particularly relevant for how extremism is measured. Perhaps few academics would still refer to themselves as ‘communist’ nowadays, and probably almost none as ‘fascist’. But this may rather reflect a change of discourse than a change of underlying extremism. Based on my subjective experience, dogmatic thinking and willingness to impose viewpoints through force and intimidation have not exactly disappeared from academia and may even be making a comeback. It’s just that the words that are used have changed.
But more importantly, I think it’s a red herring. In my opinion, the pertinent question is not how political ideology is distributed among academics. It is rather a) how prevalent it is among academics to let the standard of evidence they require co-vary with their political/ideological orientations, and b) to what degree they are basing their evaluation of evidence on the perceived political/ideological of the messenger rather than on the quality of the argument. In my own subjective experience, both have been trending upwards quite sharply. At the same time, I wouldn’t be to too sure a priori that they correlate with political/ideological orientation, i.e. I think they may be equally likely across the political spectrum.