An economist who would prefer to remain anonymous writes:
There is an important question of the degree of legitimacy we grant to academic research as advocacy.
It is widely accepted, and I think also true, that the adversary system we have in courts, where one side is seeking to find and present every possible fact and claim that tends to mitigate the guilt of the accused, and the other side strives to find and present every possible fact and claim that tends to confirm the guilt of the accused, and a learned and impartial judge accountable to the public decides, is a good system that does a creditable job of reaching truth. [Tell that to the family of Nicole Brown Simpson. — ed.] (Examining magistrates probably do a better job, but jurisdictions with examining magistrates also have defense attorneys and DAs; they just allow the judge also to participate in the research project.) It is also widely accepted, and I think also true, that it is important that high standards of integrity should be imposed on lawyers to prevent them from presenting false and misleading cases. (Especially DAs.) There is no question of “forking paths” here, each side is explicitly tasked with looking for evidence for one side.
I don’t think that this is a bad model for academic policy research. Martin Feldstein, like most prominent academic economists from both right and left, was an advocate and did not make a secret of his political views or of their sources. He was also a solid researcher and was good at using data and techniques to reach results that confirmed (and occasionally conditioned) his views. The same is true of someone like Piketty from the social-democratic left, or Sam Bowles from a more Marxist perspective, or the farther-right Sam Peltzman from Chicago.
All these individuals were transparent and responsible advocates for a particular policy regime. They all carried out careful and fascinating research and all were able to learn from each other. This is the model that I see as the de facto standard for my profession (policy economics) and I think it is adequate and functional and sustainable.
Romer’s whole “mathiness” screed is not mostly about “Chicago economists are only interested in models that adopt assumptions that conform to their prejudices”, it is IMHO mostly about “Chicago economists work hard to hide the fact that they are only interested in models that adopt assumptions than conform to their prejudice”. I think Romer exaggerates a bit (that is his advocacy) but I agree that he makes an important point.
I’m coming at this as an outsider and have noting to add except to point out the converse point, that honesty and transparency are not enough, and if you’re a researcher and come to a conclusion that wasn’t expected, that you weren’t aiming for, that you weren’t paid or ideologically predisposed to find, that doesn’t automatically mean that you’re right. I’ve seen this attitude a lot, of researchers thinking that their conclusions absolutely must be correct because they came as a surprise or because they’re counter to their political ideologies. But that’s not right either.
People try much harder to dismiss/p-hack away results they don’t like though. So as a general rule, unwanted observations have been through a more stringent filtering process.
Eg, Ivermectin is considered a miracle drug for river blindness, despite no RCT and conflicting/confounded observational results. Usually the affected locations also take measures like clearing marshland at the same time, and even then there is often little change in river blindness after 15 years. Yet, nobel prize.
For covid it was obviously held to a higher standard due to increased skepticism. That is a good thing, it more closely approaches the scientific ideal.
Anoneuoid –
> So as a general rule, unwanted observations have been through a more stringent filtering process.
As the famous philosopher Sara Lee once said “Everybody doesn’t like something.”
Any observation amcsn be, and often is, considered as unwanted. The problem with your rule of thumb is that it’s more or less arbitrary in many important contexts. For example.
> For covid it was obviously held to a higher standard due to increased skepticism.
There are many, many who argue that IVM was held to a terribly low standard due to inherent biases. Or perhaps to an idrarjo ally high standard because it necessarily had to be rejected. Those advocates way that the “unwanted” evidence that it’s a miracle drug for COVID resulted in bad science. Of course others argue that the “wanted” results resulted in bad science (by those in the other side). There’s often no utility to your shortcut unless you start with a dubious premise, usually based on a particular prior. In which case all you’re doing is confirming a bias.
There were/are IVM advocates using the usual (low) standards. But that is totally unexceptional.
The exceptional thing was that some people figured out to use the correct (higher) standards. Unfortunately, most of them fail to apply those standards to other medical topics, so they were not behaving correctly due to higher principles. But usually this doesn’t happen at all (eg, IVM for river blindness).
Sounds like your academic economist is just that. My sister believed the US court system would find the truth at the point she finished law school. For example, she argued capital punishment was reasonable given how good the court system was at finding the truth. Of course, we know lots of innocent people are thrown in jail because their convictions are later overturned for very strong reasons. A year as a public defender convinced her the whole system is ridiculously biased. The biases include which judge is drawn, which prosecutorial team is drawn (they’re not all equally talented and don’t all have the same resources in terms of time), how good your defense is (often with enough money to bury the prosecution, which is onl a limited. budget), and then there’s the 12 random people aspect who are asked to make judgements based on an incredibly vaguely defined notion of “reasonable doubt”. The fact that juries are rarely unanimous should be a tip as to how uncertain things are after a trial. A further very strong bias, which is behind so many convinctions being overturned, is fraud on the part of the prosecution. Before that, there’s just the cozy in-relationship with the judge. Judges typically move up from being prosecutors, and bring their own pile of biases. The judge gets to decide what counts as evidence and who can be called as a witness and sets the tone by addressing the defense and defendant as if they were guilty children. The bias starts earlier with who is arrested—a trial doesn’t happen if nobody is arrested (and the dozens of police forces operating in most jurisdictions just add to the confusion). Then there’s managing evidence, which is also under control of the courts.
Now let’s look on the low end, where my dad and sister worked. You get a court appointed lawyer for a felony case (like murder or armed robbery), and your defense gets paid less than a single paralegal would get paid working for a rich defendant. You can’t afford expert witnesses. You can’t afford to delay for years with motions. Many of the court-appointed lawyers get piles of cases, a relatively small hourly rate, and just plead them all out. So the bias creeps in before you even get to a trial. My sister told me that after she started winning a lot of trials, the prosecutors would offer her much better settlement deals. Not the “truth” of the case, but the fear of the opposing attorneys. Different attorney, different result.
Not a good recipe for arriving at the truth. My father, a family lawyer, preached this his whole life. He urged me and my sister to avoid the law at all costs because the biases are so frustrating. He said that once he knew the judge, sex, employment status, marital status, and dependents for his client, he could tell you what the verdict would be. And of course it differed by judge.
Juries are rarely unanimous? I’m not disputing that, I’m just surprised. I suspect a major factor is that most cases of convincing guilt are settled/plead out of court, and most cases of convincing innocence are thrown out (or never brought). The whole thing should obey principles of arbitrage: If too few juries convict, more of the accused will take their chance with a jury, then more cases with stronger evidence against the accused will make it to juries, then juries will convict more often, then the accused will become more cautious and plead out more, then juries will get more ambiguous cases and be hung more, and so on. Ultimately, it should reach equilibrium. (In the abstract, of course. Both prosecutors and public defenders will always have incentives to push defendants to plead.)
Overall, though, you make good points. I couldn’t even finish season 3 of the Serial podcast because its exposé of the Cleveland justice system was so unrelentingly depressing and frustrating.
It’s definitely worth thinking a lot harder about the legal system’s epistemics as compared to scientific epistemics, and why they both are and should be different, rather than OP’s handwaving ‘eh they’re both kinda adversarial and kinda sound similar and involve a lot of words and people, good enough for me’. For example: a lot of legal procedure and rules is simply about restraining the power of government; even in the more policy-relevant applied areas of science, that’s not a major factor.
It’s also worth noting that it seems like most scientific evidence is fundamentally different than legal evidence, which are more like history – I saw this morning a reference to Einstein’s quip about 100 authors claiming to refute relativity that if relativity was false, ‘one would have been enough’, but of course, in a court, if 100 highly-credentialed expert witnesses trooped up to testify that ~X, the judge & jury would probably be convinced at least half-way that ~X. Conversely, what would be the equivalent of p-hacking samples until you got the desired result in a criminal case? After all, the crime is a single case, it either happened or didn’t, and it left whatever forensic evidence and witnesses it left, and you can’t collect ‘more’ like you usually can in a scientific field. It seems to me that an analogous practice would have to be something like ‘go around and ask samples of random people if you think the accused John Smith committed the crime until you get 51% agreeing he’s innocent’, which, despite some historical instances like compurgation, we would consider a bizarre way of collecting legal evidence (eg Jaynes’s “Emperor of China fallacy”).
Seems to me that law and science are fundamentally different enterprises.
Science focuses on discovering “truth.” Law focuses on justice and guilt/innocence.
Those goals are surely overlapping in some ways, But they surely aren’t congruent. It would thus seem to stand to reason that they involve different kinds of evidence.
spitballing here –
in science (at least some mythical Platonic ideal) truth as the ends itself; in law, truth as a means to an end (“justice” or some social/moral aim), ideally necessary but generally not sufficient (requiring reference to societal context and such)
ZJ –
I like it.
“if you’re a researcher and come to a conclusion that wasn’t expected, that you weren’t aiming for, that you weren’t paid or ideologically predisposed to find, that doesn’t automatically mean that you’re right. I’ve seen this attitude a lot, of researchers thinking that their conclusions absolutely must be correct because they came as a surprise …”
Being surprised by a conclusion doesn’t mean that one wasn’t predisposed to find the surprising conclusion. (I know you weren’t implying this, but I’ll stress it nonetheless.) For many of us, we *want* to reach surprising conclusions, and I’ve seen many people, myself included, who have been led astray by the desire to be surprised. And, of course, publishing is far easier with surprising results…
The model suggested by your correspondent only works if the advocacy takes the form of rational arguments over agreed-upon facts. To the extent that the facts themselves are in dispute–i.e., either side is suspicious of the research results and the methods used to obtain them–a lot depends upon the honesty, transparency, and technical skill of those producing them. At the moment, we don’t really have a (decent) system for resolving those disputes, or even for knowing when facts ought to be considered questionable.
One wonderful thing about our adversarial legal system is that the judge (and the parties, to an extent) is empowered and motivated to monitor, regulate, and investigate ethical violations and errors.
In academic research, no one is so empowered. If I falsify my results, or I hide/deny my errors after an honest mistake, the peer review process often enables and rewards me. If I make it past peer review to publication, I’m virtually guaranteed a vigorous defense from the journal editors, which they see as defending themselves. My university is likely to ignore reported malfeasance, and my department may well see defending me as necessary to defend its own reputation. Virtually the only one empowered and motivated to provide oversight is the funder, which is why most of the high-profile comeuppance begins with people being banned from federal grants.
What’s more, an attorney’s record of wins has no impact on a jury’s verdict in the present case, and no impact on a judge’s oversight (in principle). In academia, legitimate and even conclusive evidence that longstanding research conclusions ought to be set aside (e.g., evidence of dishonesty or poor methodology or unreplicability), have often been successfully fended off–at least in the short term–by a strong reputation, a shiny CV, and/or a long list of citations and “conceptual replications.”
In the climate wars, people on both sides argue frequently that results of scientists who are “advocates” should be dismissed out of hand (well, if they don’t agee with the scientists’ findings) because of their advocacy. After all, if they’re advocates then their work must be biased, right?
The problem there (imo) is that much good has been accomplished in this world by advocates. So then is it really a good idea to throw out the science of any advocate? And pretty much all scientists are advocates in a sense, in the sense thst they advocate for their own work, even if they aren’t strongly identified as political advocates. So the idea of discounting work if the scientist is an advocate is generally based on selective reasoning (to define who’s an advocate).
I get the idea of having an algorithm to help judge science in a complex world. It’s nice to have rules of thumb that can be shortcuts, rather than needing to understand all the relevant science to judge a scientist’s work.
But the tiles of thumb need to be seen as shortcuts – that really need to be placed into context rather than applied as some kind of universal law. Advocacy in itself is a good thing! A world of science where advocates are barred from entry would be impoverished, imo ( not to mention bullshit – everyone claims that they’re not advocates, they’re only searching for the unbiased “truth”).
Imo, what’s important is to distinguish between the quality of the advocacy. Does it rest on bias? I happen to think a good way to assess that, is to look for bias in the advocacy itself, not so much the scientific product of the advocate. What quality of reasoning can you see more generally, in how the scientist reasons in their advocacy? Does she mischaracterize opposing views? Does he conflate correlation and causation?
Look for fallacious thinking in their general advocacy as a way to get a foothold into the quality of their advocacy. Of course, a scientist can display fallacious reasoning in their advocacy and still produce quality science. But I think you can gain a way to assess probabilities better when you evaluate the quality of someone’s advocacy.
“Look for fallacious thinking in their general advocacy as a way to get a foothold into the quality of their advocacy.”
The IPCC was chartered to produce high quality advocacy and has mostly done so. Lots of amazing work in climate right now, truly a golden age.
At the other end of the spectrum is extreme climate event attribution, a DOA concept that has produced nothing but nonsense. So the quality of the advocacy is a mixed bag. At the same time I can hardly blame the climate scientists for climbing onto this particular statistical/probabilistic bandwagon, you can go over to the Fangraphs baseball site right now and find a prediction of the day on which Aaron Judge will hit his record-breaking home run based upon rest-of-season simulations.
Matt –
> At the other end of the spectrum is extreme climate event attribution, a DOA concept
I will go with that, where advocates say “this extreme event was caused by climate change.”
But my guess is you want to apply it also to saying, “We can’t say climate change caused this event, but this extreme event is consistent with what we expect to see more frequently with climate change” or “We can’t say climate change caused this event but the probability of this type of event occurring is likely to increase with climate change.”
When I see the former, I consider that to be evidence of poor advocacy, which imo would likely reflect back on the science of the advocate. When I see the latter. I consider that evidence of careful advocacy and that would also likely reflect back on the science of the advocate.
Likewise, when I see an advocate conflating those frames to attack extreme event attribution, I consider that evidence to use in evaluating the advocate.
“Likewise, when I see an advocate conflating those frames to attack extreme event attribution, I consider that evidence to use in evaluating the advocate.”
I would be curious how you view my personal advocacy, since we have followed each other from blog to blog a bit. But rather than ask I will tell you. I advocate for a better approach to the environmental perils we face, with habitat loss and the collapse of the global fishery at the top of my list. I think burning our precious and very limited fossil fuel inventory in motor vehicles was never a good idea and should be phased out quickly. At the same time, building more infrastructure to capture carbon would be folly, it is the building of infrastructure that has to stop.
We probably have a lot in common.
Apologies to Andrew for the thread drift. My personal rule on this blog is to not bring up climate myself.
Matt –
> I would be curious how you view my personal advocacy,…
My sense is that you’re a pretty reasonable advocate, regarding climate, and that you write quite interesting and insightful comments on other topics.
Which is why I was surprised to read some of your more recent comments in climate, in particular the one where you seemed to lump all references to tipping points under the “oceans boiling up” framing.
> I advocate for a better approach to the environmental perils we face, with habitat loss and the collapse of the global fishery at the top of my list. I think burning our precious and very limited fossil fuel inventory in motor vehicles was never a good idea and should be phased out quickly.
That’s consistent with my impression.
> At the same time, building more infrastructure to capture carbon would be folly, it is the building of infrastructure that has to stop.
No doubt, the amount of infrastructure needed to transition out of fossil fuels (to other fuels, and to capture carbon) and also double our energy capacity in 40 years would require an absolutely enormous amount of new infrastructure that will necessarily have to come at a cost.
But (1) I think there’s no free lunch, (2) I think we have sufficient resources to do it although they necessarily would have to be more equitably distributed, (3) we’ve been eating off the stored fat of our existing infrastructure for that past 5 decades without materially increasing capacity, concurrent with a relative decline in the rate of increase of our standard of living (at best, maybe we’re even starting see an absolute decline). The previous period of growth in that standard was was current with a massive build out of our infrastructure. I don’t think that is a coincidence, (4) I think there is a high damage potential of risk from just maintaining the status quo.
> We probably have a lot in common.
Probably so.
Probably you read my comment as a shot at your advocacy. Maybe it was somewhat so, in reaction to those recent comments of yours which surprised me. That was poor form on my part. The target really wasn’t appropriately towards you, but towards a shit ton of other advocates out there on the “skeptic” side of the aisle.
“There is an important question of the degree of legitimacy we grant to academic research as advocacy.”
Tom Wolfe, in Bonfire of the Vanities, discusses how the New York Times is a liberal paper. Up through, say, 2000, it was a liberal paper in the editorial page and the stories it chose to cover. Starting around, say, the mid-2000s, it became a liberal paper in the sense of throwing irrelevant opinions into new articles, aligning with party narratives, and gaslighting and doublethink. From lower starting points of objectivity, the same thing happened at the New York Post and New York Daily News.
Even though she doesn’t agree with Piketty’s conclusions, his framework is solid enough that McCloskey can write a rebuttal that elucidates his contentions. That’s a lot different than Paul Krugman putting aside basic economic frameworks to advance the war with Eastasia.
Publius:
Agreed that when Krugman, or for that matter Tom Wolfe, offers a political opinion, it’s not backed up by reasoning in the same way that a scientific opinion is backed up. On the plus side, neither Krugman and Wolfe are/were in the position of making stuff up in the manner of Joe McCarthy, Alex Jones, etc. What I’m saying is that there are different levels of advocacy, and I would not put Krugman or Wolfe in the “Oceania had always been at war with Eastasia” category.
For the most part I ignore Krugman, but every time I’ve read an article because it flashes up on my web screen while surfing and I say to myself “This guy hasn’t got the slightest clue about economics or how to make an argument, he’s quoting totally the wrong statistics and in an almost obtuse manner, and he’s calculating the differences between things that don’t even have the same dimensions…” When I look at the by-line, it’s always Krugman.
I haven’t read anything Krugman has written for some years now, so your comment spurred me to look at least a little at some of his columns. I think you are being too harsh. Krugman’s academic work was stellar. His columns are clearly biased, enough so that I don’t see the point in reading them. However, I don’t see that he regularly quotes the wrong statistics, calculates incomprehensible differences, or the types of mathematical errors you are referring to. My criticism would be that he is quite selective in the data he cites, only serving to bolster his preferred view of the world.
To be clear, I didn’t say that he always does these things, I wouldn’t know because I mostly avoid reading him. It’s just that whenever I happen to read someone doing those things… it turns out that it’s Krugman.
My feeling is that Krugman is or was exceptionally good at building intellectually appealing, mathematically impressive models of international trade, and is otherwise a pretty ordinary liberal Democrat.
RE: Somebody said, “My feeling is that Krugman is or was exceptionally good at building intellectually appealing, mathematically impressive models of international trade, and is otherwise a pretty ordinary liberal Democrat.”
A long time ago I saw a movie with a little kid (8-10 years old?) who was some kind of mathematics prodigy but he liked to run around dressed as Superman. At some point he admitted, “I know if I weren’t so good with numbers I’d be just another asshole in a cape”. That kind of reminds me of Krugman, except I don’t think Krugman is as self-aware as that kid in the movie.
“if you’re a researcher and come to a conclusion that wasn’t expected, that you weren’t aiming for, that you weren’t paid or ideologically predisposed to find, that doesn’t automatically mean that you’re right. ”
If a researcher reaches a conclusion that was unexpected or undesired, it doesn’t mean it’s right, but it often means s/he probably made a much bigger effort to refute it than s/he would have if it was expected and desired. If s/he did that, at least s/he can say the conclusion is more thoroughly tested than it would have been otherwise.
Once upon a time I was on a jury. It was a low-level criminal case involving a kid (about 20) who was threatening his girlfriend and then took on the police. This was in a small town, the sort of place where everyone knows everyone else. Both the prosecution and the defense came across to me as anemic, barely competent. Still, the weight of the evidence seemed to be against the kid, and we all voted to convict.
Later I discovered that the police had had it out for this kid for years; there had been prior run-ins where apparently the police were the aggressors. Also, it was not at all clear that there was a threat of domestic violence in this particular case; it was a cop who claimed he saw the behavior in question and talked the girlfriend into filing a complaint which she later tried to get out of.
Maybe he was guilty, maybe not. But what was beyond doubt was that his court-appointed defense was so inferior to the prosecution (itself pretty bad) that he had no hope. The resources were all on one side.
That happens a lot in the legal system, and it’s an argument for reducing the resource disparity. I don’t think anyone would say the solution is for the system to be less adversarial. OK, some people say that, but let’s put that thought aside.
The point I’m getting to is that, while science is supposed to proceed through an agonistic process of scientists trying to show each other wrong, sometimes the imbalance of resources is so great that shoddy research triumphs. If a questionable literature nevertheless takes hold of a field — and this happens all the time — dissenters face a difficult challenge. This has certainly been the case in economics, where terribly flawed work (should I name names?) is entrenched in many subfields, in spite of having been convincingly undermined by researchers outside the charmed circle. I suspect it is also the case in other disciplines, although I don’t have direct knowledge.
If scientific advance takes place through contest, the contest has to be approximately fair.
My problem with the adversary system in (social) science is that it slows down the process of (social) science. When I know a paper is decidedly and intentionally one-sided, I know there is an eventual rejoinder on the way. Until the adversaries have duked it out for a while and the dust has settled a bit, it isn’t worth my time to visit the topic (much less use it as a launching pad for related research) unless it is my intention to participate in the “fight,” and I’m typically only willing to participate in the fight if it happens to fall comfortably within my niche.
In other words: any adversarial paper is essentially incomplete until the adversaries have spoken. Why drag things out?
Now compare that to a less adversarial system in which the author tries to be as objective and unbiased as possible, putting in the effort to critique and falsify and qualify what might be their own hunch. The paper is immediately worth my time because I know I won’t have to wait for an informed critique to get a decent grasp of the big picture.
More philosophically, I am skeptical of any approach that doesn’t demand researchers emphasize falsification of their own hypotheses.
In persuasion research, it’s long been known that counter-attitudinal advocacy, i.e., advocating a position against your own interest, enhances perceived credibility. Something like this seems to be going on when people make a counter-attitudinal empirical finding. It enhances the perceived validity of the finding in their own eyes.
Bruce:
Yes, I think that’s what’s happening when the economist James Heckman promotes early childhood education. He’s conservative, but he thinks he discovered that this liberal program has huge positive effects, and this makes him even more sure that he’s correct. Other times, of course, people are persuaded by ideas that fit their preconceptions. So it can be hard to say much about this kind of bias!
Or in other words, so-called scientists basically pitch ideas based on wishful thinking just like politicians or business people or the proverbial man on the street. The only difference is what social norms and pro forma framing conventions different people adhere to in making their pitch. Stuff like p-values serve the same purpose for a social scientist as dog whistles evoking racism or sexism serve for politicians.
I think maybe there’s something missing here.
Personally, I find that I’m more sure about my beliefs if I’ve worked hard at understanding, giving credence to, and investigating counter viewpoints. If my beliefs stand through that process, I think there closer to being validated. Of course, I understand that my ability to truly vet counter viewpoints is inherently limited by a set of biases.
This is why I think that whether or not someone is an advocate is a poor heuristic for evaluating their analysis. What’s important is the attributes of their advocacy. Is the goal to over-power counter viewpoints by the strength of the advocacy? Or is the goal to advocate by showing why the counter viewpoints are wrong. I see value in both kinds of advocacy, but with one kind you can be effective by presenting false arguments whereas with the other you can’t.