An economist who would prefer anonymity points to the above wacky graph of a “robust regression.” It’s from a paper written by 2 out of the 3 recent Nobel prize winners in economics!
The full paper is here, and my correspondent points us to p. 921 of the published version.
My correspondent writes:
They can do what they want. They ignore all criticism, even when repeated by mainstream economists in whispers.
“They ignore all criticism” seems pretty standard in science. I guess the only hope is for the field to advance through external criticism. In that sense, it’s fine for questionable papers to be published, as long as data and code are made available and as long as the journals do not hold criticism to higher standards than the original work.
Unfortunately, giving out Nobel prizes is kind of the opposite of criticism (and here’s another recent example).
Wow! I sort of read through the 65 page NBER paper and didn’t bother with the 52 page QJE paper (among the top econ journals), but I am completely dumbfounded about this research. If I can paraphrase the principal finding it is that the places where the Holocaust was more prevalent experienced worse economic outcomes over time. There are quantitative estimates which I might summarize by their finding that the average occupied city was 15% smaller in 1989 than if it had escaped from German occupation. I suppose providing evidence that the Holocaust had bad outcomes on population and economic conditions and quantifying those effects is worthwhile – and it certainly took a lot of effort.
However, the paper reads like a litany of forking paths (required by the messiness of historical data), NHST findings of significant coefficients (enough to call for Anoneuoid’s ire, I would think), and “robustness checks” that make me wonder exactly what their finding means. For one thing, none of the R squares are reported for any of their many regressions – but the few scatterplots shown look very noisy to me. I’m far from convinced that they have found anything more than the fact that the Holocaust had bad effects. There are even a few pie charts (which made it to the published version), despite the almost universal advice to never use them (bar charts are always, in my mind, superior).
There’s lots of interesting history though I can’t speak to whether any of that is new or how it compares with other historical accounts. But the empirical findings seem like a lot of work for little result. The fact that the Holocaust had negative impacts on places where it was most active seems not surprising to me. Estimating the quantitative size of those effects would be the main contribution I see from the paper – but I’m not sure I buy those estimates as particularly accurate or insightful. Removing the Jewish portion of the educated middle class in these places surely would have negative consequences, but there are many dimensions in which these places differ from those that were not occupied by Germans. There were reasons why some places had larger Jewish populations than others before the Holocaust, and I’m unsure that this paper handled that endogeneity. Perhaps it was hidden from my cursory reading by all those NHST results.
Those side-by-side graphs are confusing in the extreme. Ignoring the mumbo-jumbo of t-values and the foolishly excessive number of decimal places, visually it is obvious that Darbent seems to look different. Removing the outlier, Darbent, as would be anticipated, markedly changes the slope.
Despite my being a MOT my entire life, I had never heard of Darbent, the place or its history
https://en.wikipedia.org/wiki/History_of_the_Jews_in_Derbent
As is often remarked, it is possible to learn a great deal via this statistics blog.
In the above, I managed to misspell Derbent, not once, not twice, but three times, albeit consistently.
I always get suspicious when I read about people using STATA (p. 922). I think there is a selection into using STATA for people who are less skilled in statistics, because STATA contains more automation than R, Matlab, Python or Julia. In the words of a colleague of mine, ‘You just have to click!’, i.e. there is no coding involved. This is not a bad thing, but of course it is attractive to those who do not think much about statistical analysis.
I’m often surprised when people look for ways to abstract/automate* data analysis. A fellow researcher was excitedly telling me about some sort of graphing software that makes all the choices for you. If I haven’t been elbows deep in graphing my data, I don’t understand my data!
I think another factor is reproducibility—if I can’t audit your code I’m going to operate with a little more skepticism. Publishing your repo doesn’t make your science good, but if you’re doing good science you’re more likely to play with an open hand.
*I suppose a fair rebuttal is that using R packages is a step more abstracted than writing models in Stan is a step more abstracted than doing my own matrix operations is a step more abstracted than…
Let’s not get sloppy with what we mean by automation. I personally don’t code and I like software with easy to use GUIs. But I do not use software that “makes all the choices for you” and would never recommend these for anyone. Of course, if you call making the matrix operations “making choices” for me, then I’ll have to backtrack, along the lines of your starred footnote. What I believe is that all automation potentially creates problems while at the same time offering improved productivity to users. The important human task is to decide which automation is useful and avoiding that which hides or replaces critical thinking. We can debate where that line lies, but I wouldn’t draw the line at pressing keys versus writing lines of code.
Do you report the steps that you took with the software in your papers? Are these steps easily reproducible by others?
The steps are easily reproducible since the software saves a script of the entire session. Still, as is true for all computer codes, it isn’t always easy to reproduce all the forking paths taken. Data cleaning involves many choices, and it isn’t always easy to follow all of these.
@Entomophagist:
*Especially* when it comes to plotting data, I rely on AI (‘automation’) to provide the code for me. I want plots that look nice and polished, which is why I use ggplot2. But with so many components to choose from, and so many different ways to achieve the same thing, I prefer the automation route. In general, I like the idea of defaults when graphing data. They are reasonable in most cases. Therefore, graphing the data is the first thing I would automate in data analysis (and probably the last, for the reasons given by Dale Lehman).
I would also push back against the notion of irreproducible code. STATA also has code that is stored as do files, a proprietary format. It can be used with a command line. From what I have read, STATA is not the software you want to go to for cutting-edge statistical analysis, but I think you can do good research with it. I am sure we can agree that the problem with most research is not that the software is bad, but that the reasoning behind the analysis is problematic. I agree with Dale Lehman that we must be careful whenever we abdicate our critical thinking to automation or blindly follow defaults.
I respectfully disagree. Not having to code can give you more time to actually analyze. But not everyone uses that free time in this way :D. So there you have at least one Covariate that confounds the causation you see between stata and bad analysts.
On the scatterplot above – they did flip DV and IV, didn’t they? Spectacular fail.
I do not see where my position is contrary to yours. You write: ‘Not having to code can give you more time to actually analyze’ and I could not agree more. But especially when using techniques beyond what could theoretically be done by hand (I am looking at you, OLS!), I think it is good to understand what is going on under the computer hood. I am not suggesting that everyone needs to understand everything that the computer does (I know I know next to nothing compared to what there is to know), but it certainly helps with evaluating the results that the computer presents to you.
How did the plot get published with that typo in the caption: “Conditinal”?
Also, the points do not follow the line they fit.
Apparently predictive skill is not a concern for the authors/reviewers. Why not?
Economics is not a science. Almost nothing is.
Even stuff like astrology strives to make predictions and uses that for legitimacy.
This line that was “fit” though. Its showing their method doesn’t predict anything, but somehow concluding it does anyway since a line of zero slope is even worse.
If you’re genuinely curious it’s because it’s from a working paper, not a journal published article. Working papers shared via NBER often have small mistakes/typos, and may even be incomplete. The plots look different in the QJE publication
In the real sciences, we do not ignore criticism. This is not real science.
Oh my. Well, obviously the biggest problem here is that they’re using real data, which is making their plots *appear* to differ from their conclusions. I asure you it’s only an illusion. If, for example, they were projectiong the social cost of carbon from inefficient water heaters 100 yrs into the future, we could be sure the effect of inefficient water heaters would stupendous, since the data would be scientifically configured to fit the model. Now *that* is Real Science.
Well, anyway, we already know that the world is more complex than it seems to the uneducated unacademic riff-raff. To get the nobel peace prize, you have to start multiple wars. It *seems* counter intuitive. But, of course, thats just due to the cognitive dissonance among the illiteratti.
Chutz:
The layers of sarcasm in your comment are so thick here that it’s impossible to understand what you are saying. For example, your statement, “To get the nobel peace prize, you have to start multiple wars,” is not at all true. Just to start, go to the webpage for the Nobel peace prize. It has the citations for the prizewinners from the past five years, none of whom seem to have started any wars. Again, I get it that you’re trying to make a joke of some sort, but in order to keep our comment section sane, I ask you and others to be more direct in your statements. Thank you for understanding.
Apologies! I’m not sure how to respond. I had no intention of being cryptic. But with due respect the confusion is surprising in the context of a discussion about shady Nobel prizes, since our protagonist is likely the most widely known Nobel recipient in my lifetime (over 50 yoa) and a person who is commonly in the news. But since it is what it is, pray allow me to play the game this one time.
I confess in using the term “started” I took some, but very little, poetic license. Technically, his six wars might be better described using terms like “dramatically escalated”; “instigated”; “abetted”; “perpetuated”; “funded” or numerous other terms that indicate that our protagonist actively avoided being fingered for the violence, even as they sought to use violence to achieve their means. While much of this was obvious – to use the famously ambiguous NPR term “to some” – at the time, the adoring media and cheering crowds claimed that our protagonist offered “hope”.
Afghanistan, Iraq, Libya, Syria and ultimately Ukraine and Gaza. Are you there yet? I understand if not. Tens of thousands of bodies is just the collateral damage of “hope”, so it’s understandable why the “hopers”, crassly used by our warmongerer to advance their objectives, might not notice them. They got something out of the deal. It’s natural under those circumstances to ignore colateral damage.
Of course the Nobel committee would not have had any way of knowing they had selected a rabid war monger for a peace prize. I mean, other than the fact that this particular winner had never done anything remotely relevant to world peace and also had relationships with personalities who promote violence. But we can’t expect them to know everything. The brazen political motivation of this prize shocked me even before the wars, when I was stiall a tepid supporter of its recipient. it was the first even that, over time, lead me to realize the true nature of our protagonist. It foreshadowed many momentous events of recent months and years. I’d think you would have noticed since you’re interested in how authors of fiction foreshadow or give clues about events that occur later in their books. Here we have unpredictable real events foreshadowed by events 15 years previous. Tragically, tens of thousands of people have perished between due to this prize winner’s actions. That alone should have generated more than enough mockery to end the Nobel Peace Prize, but in a world where the Farrakhanians accuse their protagonists of being Nazis, goodness only knows fat the whuck is going on.
The Ukraine issue even as it is today – up top this very moment – is startling. Even a mildly competent reader of Morganthau would understand the Ukrainian situation from the Putin perspective. Apparently people who get their degrees from “universities” where the truth researchers are liars and the presidents whom they idolize are frauds don’t get much contact with Realpolitik. Is self-seromg ignorance an excuse for a towering ego to ignore established knowledge and destroy tens of thousands of lives? Forecasts of a gravity inversion are common, but evidence is scarce. Real evidence – bodies – suggest over 2M dead in Ukraine and Syria alone as a consequence of our peace-prize-winners actions.
Chutz:
To the extent that the wars you mention can be blamed on single people, I’d pin them on Osama Bin Laden, Saddam Hussein, Muammar Gaddafi, Bashar al-Assad, Vladimir Putin, Benjamin Netanyahu, and the leaders of Hamas. You could say George Bush and Dick Cheney too, but I’d give them secondary responsibility as they were reacting to what had been done by others. None of these people received a Nobel prize.
In its over hundred-year history, I don’t know if any Nobel peace prizes have been given to people who have started a war. The most controversial selection may have been Henry Kissinger, who didn’t start the U.S.-Vietnam war but seems to have prolonged it. They also gave it to Woodrow Wilson, but I’m guessing that the entry of the U.S. shortened World War 1, and I wouldn’t blame Wilson for what happened after. And in 1994 it was given to Arafat, Peres, and Rabin, who in retrospect did not make peace in the Middle East, but at least they tried–I wouldn’t say they made things worse. In your above comment I think you’re expressing Barack Obama but I think it’s ridiculous to blame him for all these wars started by others. It seems reasonable to say that he could’ve done a better job as president, not so reasonable to blame him for all those wars. I wouldn’t blame Bush or Clinton for all of them either. Obama’s prize was pretty clearly given in reaction to the previous administration–it was a way to give a kind of anti-Nobel prize to Bush and Cheney. So in that sense it was different from all the other Nobel peace prizes, which were given in a more positive sense. To put it another way, I think you should be able to say that you disagree with the prize being given to Obama without blaming him for six wars that he didn’t start.
In any case, I’d say that, given the long history of the prize, the statement, ““To get the nobel peace prize, you have to start multiple wars,” is pretty much as far from the truth as can be. At best, you can say that the Nobel prize has often been given to powerful public figures, and you could argue that they could’ve been more effective than they had been in preventing or ending war.
Yeah, the idea that Obama was responsible for every – or any – death that happened in the Arab Spring uprisings is some pretty low-brow stuff.
I hesitate to add a veneer of intellectualism where none exists, but this is the “Great Man” theory of history taken to its most wacky extreme. In this tired old trope, the ordinary citizens of places like Libya and Syria are incapable of agency, so their actions must be attributable to the choices of some Great Man somewhere.
Andrew,
“You could say George Bush and Dick Cheney too, but I’d give them secondary responsibility as they were reacting to what had been done by others.”
The Iraq War was a reaction to what had been done by others? That would be news to…everyone. Do you still seriously believe that Hussein had WMDs?
The Taliban were also willing to give Bin Laden up, but Bush/Cheney just made a show of negotiating.
Obama is responsible for starting Libya, and without his support Saudi Arabia would not have started Yemen, which the previous commenter did not mention.
FYI, Abiy Ahmed, the 2019 laureate and Ethiopia PM, is responsible and accused of starting multiple Ethiopia civil wars, which resulted in over 100,000+ deaths. That said, it is indeed rare for Nobel peace prize laureates to start wars given most of them aren’t even heads of state.
Anon:
On the plus side, they never gave one to Pierre Laval.
“Unfortunately, giving out Nobel prizes is kind of the opposite of criticism…”
And then you have the current Nobel in physics, won jointly by John Hopfield and Geoffrey Hinton for their work in machine learning. I know some object that that’s not really physics. I’m neither here nor there on that one.
But even before he got the Nobel Hinton was talking nonsense about current LLMs being conscious and on-track to achieve the kind of super-intelligence that represents an existential threat to humanity. I have no idea whether he’s intensified his assault on reason since winning the prize or not.
Alas, those particular bits of silliness are wide-spread in “Silicon Valley,” by which I mean a socio-cultural phenomenon that’s spread geographically beyond Silicon Valley.
+1. The nobel prize for literature should also have been given to Chat GPT. That would have continued the AI hype nonsense theme quite nicely.
They also misinterpret the interaction. In talking about a specification where the estimate for the interaction term (% jewish by occupation) is -.077, they say “according to this estimate, an occupied city with a 1% share of Jewish populationin1939 should be 7.7% smaller in 1989 than it would otherwise be.” Then they observe that the average Jewish population in occupied cities was about 2% and say “the average occupied city should be about 15% smaller in 1989 than it would have been had it escaped German occupation.” But the estimated main effect of occupation is .327, so the estimated effect for the average occupied city is .327-2*.077=.173–that is, the predicted 1989 population is about 19% BIGGER than a city with 2% Jewish population that wasn’t occupied. This estimate varies across the specification, but it’s positive for 4 of the 6 listed in Table II.
You could also compare it to an occupied city with no Jewish population, which may be more relevant to the story they offer. The estimated main effect for Jewish share is .044, so that would give 2*.044-2*.154=-.07, or about 7% smaller. That estimate is pretty consistent across specifications, and is up to 10% in the specification that they seem to regard as the best.
In a way, this is beside the point: I don’t think they come close to controlling for potentially relevant factors, so I don’t have much confidence in any of the estimates. But it’s still disturbing to see it in a leading journal.
A correction: the comparison of an occupied city with 2% Jewish population to an occupied city with no Jewish population is 2*.044-2*.077= -.066.
My advice is to be suspicious of any “historical” article in a Top 5 economics journal. The history is Wikipedia level (really, sometimes Wikipedia is a lot better because the student who cribbed for the article did a poor job of paraphrasing Wikipedia); the data are often not what the authors say (or believe); and the econometrics can be egregiously wrong. AJR shoulder a lot of the responsibility for this development.
The economics journals do not publish replies, so this cannot be corrected. Sometimes authors get a criticism into an economic history journal, but that is hard, as well, because referees for those journals want to defend the core approach and do not want to encourage anyone asking too many questions.
Why does the R2 matter here?
One of the assumptions made when doing regression is that the model is true (describes the process that generated the data).
This is always a problem, but sometimes an argument can be made its approximate enough for the coefficients to have some meaning. The violation of that assumption is very obvious when there is essentially no fit (low R^2) like in this case.
first time I’ve seen mis-specification of the model tested using R2. I’ve read Andrews books and he’s never highlighted R2 to my knowledge as overly important. (correct me if I’m wrong andrew)
I’d guess he just used the most basic stat everyone knows. Only the NHST “fits better than slope = 0” statistic could possibly make this seem like something worth looking at. Which is impressive, really.
It isn’t something worth wasting further energy on.
Gabby:
I have written some papers on R-squared and it does come up in our books, but really the story is in the above graphs. It’s kind of hilarious that they called their regression results “robust.” The near-zero R-squared is just one part of the story, and it seemed like a good headline.
Not an economist, but regarding your link to the apparent controversy around the 2022 award, didn’t the Diamond-Dybvig model (named after two of that year’s winners) hold up pretty well when Silicon Valley Bank went under a few months later?
If the Nobel Memorial prize has gained some kind of prescience, then maybe we ought to look out for how the relative inclusivity of institutions today will shape future economic development — with, say, an eye on the state imposed abandonment of inclusive principles in the government agencies, private corporations and academic institutions of a major modern economy …