I was talking with an economist today about the recent prize given to the authors of the very influential 2001 article, The Colonial Origins of Comparative Development: An Empirical Investigation. According to my colleague, many economists have issues with that paper, with issues regarding data quality, the weakness of the instrument, and problems of selection bias in the analysis. The concern seems to be that those data could be used to show just about anything. Which, as usual, does not mean that their theories are wrong, just that their data are consistent with other theories.
I’ve never looked into this particular example, and a search of the blog turned up only this comment, so I’ll just pass along some references that my colleague sent to me:
Daron Acemoglu, Simon Johnson, and James Robinson (2001), The colonial origins of comparative development: An empirical investigation
David Y. Albouy (2012), The Colonial Origins of Comparative Development: An Empirical Investigation: Comment
Morgan Kelly (2019), The Standard Errors of Persistence
This recent post from Alex Tabarrok gives some sense of the importance and ideological dimensions of the work under discussion.
Some people love this work, some people don’t
From a sociology-of-science perspective, it’s interesting how this work is viewed differently in different corners of economics. As discussed by Tabarrok, “The Colonial Origins of Comparative Development” has had a huge influence within and outside the field, and it generally appears to be viewed very positively. But researchers who focus on methodology and replication don’t trust it. I wonder whether some of the popularity of that paper and subsequent work in that area is that it has something to offer to both the right and the left, unlike a lot of work in macroeconomics which will push in just one direction.
The newer version of that Morgan Kelly paper now called Understanding Persistence is even more comprehensive and undermines an even broader set of persistence studies, including the colonial origins paper if I remember correctly. https://economics.yale.edu/sites/default/files/understanding_persistence_ada-ns.pdf
Although they emphatically say that their goal is not to critique any specific studies (p. 8) , it’s hard for me to look at their figures and not do exactly that.
I wanted to lift up this comment from their conclusion:
“The simplest and most important step [for global studies based on country level data] however is simply to graph your data. It is always advisable to be skeptical of any claimed regression result where a scatter plot of the main variables is not provided. But with spatial data it is equally important to see simple coloured maps of the dependent and explanatory variables along with residuals to understand quickly whether a regression is fitting anything more profound than spatial trends” (p. 29).
But hardly anyone does this, at least in the stuff I read. Why don’t we get basic pictures? On a meaningful, intuitive scale. Not transformed into oblivion so that I then am tempted to look for asterisks to tell me why I should care. In 2024 why is a simple picture not a basic publication requirement? People take up space in their articles with the formula for linear regression for heaven’s sake. My kingdom for even a link to a scatterplot. Is it because it’s too hard with nesting?
And with lots of nested levels in applied work (e.g., cities, counties, countries, regions, etc.), why fiddle with the residuals at this level and not that one?
My takeaway is that you don’t need to pick on individual studies because the entire literature is some mixture of aggressively underpowered due to only having a small number of highly autocorrelated variables and where there is power it is due to endogeneity from not dealing with spatial trends. Sadly, I’m not convinced that this will kill off unreliable studies in this area. I’ve seen several papers cite the Kelly paper and then use Conley standard errors as their spatial autocorrelation solution when there are nowhere near enough effective clusters in the data for those to work.
The final version of “The Standard Errors of Persistence”, now co-authored with Timothy Conley, has recently been accepted by the Journal of International Economics. It can be found here
https://www.researchgate.net/publication/384990248_The_Standard_Errors_of_Persistence
Timely!
“Institutions” is just the type of concept as rejecting the null hypothesis: there are just too many variations on what that means. I think everyone agrees that institutions matter. But it is quite a different thing to show that a particular type of institution (e.g., “free” markets, or “free markets with regulation”) causes a different level of wealth or a different distribution of wealth. I have not read the literature in question, so I’m not prepared to say these economists have or have not conducted appropriate analyses to support their case. But I do know that this literature is seized upon to support particular views (markets supported by well defined property rights) or to attack others (the idea that such markets need to have appropriate regulations).
I think the problem of tying particular institutions to particular aggregate and distributional outcomes is tremendously difficult. The number of observations is very limited, especially for which there is data available – just how many countries and how many years do we have such data? Probably more importantly, measurement is elusive: how “much” are property rights well defined? For that matter, which property rights are we talking about? It seems to me quite like those rankings of democracies. Such measures are bound to be noisy.
At the same time, I think studying the impact of institutions on economic growth and prosperity is to be congratulated. Too much of the work in economics has ignored this and/or failed to be rigorous in the least. Tackling important and difficult issues should be recognized and rewarded. And if it spurs more careful critique and analysis, that is what we want from science isn’t it? I can’t comment on the particulars of this body of work since I haven’t read it, but I am not surprised that is attracts the kind of attention and reaction noted in the post.
Part of the skepticism about this paper reflects their general approach to research. The Albouy paper documents a fast-and-loose treatment of the data that has been noted in many other of their papers. You actually linked to a (different) article that points out how they use nonsense data.
+1. The data for a large portion of the sample was simply guessed at.
I get the impression that they wrote they wrote their conclusions first, and then acquired a mountain of data that would make their reasoning hard to attack.
To make it a little bit more specific, a couple of alternative hypotheses *even if* their data are good are that it was European human capital or trade with Europe that caused the observed pattern. This cannot be ruled out with their data (and it probably never can be ruled out). This is even stated in the Nobel announcement. See Noah Smith’s post:
https://www.noahpinion.blog/p/a-nobel-for-the-big-big-questions
Noah speculates that the authors won due to their influence (like many other winners), and the Econ Nobel is going back to its pre-2012 roots as a prize for popularity instead of empirically correct findings.
As someone previously unfamiliar with the “Settler Mortality” debate, a quick look at the original paper and some commentary seems to indicate that the contrast is made of the US, Canada, Australia, and New Zealand vs. lots of other, mostly tropical, colonies. I’m not sure what the quantitative modeling can do here, but the most obvious things is that the first four were “colonies” in the original Roman sense: a settlement of Roman citizens, ruled as Romans, *not* a conquered region ruled over by Roman officials or a client king (e.g. Jerusalem), nor an allied state which became a protectorate and eventually fully incorporated into the empire (e.g. Masilia, modern Marseille). Few other British or European colonies were established in the original Roman manner.
The numerous concomitants and accompanying features of such a form of colonization would be hard to disentangle, making it hard to say “private property” is the key.
All the settlers dying would, indeed, squelch the colony, but other alternatives– such as “human capital or trade with Europe” mentioned by anon, and many others one could imagine– seem equally plausible.
(The above does not address the problem of data quality in the original study raised by many others.)
Like many comments on this blog, many comments here have little to do with the original post and are instead just flippant reactions to what someone thinks the research Andrew discussed was about. This kind of comment demonstrates a big problem with research: everyone is so quick to add their two cents that there is little effort to understand and think. The AJR paper is both better and worse than one would think after skimming the abstract
How is it better and how is it worse?
The paper tries to contend with some of the objections raised here. It is less naive than the commenters assume.
Empirically, however, it is outrageous.
Economist Dietz Vollrath has a four part “skeptics” take on AJR and the broader institutions literature that they are a part of. It is purposefully harsh, but pretty forcefully argues that much of that literature (as of 2014) fails to prove “institutions” as a coherent and specific category, matter.
https://growthecon.wordpress.com/2014/11/18/the-skeptics-guide-to-institutions-part-1/