The book is by Seung-Whan Choi and my review is here. I’ll repost it in a moment, but first I’ll share the strange story of how this review came to be and why it was never published.

Back in 2017 I received an email from the journal Perspectives on Politics to review this book. I was kinda surprised because I don’t know anything about international relations. But then when I received the book in the mail and read it, I understood, as its arguments were all about quantitative methods.

So I wrote my review and sent it in. A few months later I received the page proofs, and they’d changed a lot of my writing! I assumed someone screwed up somewhere and I replied with an annoyed message that the article they sent me was not what I wrote, and they should change it back right away. Then the editors replied with their own annoyed message that as the journal editors they had the right to rewrite whatever they wanted. Or something like that. I don’t remember all the details and the relevant emails don’t seem to be saved on my computer, but the upshot was that I was annoyed, they were annoyed, and they decided not to run the review. Which is fine. It’s their journal and they have the absolute right to decide what to publish in it.

I pretty forgot the whole story but then I was cleaning my webpage and noticed this article that was never published, and it struck me that I went to the trouble of reviewing this person’s book but then neither he nor pretty much anybody else had seen the review.

That seemed like a waste.

So I’m reposting the review here:

New Explorations into International Relations: Democracy, Foreign Investment, Terrorism, and Conflict. By Seung-Whan Choi. Athens, Ga.: University of Georgia Press, 2016. xxxiii +301pp. $84.95 cloth, $32.95 paper.

Andrew Gelman, Columbia University (review for Perspectives on Politics)

This book offers a critical perspective on empirical work in international relations, arguing that well-known findings on the determinants of civil war, the democratic or capitalist peace, and other topics are fragile, that the conclusions of prominent and much cited published papers are crucially dependent on erroneous statistical analyses. Choi supports this claim by detailed examination of several of these papers, along with reanalyses of his own. After that, he presents several completely new analyses demonstrating his approach to empirical work on international relations topics ranging from civilian control of the military to the prevention of terrorism.

I have no expertise on international relations and would not feel comfortable weighing the arguments in any of the examples under consideration. Suffice it to say that I find Choi’s discussions of the substantive positions and the reasoning given on each side of each issue to be clear, and the topics themselves are both important and topical. The book seems, at least to this outsider, to present a fair overview of several controversial topics in modern international relations scholarship, along with an elucidation of the connections between substantive claims, statistical analysis, and the data being used to support each position; as such, I would think it serve as an excellent core for a graduate seminar.

As a methodologist, my main problem with Choi’s reanalyses are their reliance on a few tools—regression, instrumental variables, and statistical significance—that I do not think can always bear the burden of what they are being asked to do. I am not saying that these methods are not useful, nor am I criticizing Choi for using the same methods for different problems—it make sense for any analyst, the present writer included, to heavily use the methods with which are most familiar. Rather, I have specific concerns with the routine attribution of causation to regression coefficients.

For the purpose of this review I do not attempt to carefully read or evaluate the entire book; instead I focus on chapter 1, a reevaluation of James Fearon and David Laitin’s paper, “Ethnicity, Insurgency, and Civil War” (American Political Science Review, 97(1)), and chapter 2, on the democratic or capitalist peace. In both chapters I am convinced by Choi’s arguments about the fragility of published work in this area but am less persuaded by his own data analyses.

The abstract to Fearon and Laitin (1993) begins: “An influential conventional wisdom holds that civil wars proliferated rapidly with the end of the Cold War and that the root cause of many or most of these has been ethnic and religious antagonisms. We show that the current prevalence of internal war is mainly the result of a steady accumulation of protracted conflicts since the 1950s and 1960s rather than a sudden change associated with a new, post-Cold War international system. We also find that after controlling for per capita income, more ethnically or religiously diverse countries have been no more likely to experience significant civil violence in this period.”

Fearon and Laitin’s language moves from descriptive (“proliferated rapidly”) to causal (“root cause . . . the result of . . ..”), then back to descriptive (“no more likely”). The paper continues throughout to mix descriptive and causal terms such as “controlling for,” “explained by,” “determinant,” “proxy,” and “impact.” In a June 27, 2012, post on the Monkey Cage political science blog, Fearon was more explicitly predictive: “The claim we were making was not about the motivations of civil war participants, but about what factors distinguish countries that have tended to have civil wars from those that have not.” Fearon also wrote, “associating civil war risk with measures of grievance across countries doesn’t tell us anything about the causal effect of an exogenous amping up grievances on the risk of civil war.”

There is nothing wrong with going back and forth between descriptive analysis and causal theorizing—arguably this interplay is at the center of social science—but the result can be a blurring of critical discussion. Choi also oscillates between descriptive terms such as “greater risk” and “likely to experience” civil war and causal terms such as “endogeneity” and “the main causes” (p.2). Choi criticizes Fearon and Laitin’s estimates as being “biased at best and inaccurate at worst” (p.3), a characterization I do not understand—but in any case the difficulty here is that bias of an estimator can only be defined relative to the estimand—the underlying quantity being estimated—and neither Fearon/Laitin nor Choi seem to have settled on what this quantity is. Yes, they are modeling the probability of outbreak of civil war, but it is not clear how one is supposed to interpret the particular parameters in their models.

Getting to some of the specifics, I am skeptical of the sort of analysis that proceeds by running a multiple regression (whether directly on data or using instrumental variables) and giving causal interpretations to several of its coefficients. The difficulty is that each regression coefficient is interpretable as a comparison of items (in this case, country-years) with all other predictors held constant, and, it is rare to be able to understand more then one coefficient in this way in a model fit to observational data.

I have similar feelings about the book’s second chapter, which begins with a review of the literature on the democratic or capitalist peace, a topic which is typically introduced to outsiders in general terms such as “Democracies never fight each other” but then quickly gets into the mud of regression specifications and choices of how exactly to measure “democratic” or “peace.” As in the civil war example discussed above, I am more convinced by Choi’s criticisms of the sensitivity of various published claims to assumptions and modeling choices, than I am by his more positive claim that, after correction of errors, “democracy retains its explanatory power in relation with interstate conflict” (p.38). Explanatory power depends on what other predictors in the model, which reminds us that descriptive summaries, like causal claims, do not occur in a vacuum.

Where, then, does this lead us? The quick answer is that statistical analysis of historical data can help us build and understand theories but can rarely on its own provide insight about the past and direct guidance about the future. We can use data analysis within a model to estimate parameters and evaluate, support, or rule out hypotheses; and we can also analyze data more agnostically or descriptively to summarize historical patterns or reveal patterns or anomalies that can raise new questions and motivate new theoretical work.

As Choi both explains in his book, just about any dataset worth analyzing is worth reanalyzing: “The charge that replication studies produce no stand-alone research is ironic in the sense that most empirical research already relies on publicly available data sources . . . Stand-alone researchers claim to be doing original work, but their data often comes from collections previously published by private and government agencies” (p.xxv). I expect that Choi’s explications and reanalyses in several important areas of international relations will be of interest to students and scholars in this field, even if I have qualms about his readiness to assign causal interpretations to regression coefficients.

Perspectives on Politics really dodged a bullet by not publishing that review as written, huh? The only thing that puzzles me is why they asked me to write the review in the first place. What did they think they were going to get from me, exactly?