Statistical controversy regarding human rights violations in Colomnbia

Megan Price wrote in that she and Daniel Guzmán of the Benetech Human Rights Program released a paper today entitled “Comments to the article ‘Is Violence Against Union Members in Colombia Systematic and Targeted?'” (o aqui en español), which examines an article written by Colombian academics Daniel Mejía and María José Uribe. Price writes [in the third person]:

The paper reviewed by Price and Guzmán concluded that “. . . on average, violence against unionists in Colombia is neither systematic nor targeted.” However, in their response, Price and Guzmán present – in technical and methodological detail – the reasons they find the conclusions in Mejía and Uribe’s study to be overstated. Price and Guzmán believe that weaknesses in the data, in the choice of the statistical model, and the interpretation of the model used in Mejía and Uribe’s study, all raise serious questions about the authors’ strong causal conclusions.

Price and Guzmán point out that unchecked, those conclusions may distort the truth about violence against unions and can mislead important social, economic and political decisions in Colombia. They welcome the opportunity to have a methodological rather than ideological debate about union violence.

The public debate about union violence intensified from 2008 to 2010 as countries negotiating free trade agreements with Colombia, including the U.S., explicitly cited union violence as an obstacle to finalizing agreements. Given the importance of this debate, any study that makes claims about the patterns and magnitude of union violence in Colombia requires the highest level of precision and scientific rigor.

Based on careful review and critique, Price and Guzmán conclude that Mejía and Uribe’s study does not resolve the question, “is violence against union members in Colombia systematic and targeted?” for the following reasons:

• Unknown under-registration. The Mejía and Uribe study uses convenience sample data as the basis for its claims. These data are based on available, observable reports on union homicides and union activity which have been collected without a scientific random selection method. These data cannot be relied upon to represent an underlying larger population or to accurately describe patterns over time and space.

• Possible violation of model assumptions. The statistical methods used in the Mejía and Uribe study to evaluate the relationship between union homicides and “union activity” are based on commonly used least squares regression and instrumental variables analyses. These methods rely on very strong assumptions. However, we do not believe that the study adequately addresses the data’s potential violations of these assumptions and the potential ramifications on the estimates when these assumptions are violated. Violations of these assumptions could change the magnitude of parameter estimates (used to quantify the relationship between union activity and union violence) and the significance of parameter estimates (used to determine the presence or absence of any relationship).

• Uncorrelated errors. One of the main modeling assumptions which we believe these data violate, uncorrelated errors, has a direct affect on significance tests used in the Mejía and Uribe study. Therefore we are highly skeptical about the reliability of the conclusion that union activity and union violence are not significantly associated.

• Poor quality models. The descriptive and analytic results presented in Mejía and Uribe’s study indicate that union member homicide rate is a highly variable outcome measure. This variation results in poor-quality models. We find that control variables in the various model formulations show inconsistent and indeed reversed effects, suggesting problems with the model specification, with the data on homicides, or both.

The poor quality of these models, the unknown under-registration inherent in the data, and the questionable modeling decisions, mean that the strong conclusions in Mejía and Uribe’s study are unsupported by the analyses.

I have not tried to evaluate any of this but I thought some of you might be interested.

3 thoughts on “Statistical controversy regarding human rights violations in Colomnbia

  1. Glancing at their response, it doesn't look particularly persuasive. For instance, they worry about autocorrelation but just say the standard errors will be "incorrect." The type of autocorrelation they describe will lead tests to over-reject not under-reject.

  2. Like John above, I find their critique, as summarized here, unpersuasive. Agree with John that their point about the uncorrelated error assumption works against their critique! Also, from what I gather, they seem to be picking on problems with estimates of nuisance parameters (on "control variables"). I guess the strongest ground that they could have here is some kind of exclusion restriction violation, but they are vague on that. Better for them to work on that as the basis of a critique.

  3. We appreciate John and Cyrus’s interest in our critique, and thank you to Gelman for posting our critique. Although we agree that in general autocorrelation leads to over-rejection of hypothesis tests, we disagree that this weakens our response. Our primary concern with the Mejia and Uribe paper is that we found various assumptions to be inadequately addressed. This led us to question both the methodology and subsequent conclusions. The authors’ failure to describe if or how they accounted for potential autocorrelation is but one example of a lack of explicitly addressing a variety of assumptions.

    We want to emphasize that we are not trying to determine the answer to Mejia and Uribe's question regarding whether or not union violence is systematic and targeted. However we do want to point out that we do not believe they have rigorously answered this question.

    Dr. Price and Mr. Guzmán

Comments are closed.