Jack Williams writes:
I recently saw your blog here and it made me think of a recent paper that has confused me a ton.
Reading about ranked-choice voting, I found this paper. There appears to be no related work I can easily find, except for causal inference with ordinal data. My thought initially was that it was trying to assume that the voters ranks don’t affect each other (where the unit is the entire ranking), but the unit is actually the individual ranks themselves.
I am not well versed in causal inference, but wouldn’t this violate SUTVA pretty badly? Even in an experimental setting, measuring the treatment effect for any rank is going to have a ton of spillover based on the relative popularity of individual items. How would you differentiate between a single items treatment effect or each other items opposite treatment effect? How would one even verify the rank effects with a different number of ranks or changing one of the choices? What does this even measure if internally you can make the necessary assumptions?
My response: Without reading the paper in detail, I’d say that I follow the Rubin approach of considering causal inference as a prediction problem, predicting potential outcomes given available information. So I’d say that the appropriate way to do causal inference for ranked data is just to model the ranks. I’m not saying that modeling the ranks is easy, just that I see this as more of a “modeling problem” than a “causal inference problem.”
I have nothing of value to contribute to this topic, but I’d say that any paper that begins with “While rankings are at the heart of social science research…” makes me suspicious and cringe to begin with.
Wouldnt SUTVA violations be at core here?
Pedro,
From page 10: “Assumption 2 (SUTVA). Each unit’s observed ranking only depends on her treatment assignment, and there is only a single version of the treatment (Rubin, 1980). While this paper focuses on unit-wise ignorability and SUTVA, future research may relax the two assumptions by focusing on a particular item or a pair of items.”
Do you think this assumption is insufficient? Even if the estimator of interest is some distance measure for a unit’s set of rankings? Happy to hear your thoughts.
If someone is ranked first bc if treatment, then someone will be necessarily pushed down because of that.
I think this a violation of SUTVA. Rankings will depend on the treatment assignment of every unit, I suspect.
There may be some work related to this in different fields, e.g., https://home.uchicago.edu/~amshaikh/webfiles/rankingsconf.pdf
Or https://www.frontiersin.org/articles/10.3389/fdata.2022.888592/full
I have not read the second paper yet.
Note that the paper mentioned in the post is not about ranking individual outcomes. Nobody is ranked first. The outcome for each individual is that individual’s ranking of the elements of a set like {Victim, Officers, Chief, DA, Mayor, Governor, Senators} or a list of candidates.
Thanks, Pedro. I looked at the first paper but this seems to be a rather different question. Here, as far as I understood it, you imagine a potential outcome for ranked choice voting. So without treatment unit i chooses candidates {A, B, C} in that order, but under treatment {B, A, C}. You could imagine treatment as say a email from a campaign.
This was also Jack Williams concern, but I don’t see have this set up is incompatible with SUTVA. You have one potential outcome under control (a set of choices) and a different potential outcome under treatment. You assuming that unit i’s vote (rankings) are not influenced by other votes (rankings), and you’re assuming everyone in treatment gets the same treatment (email).
So the assumption is a lot larger than it seems at first glance. Lets say you have {a,b,c} as the most common ordering and you are trying to measure what the effect of some experiment on that ordering. It has a strong effect and now the most common ordering is {b,c,a}. Did it affect b,c positively or a negatively? What is the magnitude of each effect?
After this, you want to replicate this experiment but with an additional option to see if it changes results. How do add a rank or choice to this experiment completely rewriting the design? The number of possible orders does not increase by 1, but instead by 4x (if you restrict from perm(3,3)->perm(4,4).
What is the difference in the strength of the 1st vs 2nd choice vs 3rd choice vs 4th choice… and how does that translate to the average treatment affect? Are these units actually the same? Is a movement from 5->6 the same unit as a movement from 1->2? When people become less knowledgeable about the final choices, the effect would be weaker right? What if there are other effects that begin to overtake the intended measure at the Nth ranking (like preference for names or ordering of choices)? How do you present these results to a reader?
Thanks for this. Jack Williams writes:. .. “Reading about ranked-choice voting, I found this paper. There appears to be no related work I can easily find, except for causal inference with ordinal data.”
Andrew says ” just to model the ranks.”
Here is a serious (with cheesy graphics) “Explorabke Explanation” format / game as a voting systems simulator with sandbox. Nicky Case who developed it provides the code gratis. No need of anything. Just use it.
So I am suggesting Jack & Andrew, when you have worked you magical (to me) understandings, I urge you to place it in this container and let me play and understand.
*
“To build a better ballot.”
…
“Two answers to that: 1) some voting systems can still be more fair than others, even if none are perfect. And 2) Kenneth Arrow’s proof doesn’tapply to all voting systems! That’s a misconception. It only applies to voting systemswhere you rank candidates. Later, we’ll see some voting systems where you don’t rank candidates – along with other alternatives to our current, glitchy voting system.
“But first, let’s take a closer look at the voting system we do have:
“Note: Instant Runoff Voting is also called “Ranked Choice Voting”, even though there’s other ways to count ranked ballots. IRV is also often just called “Alternative Vote”, even though there’s a flippin’ dozen other voting methods. Such selfish naming! Sheesh!)
“IRV is a bit more complicated than FPTP, but here’s how it works:
…
“Because, this isn’t just about trying to build a better ballot.
“This is about trying to build a better democracy.
<3,
~ Nicky Case
…
https://ncase.me/ballot/
As the graphics are 'cheesy' you may be put off. Don't be. Your graphics can be as academic or corporate as you deem.
Looking forward to playing "… the appropriate way to do causal inference for ranked data is just to model the ranks."
Other "Explorable Explanations".
https://explorabl.es/